Repository: great-expectations/great_expectations

Issue:
get_validator method do not work
Hello!
I have a problem with get_validator component.

Here’s my code:
```
batch_request = BatchRequest(
    datasource_name="redshift_",
    data_connector_name="default_inferred_data_connector_name",
    data_asset_name="daily_chargeback_table_v1",  # this is the name of the table you want to retrieve
)
context.create_expectation_suite(
    expectation_suite_name="test_suite", overwrite_existing=True
)
validator = context.get_validator(
    batch_request=batch_request, expectation_suite_name="test_suite"
)
print(validator.head())
```

I get this exception:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-67-16f90e0aa558> in <module>
      8 )
      9 validator = context.get_validator(
---> 10     batch_request=batch_request, expectation_suite_name="test_suite"
     11 )
     12 print(validator.head())
.
.
.

~/anaconda3/lib/python3.7/site-packages/great_expectations/execution_engine/sqlalchemy_execution_engine.py in _build_selectable_from_batch_spec(self, batch_spec)
    979                     )
    980                     .where(
--> 981                         sa.and_(
    982                             split_clause,
    983                             sampler_fn(**batch_spec["sampling_kwargs"]),
TypeError: table() got an unexpected keyword argument 'schema'
```

My Datasource configuration like:
```
name: redshift_
class_name: Datasource
execution_engine:
  class_name: SqlAlchemyExecutionEngine
  credentials:
    host: redshift_host
    port: '5443'
    username: username
    password: password
    database: dbname
    query:
      sslmode: prefer
    drivername: postgresql+psycopg2
data_connectors:
  default_runtime_data_connector_name:
    class_name: RuntimeDataConnector
    batch_identifiers:
      - default_identifier_name
  default_inferred_data_connector_name:
    class_name: InferredAssetSqlDataConnector
    name: whole_table
```

My environment:
MacOS
python 3.7.4
great_expectations 0.13.34

I will be grateful for any help.


Discussion:
Hi @redreckless! Thank you for opening this issue. We will take a look and respond over the next week or so.
I appear to be having a similar error using MacOS, GE version `0.13.42`, python `3.6.9`, also using Redshift. In my case though, I'm trying to create and test a checkpoint, which invokes `get_validator()`. @redreckless did you find a solution?
Hi @redreckless - thanks for this and apologies for delays! Is this still an issue for you?

@tdstark - thanks for posting here. Would you please include your `batch_request` and `Datasource` config (removing an sensitive information)? Are you able to instantiate a `Validator` using this, and the `Checkpoint` is just not working?
@talagluck 
```
# Welcome to Great Expectations! Always know what to expect from your data.
#
# Here you can define datasources, batch kwargs generators, integrations and
# more. This file is intended to be committed to your repo. For help with
# configuration please:
#   - Read our docs: https://docs.greatexpectations.io/en/latest/reference/spare_parts/data_context_reference.html#configuration
#   - Join our slack channel: http://greatexpectations.io/slack

# config_version refers to the syntactic version of this config file, and is used in maintaining backwards compatibility
# It is auto-generated and usually does not need to be changed.
config_version: 3.0

# Datasources tell Great Expectations where your data lives and how to get it.
# You can use the CLI command `great_expectations datasource new` to help you
# add a new datasource. Read more at https://docs.greatexpectations.io/en/latest/reference/core_concepts/datasource.html
datasources:
  data_dir:
    module_name: great_expectations.datasource
    class_name: Datasource
    data_connectors:
      default_inferred_data_connector_name:
        base_directory: ../../../data
        default_regex:
          group_names:
            - data_asset_name
          pattern: (.*)(\.csv|\.json)
        class_name: InferredAssetFilesystemDataConnector
        module_name: great_expectations.datasource.data_connector
      default_runtime_data_connector_name:
        batch_identifiers:
          - default_identifier_name
        class_name: RuntimeDataConnector
        module_name: great_expectations.datasource.data_connector
    execution_engine:
      module_name: great_expectations.execution_engine
      class_name: PandasExecutionEngine
  database_connection:
    module_name: great_expectations.datasource
    class_name: Datasource
    data_connectors:
      default_runtime_data_connector_name:
        batch_identifiers:
          - default_identifier_name
        class_name: RuntimeDataConnector
        module_name: great_expectations.datasource.data_connector
      default_inferred_data_connector_name:
        class_name: InferredAssetSqlDataConnector
        module_name: great_expectations.datasource.data_connector
    execution_engine:
      module_name: great_expectations.execution_engine
      class_name: SqlAlchemyExecutionEngine
      credentials:
        host: hostname.com
        port: '5439'
        username: username
        password: password
        database: db_name
        query:
          sslmode: prefer
        drivername: postgresql+psycopg2
config_variables_file_path: uncommitted/config_variables.yml

# The plugins_directory will be added to your python path for custom modules
# used to override and extend Great Expectations.
plugins_directory: plugins/

stores:
# Stores are configurable places to store things like Expectations, Validations
# Data Docs, and more. These are for advanced users only - most users can simply
# leave this section alone.
#
# Three stores are required: expectations, validations, and
# evaluation_parameters, and must exist with a valid store entry. Additional
# stores can be configured for uses such as data_docs, etc.
  expectations_store:
    class_name: ExpectationsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: expectations/

  validations_store:
    class_name: ValidationsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: uncommitted/validations/

  evaluation_parameter_store:
    # Evaluation Parameters enable dynamic expectations. Read more here:
    # https://docs.greatexpectations.io/en/latest/reference/core_concepts/evaluation_parameters.html
    class_name: EvaluationParameterStore

  checkpoint_store:
    class_name: CheckpointStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      suppress_store_backend_id: true
      base_directory: checkpoints/

expectations_store_name: expectations_store
validations_store_name: validations_store
evaluation_parameter_store_name: evaluation_parameter_store
checkpoint_store_name: checkpoint_store

data_docs_sites:
  # Data Docs make it simple to visualize data quality in your project. These
  # include Expectations, Validations & Profiles. The are built for all
  # Datasources from JSON artifacts in the local repo including validations &
  # profiles from the uncommitted directory. Read more at https://docs.greatexpectations.io/en/latest/reference/core_concepts/data_docs.html
  local_site:
    class_name: SiteBuilder
    # set to false to hide how-to buttons in Data Docs
    show_how_to_buttons: true
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/
    site_index_builder:
      class_name: DefaultSiteIndexBuilder

anonymous_usage_statistics:
  data_context_id: h2h2h2-h2h2h2h-2h2h2h
  enabled: true
notebooks:
concurrency:
  enabled: false
```
Here is my checkpoint config:
```
my_checkpoint_name = "test_checkpoint"  # This was populated from your CLI command.

yaml_config = f"""
name: {my_checkpoint_name}
config_version: 1.0
class_name: SimpleCheckpoint
run_name_template: "%Y%m%d-%H%M%S-my-run-name-template"
validations:
  - batch_request:
      datasource_name: database_connection
      data_connector_name: default_inferred_data_connector_name
      data_asset_name: data_asset_name
      data_connector_query:
        index: -1
    expectation_suite_name: data_asset_suite
"""
```

I'm actually trying to set up a Checkpoint using the Jupyter Notebook, and when I get to the final step to test, I get an error. Here is the traceback:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-024902fcd744> in <module>
----> 1 context.run_checkpoint(checkpoint_name=my_checkpoint_name)
      2 #context.open_data_docs()

~/.local/.../lib/python3.6/site-packages/great_expectations/core/usage_statistics/usage_statistics.py in usage_statistics_wrapped_method(*args, **kwargs)
    284                     nested_update(event_payload, args_payload_fn(*args, **kwargs))
    285 
--> 286                 result = func(*args, **kwargs)
    287                 message["success"] = True
    288             except Exception:

~/.local/.../lib/python3.6/site-packages/great_expectations/data_context/data_context.py in run_checkpoint(self, checkpoint_name, template_name, run_name_template, expectation_suite_name, batch_request, action_list, evaluation_parameters, runtime_configuration, validations, profilers, run_id, run_name, run_time, result_format, ge_cloud_id, expectation_suite_ge_cloud_id, **kwargs)
   3334         checkpoint_run_arguments: dict = dict(**checkpoint_config, **kwargs)
   3335 
-> 3336         return checkpoint.run(**checkpoint_run_arguments)
   3337 
   3338     def test_yaml_config(

~/.local/.../python3.6/site-packages/great_expectations/checkpoint/checkpoint.py in run(self, template_name, run_name_template, expectation_suite_name, batch_request, action_list, evaluation_parameters, runtime_configuration, validations, profilers, run_id, run_name, run_time, result_format, expectation_suite_ge_cloud_id)
    423                         run_id=run_id,
    424                         idx=idx,
--> 425                         validation_dict=validation_dict,
    426                     )
    427             else:

~/.local/.../checkpoint/checkpoint.py in _run_validation(self, substituted_runtime_config, async_validation_operator_results, async_executor, result_format, run_id, idx, validation_dict)
    281                     expectation_suite_ge_cloud_id
    282                     if self.data_context.ge_cloud_mode
--> 283                     else None
    284                 ),
    285             )

~/.local/.../data_context/data_context.py in get_validator(self, datasource_name, data_connector_name, data_asset_name, batch_request, batch_request_list, batch_data, data_connector_query, batch_identifiers, limit, index, custom_filter_function, sampling_method, sampling_kwargs, splitter_method, splitter_kwargs, runtime_parameters, query, path, batch_filter_parameters, expectation_suite_ge_cloud_id, batch_spec_passthrough, expectation_suite_name, expectation_suite, create_expectation_suite_with_name, **kwargs)
   1759                     batch_filter_parameters=batch_filter_parameters,
   1760                     batch_spec_passthrough=batch_spec_passthrough,
-> 1761                     **kwargs,
   1762                 )
   1763             )

~/.local/.../core/usage_statistics/usage_statistics.py in usage_statistics_wrapped_method(*args, **kwargs)
    284                     nested_update(event_payload, args_payload_fn(*args, **kwargs))
    285 
--> 286                 result = func(*args, **kwargs)
    287                 message["success"] = True
    288             except Exception:

~/.local/.../data_context/data_context.py in get_batch_list(self, datasource_name, data_connector_name, data_asset_name, batch_request, batch_data, data_connector_query, batch_identifiers, limit, index, custom_filter_function, sampling_method, sampling_kwargs, splitter_method, splitter_kwargs, runtime_parameters, query, path, batch_filter_parameters, batch_spec_passthrough, **kwargs)
   1658         datasource_name = batch_request.datasource_name
   1659         datasource: Datasource = cast(Datasource, self.datasources[datasource_name])
-> 1660         return datasource.get_batch_list_from_batch_request(batch_request=batch_request)
   1661 
   1662     def get_validator(

~/.local/.../datasource/new_datasource.py in get_batch_list_from_batch_request(self, batch_request)
    188                     batch_markers,
    189                 ) = data_connector.get_batch_data_and_metadata(
--> 190                     batch_definition=batch_definition
    191                 )
    192                 new_batch: Batch = Batch(

~/.local/.../datasource/data_connector/data_connector.py in get_batch_data_and_metadata(self, batch_definition)
    105         batch_spec: BatchSpec = self.build_batch_spec(batch_definition=batch_definition)
    106         batch_data, batch_markers = self._execution_engine.get_batch_data_and_markers(
--> 107             batch_spec=batch_spec
    108         )
    109         self._execution_engine.load_batch_data(batch_definition.id, batch_data)

~/.local/.../sqlalchemy_execution_engine.py in get_batch_data_and_markers(self, batch_spec)
   1103             else:
   1104                 selectable: Selectable = self._build_selectable_from_batch_spec(
-> 1105                     batch_spec=batch_spec
   1106                 )
   1107 

~/.local/.../sqlalchemy_execution_engine.py in _build_selectable_from_batch_spec(self, batch_spec)
   1046             sa.select("*")
   1047             .select_from(
-> 1048                 sa.table(table_name, schema=batch_spec.get("schema_name", None))
   1049             )
   1050             .where(split_clause)

TypeError: table() got an unexpected keyword argument 'schema'

```
@talagluck @NathanFarmer any thoughts?
Thanks for sharing, @tdstark! We are taking a look internally, and will be in touch when we have made progress or if we have any further questions.
@talagluck any progress on this? I'm probably going to be pivoting back to GE in the next few days. This could be a misconfiguration on my part but I'm still not sure what the issue is.
Hi @tdstark - thanks for the ping here! I wanted to check, what version of SQL Alchemy are you on? `sa.table` ordinarily accepts the keyword argument `schema` with no problem, so I wonder whether this has to do with a version issue.
Hi @tdstark - I've been trying to replicate this, but haven't been successful. To confirm, are you able to instantiate a Validator? Does this not work at all, or does it only not work in your Checkpoint? What versions of Great Expectations, SqlAlchemy, and the SqlAlchemy connector are you running?
@talagluck I've only been working with Checkpoints, but I can't even get the Checkpoint to work in Jupyter Notebooks. Currently, Great Expectations was pinned to `0.13.44`, SQLAlchemy was pinned to `1.3.15`.
Hi @tdstark - thanks for the info. I've successfully reproduced on SQLAlchemy 1.3.15. It looks like this has to do with the version number, and will work if you use SQLAlchemy >= 1.3.18. I'm going to work on updating our documentation to make this clearer.
@talagluck this appears to have been resolved! Thank you!

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,7 @@
     "install_requires": required,
     "extras_require": {
         "spark": ["pyspark>=2.3.2"],
-        "sqlalchemy": ["sqlalchemy>=1.3.16"],
+        "sqlalchemy": ["sqlalchemy>=1.3.18"],
         "airflow": ["apache-airflow[s3]>=1.9.0", "boto3>=1.7.3"],
         "gcp": [
             "google-cloud>=0.34.0",


Test patch: