Repository: great-expectations/great_expectations

Issue:
Batch request data asset name for postgres datasource cannot include schema
(Updating here to note that this is a Not a Contribution)

**Describe the bug**
In v0.13.7, I'm trying to specify a table in a custom schema to validate against. I've tried to set the data asset name as `schema_name.table_name` but that gives me an error here:

https://github.com/great-expectations/great_expectations/blob/903237f009cd98f93f4607b676568e72b72ade4c/great_expectations/datasource/data_connector/configured_asset_sql_data_connector.py#L139-L152

It seems that the `_data_references_cache` dict has keys that are the table name without the schema. 

I'm not sure if this is a real bug or if I'm doing something wrong and there's an easy way to specify the schema as part of the batch request (or even in the datasources definition, which would also be fine for me).

**To Reproduce**
Steps to reproduce the behavior:
1. Spin up a postgres instance (e.g., with docker as below)
```bash
docker run \
    -e POSTGRES_USER=great \
    -e POSTGRES_PASSWORD=expectations \
    -e POSTGRES_DB=ge \
    -p "5432:5432" \
    postgres:13.1
```

2. Connect to the test db...
```bash
PGPASSWORD=expectations psql -h localhost -p 5432 -U great -d ge
```
...and create some fake data
```sql
CREATE SCHEMA testing;

CREATE TABLE testing.prices (
    id VARCHAR(64),
    category VARCHAR(54),
    name VARCHAR(64),
    price NUMERIC
);

-- everything the body needs
INSERT INTO testing.prices (id, category, name, price)
VALUES
    ('abc', 'cheese',   'cheddar',          10),
    ('jkl', 'cheese',   'gouda',            15),
    ('mno', 'cereal',   'cheerios',         3),
    ('pqr', 'cereal',   'froot loops',      2),
    ('stu', 'cereal',   'frosted flakes',   4);
```
3. Start a fresh project with this config file
```yaml
config_version: 2.0

datasources:
  my_postgres_db:
    credentials:
      drivername: postgresql
      host: localhost
      port: '5432'
      username: great
      password: expectations
      database: ge
    module_name: great_expectations.datasource
    class_name: SimpleSqlalchemyDatasource
    introspection:
      whole_table: { }

config_variables_file_path: uncommitted/config_variables.yml

plugins_directory: plugins/

validation_operators:
  action_list_operator:
    class_name: ActionListValidationOperator
    action_list:
      - name: store_validation_result
        action:
          class_name: StoreValidationResultAction
      - name: store_evaluation_params
        action:
          class_name: StoreEvaluationParametersAction
      - name: update_data_docs
        action:
          class_name: UpdateDataDocsAction

stores:
  expectations_store:
    class_name: ExpectationsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: expectations/

  validations_store:
    class_name: ValidationsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: uncommitted/validations/

  evaluation_parameter_store:
    class_name: EvaluationParameterStore

expectations_store_name: expectations_store
validations_store_name: validations_store
evaluation_parameter_store_name: evaluation_parameter_store

data_docs_sites:
  local_site:
    class_name: SiteBuilder
    show_how_to_buttons: true
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/
    site_index_builder:
      class_name: DefaultSiteIndexBuilder
```

4. Run this script to get the error
```python
from pprint import pprint

from great_expectations import get_context
from great_expectations.core import ExpectationConfiguration
from great_expectations.core.batch import BatchRequest

context = get_context()
suite = context.create_expectation_suite('testing123', overwrite_existing=True)

suite.add_expectation(ExpectationConfiguration(
    expectation_type='expect_column_to_exist',
    kwargs={
        'column': 'id',
    },
))

batch_request = BatchRequest(
    datasource_name='my_postgres_db',
    data_asset_name='testing.prices',  # this fails
    # data_asset_name='prices',  # this works
    data_connector_name='whole_table',
)

validator = context.get_validator(
    batch_request=batch_request,
    expectation_suite=suite,
)

results = context.run_validation_operator(
    validation_operator_name='action_list_operator',
    assets_to_validate=[validator],
)

print(f'Success? {results.success}')
pprint(results)
```

**Expected behavior**

I expect the validation sequence to run through successfully and generate results. Note that if you replace `data_asset_name` in the `BatchRequest` constructor with `'prices'` instead of `'testing.prices'`, it works as expected.

**Environment (please complete the following information):**
 - Operating System: MacOS 10.15.4
 - Great Expectations Version: 0.13.7 (also tested on 0.13.4 with the same results)

**Additional context**
Again, I'm wondering if there's some other way to specify what schema I want -- a couple thoughts come to mind
* is there another param that `BatchRequest` takes? 
* can I specify this in the `introspection` or `tables` section of the datasources definition? I"m not sure I quite understand what the purposes of those keys are 😅 
* is there a way to run a custom sql query (similar to the `query` arg from `batch_kwargs` in v0.12)? 



Discussion:
@ryanaustincarlson Thank you for following up and filing this! Adding to the queue to be addressed by the core team.

Resolving patch:
diff --git a/great_expectations/execution_engine/sqlalchemy_execution_engine.py b/great_expectations/execution_engine/sqlalchemy_execution_engine.py
--- a/great_expectations/execution_engine/sqlalchemy_execution_engine.py
+++ b/great_expectations/execution_engine/sqlalchemy_execution_engine.py
@@ -802,7 +802,13 @@ def _build_selectable_from_batch_spec(self, batch_spec):
                         )
                     )
                 )
-        return sa.select("*").select_from(sa.text(table_name)).where(split_clause)
+        return (
+            sa.select("*")
+            .select_from(
+                sa.table(table_name, schema=batch_spec.get("schema_name", None))
+            )
+            .where(split_clause)
+        )
 
     def get_batch_data_and_markers(
         self, batch_spec: BatchSpec


Test patch:
diff --git a/tests/datasource/conftest.py b/tests/datasource/conftest.py
--- a/tests/datasource/conftest.py
+++ b/tests/datasource/conftest.py
@@ -1,3 +1,5 @@
+import os
+
 import pytest
 
 from great_expectations.datasource import (
diff --git a/tests/execution_engine/test_sqlalchemy_batch_data.py b/tests/execution_engine/test_sqlalchemy_batch_data.py
--- a/tests/execution_engine/test_sqlalchemy_batch_data.py
+++ b/tests/execution_engine/test_sqlalchemy_batch_data.py
@@ -1,6 +1,7 @@
 import pandas as pd
 import pytest
 
+from great_expectations.core.batch import BatchSpec
 from great_expectations.execution_engine import SqlAlchemyExecutionEngine
 from great_expectations.validator.validator import Validator
 
@@ -37,7 +38,7 @@ def test_instantiation_with_table_name(sqlite_view_engine):
     assert batch_data.use_quoted_name == False
 
 
-# REMOVING PENDING READD OF table.head metric
+# REMOVING PENDING READ OF table.head metric
 # def test_head(sqlite_view_engine):
 #     # Create a larger table so that we can downsample meaningfully
 #     df = pd.DataFrame({"a": range(100)})
@@ -59,7 +60,6 @@ def test_instantiation_with_table_name(sqlite_view_engine):
 
 
 def test_instantiation_with_and_without_temp_table(sqlite_view_engine, sa):
-
     print(get_sqlite_temp_table_names(sqlite_view_engine))
     assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 1
     assert get_sqlite_temp_table_names(sqlite_view_engine) == {"test_temp_view"}
@@ -73,7 +73,7 @@ def test_instantiation_with_and_without_temp_table(sqlite_view_engine, sa):
     )
     assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 1
 
-    selectable = sa.select("*").select_from(sa.text("test_table"))
+    selectable = sa.select("*").select_from(sa.text("main.test_table"))
 
     # If create_temp_table=False, a new temp table should NOT be created
     SqlAlchemyBatchData(
@@ -99,43 +99,23 @@ def test_instantiation_with_and_without_temp_table(sqlite_view_engine, sa):
     )
     assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 3
 
-
-def test_instantiation_with_and_without_temp_table(sqlite_view_engine, sa):
-
-    print(get_sqlite_temp_table_names(sqlite_view_engine))
-    assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 1
-    assert get_sqlite_temp_table_names(sqlite_view_engine) == {"test_temp_view"}
-
-    # When the SqlAlchemyBatchData object is based on a table, a new temp table is NOT created, even if create_temp_table=True
-    SqlAlchemyBatchData(
-        execution_engine=sqlite_view_engine,
-        table_name="test_table",
-        create_temp_table=True,
-    )
-    assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 1
-
-    selectable = sa.select("*").select_from(sa.text("test_table"))
-
-    # If create_temp_table=False, a new temp table should NOT be created
-    SqlAlchemyBatchData(
-        execution_engine=sqlite_view_engine,
-        selectable=selectable,
-        create_temp_table=False,
-    )
-    assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 1
-
-    # If create_temp_table=True, a new temp table should be created
+    # testing whether schema is supported
+    selectable = sa.select("*").select_from(sa.table(name="test_table", schema="main"))
     SqlAlchemyBatchData(
-        execution_engine=sqlite_view_engine,
-        selectable=selectable,
-        create_temp_table=True,
-    )
-    assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 2
-
-    # If create_temp_table=True, a new temp table should be created
-    SqlAlchemyBatchData(
-        execution_engine=sqlite_view_engine,
+        execution_engine=engine,
         selectable=selectable,
         # create_temp_table defaults to True
     )
-    assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 3
+    assert len(get_sqlite_temp_table_names(sqlite_view_engine)) == 4
+
+    # test schema with execution engine
+    # TODO : Will20210222 Add tests for specifying schema with non-sqlite backend that actually supports new schema creation
+    my_batch_spec = BatchSpec(
+        **{
+            "table_name": "test_table",
+            "partition_definition": {},
+            "schema_name": "main",
+        }
+    )
+    res = engine.get_batch_data_and_markers(batch_spec=my_batch_spec)
+    assert len(res) == 2