Repository: great-expectations/great_expectations

Issue:
Using a postgres expectation store fails in v0.13
**Describe the bug**
I can't get a postgres expectation store working in v0.13 -- I'm not sure if the format changed or if this is a bug, so I wanted to flag it here.

Existing docs for reference [here](https://docs.greatexpectations.io/en/latest/guides/how_to_guides/configuring_metadata_stores/how_to_configure_an_expectation_store_to_postgresql.html).

**To Reproduce**
Steps to reproduce the behavior:
1. Spin up a postgres instance (e.g., with docker as below)
```bash
docker run \
    -e POSTGRES_USER=great \
    -e POSTGRES_PASSWORD=expectations \
    -e POSTGRES_DB=ge \
    -p "5432:5432" \
    postgres:13.1
```
2. Use the following snippet in your `great_expectations.yml`
```yaml
stores:
  postgres_expectations_store:
    class_name: ExpectationsStore
    store_backend: 
      class_name: DatabaseStoreBackend
      credentials:
        drivername: postgresql
        host: localhost
        port: 5432
        username: great
        password: expectations
        database: ge
expectations_store_name: postgres_expectations_store
```
3. Try to list the stores
```bash
great_expectations store list
```
4. See an error similar to the one below
```
➜  great_expectations store list                                                                                                                                                                              [130 01:40:51P Wed 12/16/20]
Traceback (most recent call last):
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 84, in instantiate_class_from_config
    class_instance = class_(**config_with_defaults)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/database_store_backend.py", line 76, in __init__
    self.engine = self._build_engine(credentials=credentials, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/database_store_backend.py", line 164, in _build_engine
    engine = sa.create_engine(options, **create_engine_kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine
    return strategy.create(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 164, in create
    raise TypeError(
TypeError: Invalid argument(s) 'manually_initialize_store_backend_id' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 84, in instantiate_class_from_config
    class_instance = class_(**config_with_defaults)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/expectations_store.py", line 133, in __init__
    super().__init__(
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/store/store.py", line 42, in __init__
    self._store_backend = instantiate_class_from_config(
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 86, in instantiate_class_from_config
    raise TypeError(
TypeError: Couldn't instantiate class : DatabaseStoreBackend with config :
	store_name		postgres_expectations_store
	credentials		{'host': 'localhost', 'port': 5432, 'username': 'great', 'password': 'expectations', 'database': 'ge'}
	manually_initialize_store_backend_id		91f04917-fe8e-49b7-bea1-db671ebfeb73
	table_name		ge_expectations_store
	key_columns		['expectation_suite_name']

Invalid argument(s) 'manually_initialize_store_backend_id' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ryan/.virtualenvs/ge-13/bin/great_expectations", line 8, in <module>
    sys.exit(main())
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/cli/cli.py", line 72, in main
    cli()
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/cli/store.py", line 23, in store_list
    context = toolkit.load_data_context_with_error_handling(directory)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/cli/toolkit.py", line 404, in load_data_context_with_error_handling
    context = DataContext(directory)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 3106, in __init__
    super().__init__(project_config, context_root_directory, runtime_environment)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 261, in usage_statistics_wrapped_method
    result = func(*args, **kwargs)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 267, in __init__
    self._init_stores(self._project_config_with_variables_substituted.stores)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 354, in _init_stores
    self._build_store_from_config(store_name, store_config)
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/data_context.py", line 324, in _build_store_from_config
    new_store = instantiate_class_from_config(
  File "/Users/ryan/.virtualenvs/ge-13/lib/python3.8/site-packages/great_expectations/data_context/util.py", line 86, in instantiate_class_from_config
    raise TypeError(
TypeError: Couldn't instantiate class : ExpectationsStore with config :
	store_name		postgres_expectations_store
	store_backend		{'class_name': 'DatabaseStoreBackend', 'credentials': {'drivername': 'postgresql', 'host': 'localhost', 'port': 5432, 'username': 'great', 'password': 'expectations', 'database': 'ge'}, 'manually_initialize_store_backend_id': '91f04917-fe8e-49b7-bea1-db671ebfeb73', 'table_name': 'ge_expectations_store', 'key_columns': ['expectation_suite_name']}
	runtime_environment		{'root_directory': '/Users/ryan/local/scratch/great-expectations/ge13/great_expectations'}

Couldn't instantiate class : DatabaseStoreBackend with config :
	store_name		postgres_expectations_store
	credentials		{'host': 'localhost', 'port': 5432, 'username': 'great', 'password': 'expectations', 'database': 'ge'}
	manually_initialize_store_backend_id		91f04917-fe8e-49b7-bea1-db671ebfeb73
	table_name		ge_expectations_store
	key_columns		['expectation_suite_name']

Invalid argument(s) 'manually_initialize_store_backend_id' sent to create_engine(), using configuration PGDialect_psycopg2/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.
```

**Expected behavior**
The config should be properly processed and will print out something like the output below
```yaml
➜  great_expectations store list                                                                                                                                                                   [1 01:43:30P Wed 12/16/20]
1 Store found:

 - name: postgres_expectations_store
   class_name: ExpectationsStore
   store_backend:
     class_name: DatabaseStoreBackend
     credentials:
       database: ge
       drivername: postgresql
       host: localhost
       password: ******
       port: 5432
       username: great
```

**Environment (please complete the following information):**
 - OS: macOS v10.15.4
 - GE Version: v0.13.3

**Additional context**
* This configuration works just fine with v0.12.7
* A nearly identical configuration seems to work for _validation_ stores without issue (on v0.13.3) -- this issue seems specific to the `ExpectationsStore` class
* From the error message, it seems like we're propagating an unexpected parameter into the sqlalchemy engine. Not sure how to fix it, though! 😅



Discussion:
@ryanaustincarlson Thank you posting this issue. I will reproduce it and we will take care of it.
@eugmandel thanks! let me know if I can be helpful - if you have trouble reproducing, I'm happy to help debug what might be weird about my setup!
@ryanaustincarlson Reproduced - huge thanks for such a detailed a precise report! I think I know what happened there - we will  work on a fix.

Resolving patch:
diff --git a/great_expectations/data_context/store/database_store_backend.py b/great_expectations/data_context/store/database_store_backend.py
--- a/great_expectations/data_context/store/database_store_backend.py
+++ b/great_expectations/data_context/store/database_store_backend.py
@@ -43,11 +43,13 @@ def __init__(
         engine=None,
         store_name=None,
         suppress_store_backend_id=False,
+        manually_initialize_store_backend_id: str = "",
         **kwargs,
     ):
         super().__init__(
             fixed_length_key=fixed_length_key,
             suppress_store_backend_id=suppress_store_backend_id,
+            manually_initialize_store_backend_id=manually_initialize_store_backend_id,
             store_name=store_name,
         )
         if not sa:


Test patch:
diff --git a/tests/data_context/store/test_database_store_backend.py b/tests/data_context/store/test_database_store_backend.py
--- a/tests/data_context/store/test_database_store_backend.py
+++ b/tests/data_context/store/test_database_store_backend.py
@@ -5,6 +5,7 @@
 
 import tests.test_utils as test_utils
 from great_expectations.data_context.store import DatabaseStoreBackend
+from great_expectations.data_context.util import instantiate_class_from_config
 from great_expectations.exceptions import StoreBackendError
 
 
@@ -148,6 +149,8 @@ def test_database_store_backend_id_initialization(caplog, sa, test_backends):
     and persisted. The store_backend_id should be a valid UUIDv4
     If a new store_backend_id cannot be persisted, use an ephemeral store_backend_id.
     Persistence should be in a .ge_store_id file for for filesystem and blob-stores.
+    If an existing data_context_id is available in the great_expectations.yml, use this as the expectation_store id.
+    If a store_backend_id is provided via manually_initialize_store_backend_id, make sure it is retrievable.
 
     Note: StoreBackend & TupleStoreBackend are abstract classes, so we will test the
     concrete classes that inherit from them.
@@ -174,3 +177,36 @@ def test_database_store_backend_id_initialization(caplog, sa, test_backends):
     assert store_backend.store_backend_id is not None
     # Check that store_backend_id is a valid UUID
     assert test_utils.validate_uuid4(store_backend.store_backend_id)
+
+    # Test that expectations store can be created with specified store_backend_id
+
+    expectations_store_with_database_backend = instantiate_class_from_config(
+        config={
+            "class_name": "ExpectationsStore",
+            "store_backend": {
+                "class_name": "DatabaseStoreBackend",
+                "credentials": {
+                    "drivername": "postgresql",
+                    "username": "postgres",
+                    "password": "",
+                    "host": "localhost",
+                    "port": "5432",
+                    "database": "test_ci",
+                },
+                "manually_initialize_store_backend_id": "00000000-0000-0000-0000-000000aaaaaa",
+                "table_name": "ge_expectations_store",
+                "key_columns": {"expectation_suite_name"},
+            },
+            # "name": "postgres_expectations_store",
+        },
+        runtime_environment=None,
+        config_defaults={
+            "module_name": "great_expectations.data_context.store",
+            "store_name": "postgres_expectations_store",
+        },
+    )
+
+    assert (
+        expectations_store_with_database_backend.store_backend_id
+        == "00000000-0000-0000-0000-000000aaaaaa"
+    )