Repository: great-expectations/great_expectations

Issue:
kwargs arg of SqlAlchemyExecutionEngine passed to the SQLAlchemy engine incorrectly
**Describe the bug**

As documented, SqlAlchemyExecutionEngine has to get `kwargs` argument as a dict. Although it gets `kwargs` argument with the `**` operation.

https://github.com/great-expectations/great_expectations/blob/0.17.3/great_expectations/execution_engine/sqlalchemy_execution_engine.py#L282

```python
    class SqlAlchemyExecutionEngine(ExecutionEngine):
    """...
        kwargs (dict): These will be passed as optional parameters to the SQLAlchemy engine, **not** the ExecutionEngine
    """
    def __init__(
        ...
        **kwargs,  # These will be passed as optional parameters to the SQLAlchemy engine, **not** the ExecutionEngine
    ) -> None:
```

So kwargs will be passed to `SqlAlchemyExecutionEngine._setup_engine()` incorrectly, causing an error like:

```python
TypeError: Invalid argument(s) 'kwargs' sent to create_engin(), using configuration TrinoDialect/QueuePool/Engine.
```

**To Reproduce**

Use great-expectations==0.17.3.

```python
import great_expectations as gx
from great_expectations.checkpoint import SimpleCheckpoint

ctx = gx.get_context()
# Add a sql datasource with kwargs
ctx.sources.add_sql(
    name='test_datasource',
    connection_string='trino//:...',
    kwargs={
        'connect_args': {
            'http_scheme': 'https'
        }
    }
)

# Create a checkpoint using the sql datasource
checkpoint = SimpleCheckpoint(...)

# Run the checkpoint
checkpoint.run()

# ... and get an error like 'TypeError: Invalid argument(s) 'kwargs' sent to create_engin(), using configuration TrinoDialect/QueuePool/Engine.'
```

**Expected behavior**
The validation works well.

**Environment (please complete the following information):**
 - Operating System: Linux
 - Great Expectations Version: 0.17.3
 - Datasource: Trino

**Additional context**

Modified SqlAlchemyExecutionEngine as follows and the validation works well.

```diff
git diff
diff --git a/great_expectations/execution_engine/sqlalchemy_execution_engine.py b/great_expectations/execution_engine/sqlalchemy_execution_engine.py
index 4f692c586..61a41d54f 100644
--- a/great_expectations/execution_engine/sqlalchemy_execution_engine.py
+++ b/great_expectations/execution_engine/sqlalchemy_execution_engine.py
@@ -299,7 +299,7 @@ class SqlAlchemyExecutionEngine(ExecutionEngine):
         batch_data_dict: Optional[dict] = None,
         create_temp_table: bool = True,
         concurrency: Optional[ConcurrencyConfig] = None,
-        **kwargs,  # These will be passed as optional parameters to the SQLAlchemy engine, **not** the ExecutionEngine
+        kwargs: Optional[dict] = {},  # These will be passed as optional parameters to the SQLAlchemy engine, **not** the ExecutionEngine
     ) -> None:
         super().__init__(name=name, batch_data_dict=batch_data_dict)
         self._name = name
```


Discussion:
Hi @YoitoFes thank you for filing this issue with us. I've put it into our backlog for internal review.
@YoitoFes :) If you are comfortable with it, please feel free to make the change and add the test case you've provided. If you are not comfortable with it, I would also be happy to either pair with you or take up this work.

Resolving patch:
diff --git a/great_expectations/datasource/fluent/sql_datasource.py b/great_expectations/datasource/fluent/sql_datasource.py
--- a/great_expectations/datasource/fluent/sql_datasource.py
+++ b/great_expectations/datasource/fluent/sql_datasource.py
@@ -1007,6 +1007,26 @@ def _create_engine(self) -> sqlalchemy.Engine:
         kwargs = model_dict.pop("kwargs", {})
         return sa.create_engine(connection_string, **kwargs)
 
+    def get_execution_engine(self) -> SqlAlchemyExecutionEngine:
+        # Overrides get_execution_engine in Datasource
+        # because we need to pass the kwargs as keyvalue args to the execution engine
+        # when then passes them to the engine.
+        current_execution_engine_kwargs = self.dict(
+            exclude=self._get_exec_engine_excludes(),
+            config_provider=self._config_provider,
+        )
+        if (
+            current_execution_engine_kwargs != self._cached_execution_engine_kwargs
+            or not self._execution_engine
+        ):
+            self._cached_execution_engine_kwargs = current_execution_engine_kwargs
+            engine_kwargs = current_execution_engine_kwargs.pop("kwargs", {})
+            self._execution_engine = self._execution_engine_type()(
+                **current_execution_engine_kwargs,
+                **engine_kwargs,
+            )
+        return self._execution_engine
+
     def test_connection(self, test_assets: bool = True) -> None:
         """Test the connection for the SQLDatasource.
 
diff --git a/great_expectations/execution_engine/sqlalchemy_execution_engine.py b/great_expectations/execution_engine/sqlalchemy_execution_engine.py
--- a/great_expectations/execution_engine/sqlalchemy_execution_engine.py
+++ b/great_expectations/execution_engine/sqlalchemy_execution_engine.py
@@ -304,7 +304,8 @@ def __init__(  # noqa: C901, PLR0912, PLR0913, PLR0915
         batch_data_dict: Optional[dict] = None,
         create_temp_table: bool = True,
         concurrency: Optional[ConcurrencyConfig] = None,
-        **kwargs,  # These will be passed as optional parameters to the SQLAlchemy engine, **not** the ExecutionEngine
+        # kwargs will be passed as optional parameters to the SQLAlchemy engine, **not** the ExecutionEngine
+        **kwargs,
     ) -> None:
         super().__init__(name=name, batch_data_dict=batch_data_dict)
         self._name = name


Test patch:
diff --git a/tests/datasource/fluent/test_postgres_datasource.py b/tests/datasource/fluent/test_postgres_datasource.py
--- a/tests/datasource/fluent/test_postgres_datasource.py
+++ b/tests/datasource/fluent/test_postgres_datasource.py
@@ -123,7 +123,7 @@ def _mock_test_connection(self: PostgresDatasource) -> bool:
     monkeypatch.setattr(PostgresDatasource, "test_connection", _mock_test_connection)
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_construct_postgres_datasource(create_source: CreateSourceFixture):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -154,7 +154,7 @@ def assert_batch_request(
     assert batch_request.options == options
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_add_table_asset_with_splitter(mocker, create_source: CreateSourceFixture):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -191,7 +191,7 @@ def test_add_table_asset_with_splitter(mocker, create_source: CreateSourceFixtur
         )
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_add_table_asset_with_no_splitter(mocker, create_source: CreateSourceFixture):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -224,7 +224,7 @@ def test_add_table_asset_with_no_splitter(mocker, create_source: CreateSourceFix
         )
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_construct_table_asset_directly_with_no_splitter(create_source):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -260,7 +260,7 @@ def year_month_splitter(column_name: str) -> SplitterYearAndMonth:
     )
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_construct_table_asset_directly_with_splitter(create_source):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -290,7 +290,7 @@ def test_construct_table_asset_directly_with_splitter(create_source):
         )
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_datasource_gets_batch_list_no_splitter(empty_data_context, create_source):
     def validate_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
         assert spec == {
@@ -343,7 +343,7 @@ def assert_batches_correct_with_year_month_splitter_defaults(batches):
             assert {"year": year, "month": month} in metadatas
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_datasource_gets_batch_list_splitter_with_unspecified_batch_request_options(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -372,7 +372,7 @@ def collect_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
         assert_batches_correct_with_year_month_splitter_defaults(batches)
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_datasource_gets_batch_list_splitter_with_batch_request_options_set_to_none(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -405,7 +405,7 @@ def collect_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
         assert_batches_correct_with_year_month_splitter_defaults(batches)
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_datasource_gets_batch_list_splitter_with_partially_specified_batch_request_options(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -454,7 +454,7 @@ def collect_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
             assert expected_metadata in metadatas
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_datasource_gets_batch_list_with_fully_specified_batch_request_options(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -493,7 +493,7 @@ def validate_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
         assert batches[0].metadata == {"month": month, "year": year}
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_datasource_gets_nonexistent_asset(create_source: CreateSourceFixture):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -502,7 +502,7 @@ def test_datasource_gets_nonexistent_asset(create_source: CreateSourceFixture):
             source.get_asset("my_asset")
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     [
         "add_splitter",
@@ -575,7 +575,7 @@ def test_bad_batch_request_passed_into_get_batch_list_from_batch_request(
             source.get_batch_list_from_batch_request(batch_request)
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     "batch_request_options",
     [{}, {"year": 2021}, {"year": 2021, "month": 10}, {"year": None, "month": 10}],
@@ -606,7 +606,7 @@ def test_get_batch_list_from_batch_request_with_good_batch_request(
         asset.get_batch_list_from_batch_request(batch_request)
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     "batch_request_args",
     [
@@ -639,7 +639,7 @@ def test_get_batch_list_from_batch_request_with_malformed_batch_request(
             asset.get_batch_list_from_batch_request(batch_request)
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_get_bad_batch_request(create_source: CreateSourceFixture):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -655,7 +655,7 @@ def test_get_bad_batch_request(create_source: CreateSourceFixture):
             asset.build_batch_request({"invalid_key": None})
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     "sort_info",
     # Sort info is a list where the first element is the sort keys with an optional prefix and
@@ -747,7 +747,7 @@ def test_sort_batch_list_by_metadata(
             assert batch.metadata["month"] == expected_order[i]["month"]
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_sort_batch_list_by_unknown_key(
     empty_data_context, create_source: CreateSourceFixture
 ):
@@ -773,7 +773,7 @@ def test_sort_batch_list_by_unknown_key(
             source.get_batch_list_from_batch_request(batch_request)
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     "order_by",
     [
@@ -797,7 +797,7 @@ def test_table_asset_sorter_parsing(order_by: list):
     assert table_asset.order_by == expected_sorters
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     "batch_slice,expected_batch_count",
     [
@@ -841,7 +841,7 @@ def test_postgres_slice_batch_count(
         assert len(batches) == expected_batch_count
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_data_source_json_has_properties(create_source: CreateSourceFixture):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -861,7 +861,7 @@ def test_data_source_json_has_properties(create_source: CreateSourceFixture):
         assert f'"type": "{asset.type}"'  # noqa: PLW0129
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_data_source_yaml_has_properties(create_source: CreateSourceFixture):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -880,7 +880,7 @@ def test_data_source_yaml_has_properties(create_source: CreateSourceFixture):
         assert f"type: {asset.type}" in source_str
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_datasource_dict_has_properties(create_source):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -916,7 +916,7 @@ def test_datasource_dict_has_properties(create_source):
         )
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     "connection_string",
     [
@@ -942,7 +942,7 @@ def test_validate_valid_postgres_connection_string(
         pass
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     "connection_string",
     [
@@ -1026,7 +1026,7 @@ def bad_configuration_datasource(
     )
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_test_connection_failures(
     mocker,
     bad_configuration_datasource: PostgresDatasource,
@@ -1076,7 +1076,7 @@ def validate_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
         source.get_batch_list_from_batch_request(asset.build_batch_request())
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_non_select_query_data_asset(create_source):
     with create_source(
         validate_batch_spec=lambda _: None, dialect="postgresql"
@@ -1111,7 +1111,7 @@ def test_adding_splitter_persists_results(
     assert final_yaml["my_datasource"]["assets"]["my_asset"]["splitter"]
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_splitter_year(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -1146,7 +1146,7 @@ def collect_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
             assert spec["splitter_method"] == "split_on_year"
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_splitter_year_and_month(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -1188,7 +1188,7 @@ def collect_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
             assert spec["splitter_method"] == "split_on_year_and_month"
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_splitter_year_and_month_and_day(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -1239,7 +1239,7 @@ def collect_batch_spec(spec: SqlAlchemyDatasourceBatchSpec) -> None:
             assert spec["splitter_method"] == "split_on_year_and_month_and_day"
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 @pytest.mark.parametrize(
     [
         "add_splitter_method",
@@ -1387,7 +1387,7 @@ def test_splitter(
         assert specified_batches[-1].metadata == last_specified_batch_metadata
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_sorting_none_in_metadata(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -1411,7 +1411,7 @@ def test_sorting_none_in_metadata(
         assert batches[-1].metadata["year"] is None
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_create_temp_table(empty_data_context, create_source):
     with create_source(
         validate_batch_spec=lambda _: None,
@@ -1425,7 +1425,7 @@ def test_create_temp_table(empty_data_context, create_source):
         assert source._execution_engine._create_temp_table is False
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_add_postgres_query_asset_with_batch_metadata(
     empty_data_context,
     create_source: CreateSourceFixture,
@@ -1470,7 +1470,7 @@ def test_add_postgres_query_asset_with_batch_metadata(
             assert batches[i].metadata == substituted_batch_metadata
 
 
-@pytest.mark.unit
+@pytest.mark.postgresql
 def test_add_postgres_table_asset_with_batch_metadata(
     empty_data_context, create_source: CreateSourceFixture, monkeypatch
 ):
diff --git a/tests/execution_engine/test_sqlalchemy_execution_engine.py b/tests/execution_engine/test_sqlalchemy_execution_engine.py
--- a/tests/execution_engine/test_sqlalchemy_execution_engine.py
+++ b/tests/execution_engine/test_sqlalchemy_execution_engine.py
@@ -1,6 +1,7 @@
 import logging
 import os
 from typing import Dict, Tuple, cast
+from unittest import mock
 
 import pandas as pd
 import pytest
@@ -45,6 +46,7 @@
     get_sqlite_table_names,
     get_sqlite_temp_table_names,
     get_sqlite_temp_table_names_from_engine,
+    load_data_into_test_database,
 )
 
 try:
@@ -100,6 +102,138 @@ def test_instantiation_via_url(sa):
     )
 
 
+@pytest.mark.sqlite
+def test_instantiation_via_url_with_invalid_kwargs(sa):
+    db_file = file_relative_path(
+        __file__,
+        os.path.join(  # noqa: PTH118
+            "..", "test_sets", "test_cases_for_sql_data_connector.db"
+        ),
+    )
+    with pytest.raises(TypeError):
+        _ = SqlAlchemyExecutionEngine(
+            url="sqlite:///" + db_file,
+            connect_args={"invalid_keyword_argument": ""},
+        )
+
+
+@pytest.mark.sqlite
+def test_instantiation_via_url_with_kwargs(sa):
+    db_file = file_relative_path(
+        __file__,
+        os.path.join(  # noqa: PTH118
+            "..", "test_sets", "test_cases_for_sql_data_connector.db"
+        ),
+    )
+    my_execution_engine = SqlAlchemyExecutionEngine(
+        url="sqlite:///" + db_file, connect_args={"timeout": 10}
+    )
+    assert my_execution_engine.connection_string is None
+    assert my_execution_engine.credentials is None
+    assert my_execution_engine.url[-36:] == "test_cases_for_sql_data_connector.db"
+
+    my_execution_engine.get_batch_data_and_markers(
+        batch_spec=SqlAlchemyDatasourceBatchSpec(
+            table_name="table_partitioned_by_date_column__A",
+            sampling_method="_sample_using_limit",
+            sampling_kwargs={"n": 5},
+        )
+    )
+
+
+@pytest.mark.sqlite
+def test_instantiation_via_fluent_data_sources_with_kwargs(sa, empty_data_context):
+    db_file = file_relative_path(
+        __file__,
+        os.path.join(  # noqa: PTH118
+            "..", "test_sets", "test_cases_for_sql_data_connector.db"
+        ),
+    )
+    connection_string = "sqlite:///" + db_file
+    context = empty_data_context
+
+    datasource = context.sources.add_sql(
+        name="test_datasource",
+        connection_string=connection_string,
+        kwargs={"connect_args": {"check_same_thread": False}},
+    )
+
+    engine = datasource.get_engine()
+    assert engine
+    assert engine.dialect.name == "sqlite"
+
+    execution_engine = datasource.get_execution_engine()
+    assert execution_engine.connection_string == connection_string
+    # kwargs should be passed through as keyword arguments to create_engine
+    assert execution_engine.config["connect_args"] == {"check_same_thread": False}
+    assert execution_engine.config["class_name"] == "SqlAlchemyExecutionEngine"
+    assert execution_engine.config["connection_string"] == connection_string
+    assert execution_engine.engine
+
+
+@pytest.mark.trino
+def test_instantiation_via_fluent_data_source__trino_add_sql(sa, empty_data_context):
+    context = empty_data_context
+
+    CONNECTION_STRING = "trino://test@localhost:8088/memory/schema"
+
+    # This utility is not for general use. It is only to support testing.
+    load_data_into_test_database(
+        table_name="taxi_data",
+        csv_path=file_relative_path(
+            __file__,
+            "../test_sets/taxi_yellow_tripdata_samples/yellow_tripdata_sample_2019-01.csv",
+        ),
+        connection_string=CONNECTION_STRING,
+        convert_colnames_to_datetime=["pickup_datetime"],
+    )
+
+    with mock.patch(
+        "great_expectations.datasource.fluent.sql_datasource.sa.create_engine",
+        wraps=sa.create_engine,
+    ) as mock_create_engine:
+        datasource = context.sources.add_sql(
+            name="test_datasource",
+            connection_string=CONNECTION_STRING,
+            kwargs={"connect_args": {"http_scheme": "http"}},
+        )
+        assert mock_create_engine.call_args.args == (CONNECTION_STRING,)
+        assert mock_create_engine.call_args.kwargs == {
+            "connect_args": {"http_scheme": "http"}
+        }
+
+    datasource.add_table_asset("taxi_data")
+
+    # use a validator to create an expectation suite
+    validator = context.get_validator(
+        datasource_name="test_datasource", data_asset_name="taxi_data"
+    )
+    validator.expect_column_values_to_not_be_null("pickup_datetime")
+    context.add_expectation_suite("yellow_tripdata_suite")
+
+    # create a checkpoint
+    checkpoint = context.add_or_update_checkpoint(
+        name="my_checkpoint",
+        expectation_suite_name="yellow_tripdata_suite",
+    )
+
+    # add (save) the checkpoint to the data context
+    context.add_checkpoint(checkpoint=checkpoint)
+    cp = context.get_checkpoint(name="my_checkpoint")
+    assert cp.name == "my_checkpoint"
+
+    result = context.run_checkpoint(
+        checkpoint_name="my_checkpoint",
+        batch_request={
+            "datasource_name": "test_datasource",
+            "data_asset_name": "taxi_data",
+        },
+        run_name=None,
+    )
+
+    assert result["success"]
+
+
 @pytest.mark.sqlite
 def test_instantiation_via_url_and_retrieve_data_with_other_dialect(sa):
     """Ensure that we can still retrieve data when the dialect is not recognized."""