Repository: meltano/sdk
Pull request #1949 (2023-09-11 14:56:30)

fix: Retry SQLAlchemy engine creation for adapters without JSON SerDe support


<!-- readthedocs-preview meltano-sdk start -->
----
:books: Documentation preview :books:: https://meltano-sdk--1949.org.readthedocs.build/en/1949/

<!-- readthedocs-preview meltano-sdk end -->

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/meltano/sdk/pull/1949?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) Report
> Merging [#1949](https://app.codecov.io/gh/meltano/sdk/pull/1949?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) (af0a5e5) into [main](https://app.codecov.io/gh/meltano/sdk/commit/99e44464c16adc91edeb2a0d31580672fafa862a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) (99e4446) will **decrease** coverage by `0.05%`.
> Report is 2 commits behind head on main.
> The diff coverage is `40.00%`.

```diff
@@            Coverage Diff             @@
##             main    #1949      +/-   ##
==========================================
- Coverage   86.97%   86.92%   -0.05%     
==========================================
  Files          59       59              
  Lines        5091     5095       +4     
  Branches      825      825              
==========================================
+ Hits         4428     4429       +1     
- Misses        466      469       +3     
  Partials      197      197              
```


| [Files Changed](https://app.codecov.io/gh/meltano/sdk/pull/1949?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) | Coverage Δ | |
|---|---|---|
| [singer\_sdk/connectors/sql.py](https://app.codecov.io/gh/meltano/sdk/pull/1949?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano#diff-c2luZ2VyX3Nkay9jb25uZWN0b3JzL3NxbC5weQ==) | `81.14% <40.00%> (-0.77%)` | :arrow_down: |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano)


Patch:
diff --git a/samples/sample_custom_sql_adapter/__init__.py b/samples/sample_custom_sql_adapter/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/samples/sample_custom_sql_adapter/connector.py b/samples/sample_custom_sql_adapter/connector.py
new file mode 100644
index 00000000..6f7745a7
--- /dev/null
+++ b/samples/sample_custom_sql_adapter/connector.py
@@ -0,0 +1,32 @@
+from __future__ import annotations
+
+import typing as t
+
+from sqlalchemy.engine.default import DefaultDialect
+
+if t.TYPE_CHECKING:
+    from types import ModuleType
+
+
+class CustomSQLDialect(DefaultDialect):
+    """Custom SQLite dialect that supports JSON."""
+
+    name = "myrdbms"
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+    @classmethod
+    def import_dbapi(cls):
+        """Import the sqlite3 DBAPI."""
+        import sqlite3
+
+        return sqlite3
+
+    @classmethod
+    def dbapi(cls) -> ModuleType:  # type: ignore[override]
+        """Return the DBAPI module.
+
+        NOTE: This is a legacy method that will stop being used by SQLAlchemy at some point.
+        """  # noqa: E501
+        return cls.import_dbapi()
diff --git a/singer_sdk/connectors/sql.py b/singer_sdk/connectors/sql.py
index e05e359d..fb49b358 100644
--- a/singer_sdk/connectors/sql.py
+++ b/singer_sdk/connectors/sql.py
@@ -319,12 +319,21 @@ class SQLConnector:
         Returns:
             A new SQLAlchemy Engine.
         """
-        return sqlalchemy.create_engine(
-            self.sqlalchemy_url,
-            echo=False,
-            json_serializer=self.serialize_json,
-            json_deserializer=self.deserialize_json,
-        )
+        try:
+            return sqlalchemy.create_engine(
+                self.sqlalchemy_url,
+                echo=False,
+                json_serializer=self.serialize_json,
+                json_deserializer=self.deserialize_json,
+            )
+        except TypeError:
+            self.logger.exception(
+                "Retrying engine creation with fewer arguments due to TypeError.",
+            )
+            return sqlalchemy.create_engine(
+                self.sqlalchemy_url,
+                echo=False,
+            )
 
     def quote(self, name: str) -> str:
         """Quote a name if it needs quoting, using '.' as a name-part delimiter.


Test patch:
diff --git a/tests/core/test_connector_sql.py b/tests/core/test_connector_sql.py
index 58ba59ec..101d2d39 100644
--- a/tests/core/test_connector_sql.py
+++ b/tests/core/test_connector_sql.py
@@ -1,15 +1,19 @@
 from __future__ import annotations
 
+import typing as t
 from decimal import Decimal
 from unittest import mock
 
 import pytest
 import sqlalchemy
-from sqlalchemy.dialects import sqlite
+from sqlalchemy.dialects import registry, sqlite
 
 from singer_sdk.connectors import SQLConnector
 from singer_sdk.exceptions import ConfigValidationError
 
+if t.TYPE_CHECKING:
+    from sqlalchemy.engine import Engine
+
 
 def stringify(in_dict):
     return {k: str(v) for k, v in in_dict.items()}
@@ -283,3 +287,18 @@ class TestConnectorSQL:
                 (1, {"x": Decimal("1.0")}),
                 (2, {"x": Decimal("2.0"), "y": [1, 2, 3]}),
             ]
+
+
+def test_adapter_without_json_serde():
+    registry.register(
+        "myrdbms",
+        "samples.sample_custom_sql_adapter.connector",
+        "CustomSQLDialect",
+    )
+
+    class CustomConnector(SQLConnector):
+        def create_engine(self) -> Engine:
+            return super().create_engine()
+
+    connector = CustomConnector(config={"sqlalchemy_url": "myrdbms:///"})
+    connector.create_engine()