Repository: meltano/sdk
Pull request #1844 (2023-07-14 22:29:35)

fix(targets): Safely skip parsing record field as date-time if it is missing in schema
Closes https://github.com/meltano/sdk/issues/1836

* [x] Add tests


<!-- readthedocs-preview meltano-sdk start -->
----
:books: Documentation preview :books:: https://meltano-sdk--1844.org.readthedocs.build/en/1844/

<!-- readthedocs-preview meltano-sdk end -->

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/meltano/sdk/pull/1844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) Report
> Merging [#1844](https://app.codecov.io/gh/meltano/sdk/pull/1844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) (3ed9f0d) into [main](https://app.codecov.io/gh/meltano/sdk/commit/98fdb2e981a9a582c456671006bffce7ef670c6e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) (98fdb2e) will **decrease** coverage by `0.05%`.
> The diff coverage is `37.50%`.

```diff
@@            Coverage Diff             @@
##             main    #1844      +/-   ##
==========================================
- Coverage   86.56%   86.51%   -0.05%     
==========================================
  Files          59       59              
  Lines        5000     5004       +4     
  Branches      811      811              
==========================================
+ Hits         4328     4329       +1     
- Misses        476      479       +3     
  Partials      196      196              
```


| [Impacted Files](https://app.codecov.io/gh/meltano/sdk/pull/1844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) | Coverage Δ | |
|---|---|---|
| [singer\_sdk/sinks/core.py](https://app.codecov.io/gh/meltano/sdk/pull/1844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano#diff-c2luZ2VyX3Nkay9zaW5rcy9jb3JlLnB5) | `83.43% <37.50%> (-1.48%)` | :arrow_down: |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano)


Patch:
diff --git a/singer_sdk/sinks/core.py b/singer_sdk/sinks/core.py
index 19dfbc31..e578c208 100644
--- a/singer_sdk/sinks/core.py
+++ b/singer_sdk/sinks/core.py
@@ -365,12 +365,15 @@ class Sink(metaclass=abc.ABCMeta):
             schema: TODO
             treatment: TODO
         """
-        for key in record:
+        for key, value in record.items():
+            if key not in schema["properties"]:
+                self.logger.warning("No schema for record field '%s'", key)
+                continue
             datelike_type = get_datelike_property_type(schema["properties"][key])
             if datelike_type:
-                date_val = record[key]
+                date_val = value
                 try:
-                    if record[key] is not None:
+                    if value is not None:
                         date_val = parser.parse(date_val)
                 except parser.ParserError as ex:
                     date_val = handle_invalid_timestamp_in_record(


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 7e7c3995..cb392bd3 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -12,6 +12,7 @@ from sqlalchemy import __version__ as sqlalchemy_version
 
 from singer_sdk import SQLConnector
 from singer_sdk import typing as th
+from singer_sdk.helpers._typing import DatetimeErrorTreatmentEnum
 from singer_sdk.helpers.capabilities import PluginCapabilities
 from singer_sdk.sinks import BatchSink, SQLSink
 from singer_sdk.target_base import SQLTarget, Target
@@ -75,6 +76,7 @@ class BatchSinkMock(BatchSink):
     """A mock Sink class."""
 
     name = "batch-sink-mock"
+    datetime_error_treatment = DatetimeErrorTreatmentEnum.MAX
 
     def __init__(
         self,
diff --git a/tests/core/sinks/test_validation.py b/tests/core/sinks/test_validation.py
new file mode 100644
index 00000000..5a7ca39a
--- /dev/null
+++ b/tests/core/sinks/test_validation.py
@@ -0,0 +1,41 @@
+from __future__ import annotations
+
+import datetime
+
+from tests.conftest import BatchSinkMock, TargetMock
+
+
+def test_validate_record():
+    target = TargetMock()
+    sink = BatchSinkMock(
+        target,
+        "users",
+        {
+            "type": "object",
+            "properties": {
+                "id": {"type": "integer"},
+                "created_at": {"type": "string", "format": "date-time"},
+                "invalid_datetime": {"type": "string", "format": "date-time"},
+            },
+        },
+        ["id"],
+    )
+
+    record = {
+        "id": 1,
+        "created_at": "2021-01-01T00:00:00+00:00",
+        "missing_datetime": "2021-01-01T00:00:00+00:00",
+        "invalid_datetime": "not a datetime",
+    }
+    updated_record = sink._validate_and_parse(record)
+
+    assert updated_record["created_at"] == datetime.datetime(
+        2021,
+        1,
+        1,
+        0,
+        0,
+        tzinfo=datetime.timezone.utc,
+    )
+    assert updated_record["missing_datetime"] == "2021-01-01T00:00:00+00:00"
+    assert updated_record["invalid_datetime"] == "9999-12-31 23:59:59.999999"