Repository: meltano/sdk
Pull request #2301 (2024-03-08 20:19:49)

fix(taps): Unmapped sub-fields in object-type fields are now no longer dropped when the field declares `additionalProperties`
Closes https://github.com/meltano/sdk/issues/2300

<!-- readthedocs-preview meltano-sdk start -->
----
📚 Documentation preview 📚: https://meltano-sdk--2301.org.readthedocs.build/en/2301/

<!-- readthedocs-preview meltano-sdk end -->

codspeed-hq[bot]: ## [CodSpeed Performance Report](https://codspeed.io/meltano/sdk/branches/2300-bug-object-properties-are-dropped-even-when-additionalproperties-is-used)

### Merging #2301 will **not alter performance**

<sub>Comparing <code>2300-bug-object-properties-are-dropped-even-when-additionalproperties-is-used</code> (ccda3e7) with <code>main</code> (26021c1)</sub>



### Summary

`✅ 6` untouched benchmarks





codecov[bot]: ## [Codecov](https://app.codecov.io/gh/meltano/sdk/pull/2301?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 88.76%. Comparing base [(`26021c1`)](https://app.codecov.io/gh/meltano/sdk/commit/26021c1827b6ec6a97279cf31d5348ac0c89385b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) to head [(`ccda3e7`)](https://app.codecov.io/gh/meltano/sdk/pull/2301?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).


<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2301   +/-   ##
=======================================
  Coverage   88.76%   88.76%           
=======================================
  Files          54       54           
  Lines        4769     4769           
  Branches      928      928           
=======================================
  Hits         4233     4233           
  Misses        374      374           
  Partials      162      162           
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/meltano/sdk/pull/2301?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).


Patch:
diff --git a/singer_sdk/helpers/_typing.py b/singer_sdk/helpers/_typing.py
index fdaaecd6..74ed59cf 100644
--- a/singer_sdk/helpers/_typing.py
+++ b/singer_sdk/helpers/_typing.py
@@ -393,7 +393,7 @@ def conform_record_data_types(
 
 
 # TODO: This is in dire need of refactoring. It's a mess.
-def _conform_record_data_types(  # noqa: PLR0912
+def _conform_record_data_types(
     input_object: dict[str, t.Any],
     schema: dict,
     level: TypeConformanceLevel,
@@ -421,30 +421,22 @@ def _conform_record_data_types(  # noqa: PLR0912
     for property_name, elem in input_object.items():
         property_path = property_name if parent is None else f"{parent}.{property_name}"
         if property_name not in schema["properties"]:
+            if schema.get("additionalProperties"):
+                output_object[property_name] = elem
             unmapped_properties.append(property_path)
             continue
 
         property_schema = schema["properties"][property_name]
         if isinstance(elem, list) and is_uniform_list(property_schema):
             if level == TypeConformanceLevel.RECURSIVE:
-                item_schema = property_schema["items"]
-                output = []
-                for item in elem:
-                    if is_object_type(item_schema) and isinstance(item, dict):
-                        (
-                            output_item,
-                            sub_unmapped_properties,
-                        ) = _conform_record_data_types(
-                            item,
-                            item_schema,
-                            level,
-                            property_path,
-                        )
-                        unmapped_properties.extend(sub_unmapped_properties)
-                        output.append(output_item)
-                    else:
-                        output.append(_conform_primitive_property(item, item_schema))
+                output, sub_unmapped_properties = _conform_uniform_list(
+                    elem,
+                    path=property_path,
+                    schema=property_schema,
+                    level=level,
+                )
                 output_object[property_name] = output
+                unmapped_properties.extend(sub_unmapped_properties)
             else:
                 output_object[property_name] = elem
         elif (
@@ -473,6 +465,35 @@ def _conform_record_data_types(  # noqa: PLR0912
     return output_object, unmapped_properties
 
 
+def _conform_uniform_list(
+    element: list,
+    *,
+    path: str,
+    schema: dict,
+    level: TypeConformanceLevel,
+) -> tuple[list, list[str]]:
+    item_schema = schema["items"]
+    unmapped_properties = []
+    output = []
+    for item in element:
+        if is_object_type(item_schema) and isinstance(item, dict):
+            (
+                output_item,
+                sub_unmapped_properties,
+            ) = _conform_record_data_types(
+                item,
+                item_schema,
+                level,
+                path,
+            )
+            unmapped_properties.extend(sub_unmapped_properties)
+            output.append(output_item)
+        else:
+            output.append(_conform_primitive_property(item, item_schema))
+
+    return output, unmapped_properties
+
+
 def _conform_primitive_property(  # noqa: PLR0911
     elem: t.Any,  # noqa: ANN401
     property_schema: dict,


Test patch:
diff --git a/tests/core/test_typing.py b/tests/core/test_typing.py
index 15d56ccc..66182c52 100644
--- a/tests/core/test_typing.py
+++ b/tests/core/test_typing.py
@@ -263,6 +263,27 @@ def test_object_arrays_remove_types(caplog: pytest.LogCaptureFixture):
         )
 
 
+def test_conform_object_additional_properties():
+    schema = PropertiesList(
+        Property(
+            "object",
+            PropertiesList(additional_properties=True),
+        ),
+    ).to_dict()
+
+    record = {"object": {"extra": "value"}}
+    expected_output = {"object": {"extra": "value"}}
+
+    actual_output = conform_record_data_types(
+        "test_stream",
+        record,
+        schema,
+        TypeConformanceLevel.RECURSIVE,
+        logger,
+    )
+    assert actual_output == expected_output
+
+
 def test_conform_primitives():
     assert (
         _conform_primitive_property(