Repository: meltano/sdk
Pull request #2243 (2024-02-15 16:01:45)

fix: Force flattened record according to provided flattened schema
If a flattened schema is provided, and there is a dynamic object field, the flattened record can ignore the provided schema. This means that the resulting output may have more fields than what was originally specified in the provided schema. This can cause problems if the loader does not accept fields that are not listed in the schema.

So this PR is checking if a flattened is provided and forcing the flattened record to use it during the flatten record call.

Example:
```
RECORD:

{
    'key_1': 1,
    'key_2': {
        'key_3': 'value',
        'key_4': {
            'key_5': 1,
            'key_6': ['a', 'b']
        }
    }
}

SCHEMA:
{
    'key_1': {'type': ['null', 'integer']},
    'key_2__key_3': {'type': ['null', 'string']},
    'key_2__key_4': {'type': ['null', 'object']},
}

WRONG RESULT:
{
    'key_1': 1,
    'key_2__key_3': 'value',
    'key_2__key_4__key_5': 1,
    'key_2__key_4__key_6': '["a", "b"]'
}

EXPECTED RESULT:
{
    'key_1': 1, 
    'key_2__key_3': 'value', 
    'key_2__key_4': '{"key_5": 1, "key_6": ["a", "b"]}'
}
```

<!-- readthedocs-preview meltano-sdk start -->
----
📚 Documentation preview 📚: https://meltano-sdk--2243.org.readthedocs.build/en/2243/

<!-- readthedocs-preview meltano-sdk end -->

codspeed-hq[bot]: ## [CodSpeed Performance Report](https://codspeed.io/meltano/sdk/branches/joaopamaral:fix/flatten_record_generating_non_expected_fields)

### Merging #2243 will **not alter performance**

<sub>Comparing <code>joaopamaral:fix/flatten_record_generating_non_expected_fields</code> (0d94565) with <code>main</code> (5734e69)</sub>



### Summary

`✅ 6` untouched benchmarks





codecov[bot]: ## [Codecov](https://app.codecov.io/gh/meltano/sdk/pull/2243?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`5734e69`)](https://app.codecov.io/gh/meltano/sdk/commit/5734e694994562830a5f7d4894910f2a41ec5aac?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) 88.47% compared to head [(`dbdd011`)](https://app.codecov.io/gh/meltano/sdk/pull/2243?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) 88.48%.


<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2243   +/-   ##
=======================================
  Coverage   88.47%   88.48%           
=======================================
  Files          54       54           
  Lines        4721     4723    +2     
  Branches      919      919           
=======================================
+ Hits         4177     4179    +2     
  Misses        383      383           
  Partials      161      161           
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/meltano/sdk/pull/2243?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).


Patch:
diff --git a/singer_sdk/helpers/_flattening.py b/singer_sdk/helpers/_flattening.py
index 77e3935b..2a3e194d 100644
--- a/singer_sdk/helpers/_flattening.py
+++ b/singer_sdk/helpers/_flattening.py
@@ -415,7 +415,14 @@ def _flatten_record(
     items: list[tuple[str, t.Any]] = []
     for k, v in record_node.items():
         new_key = flatten_key(k, parent_key, separator)
-        if isinstance(v, collections.abc.MutableMapping) and level < max_level:
+        # If the value is a dictionary, and the key is not in the schema, and the
+        # level is less than the max level, then we should continue to flatten.
+        if (
+            isinstance(v, collections.abc.MutableMapping)
+            and flattened_schema
+            and new_key not in flattened_schema.get("properties", {})
+            and (level < max_level)
+        ):
             items.extend(
                 _flatten_record(
                     v,


Test patch:
diff --git a/tests/core/test_flattening.py b/tests/core/test_flattening.py
new file mode 100644
index 00000000..73169eab
--- /dev/null
+++ b/tests/core/test_flattening.py
@@ -0,0 +1,74 @@
+from __future__ import annotations
+
+import pytest
+
+from singer_sdk.helpers._flattening import flatten_record
+
+
+@pytest.mark.parametrize(
+    "flattened_schema, max_level, expected",
+    [
+        pytest.param(
+            {
+                "properties": {
+                    "key_1": {"type": ["null", "integer"]},
+                    "key_2__key_3": {"type": ["null", "string"]},
+                    "key_2__key_4": {"type": ["null", "object"]},
+                }
+            },
+            99,
+            {
+                "key_1": 1,
+                "key_2__key_3": "value",
+                "key_2__key_4": '{"key_5": 1, "key_6": ["a", "b"]}',
+            },
+            id="flattened schema limiting the max level",
+        ),
+        pytest.param(
+            {
+                "properties": {
+                    "key_1": {"type": ["null", "integer"]},
+                    "key_2__key_3": {"type": ["null", "string"]},
+                    "key_2__key_4__key_5": {"type": ["null", "integer"]},
+                    "key_2__key_4__key_6": {"type": ["null", "array"]},
+                }
+            },
+            99,
+            {
+                "key_1": 1,
+                "key_2__key_3": "value",
+                "key_2__key_4__key_5": 1,
+                "key_2__key_4__key_6": '["a", "b"]',
+            },
+            id="flattened schema not limiting the max level",
+        ),
+        pytest.param(
+            {
+                "properties": {
+                    "key_1": {"type": ["null", "integer"]},
+                    "key_2__key_3": {"type": ["null", "string"]},
+                    "key_2__key_4__key_5": {"type": ["null", "integer"]},
+                    "key_2__key_4__key_6": {"type": ["null", "array"]},
+                }
+            },
+            1,
+            {
+                "key_1": 1,
+                "key_2__key_3": "value",
+                "key_2__key_4": '{"key_5": 1, "key_6": ["a", "b"]}',
+            },
+            id="max level limiting flattened schema",
+        ),
+    ],
+)
+def test_flatten_record(flattened_schema, max_level, expected):
+    """Test flatten_record to obey the max_level and flattened_schema parameters."""
+    record = {
+        "key_1": 1,
+        "key_2": {"key_3": "value", "key_4": {"key_5": 1, "key_6": ["a", "b"]}},
+    }
+
+    result = flatten_record(
+        record, max_level=max_level, flattened_schema=flattened_schema
+    )
+    assert expected == result