Repository: meltano/sdk
Pull request #2924 (2025-03-26 00:01:02)

fix: Generate standard stream metadata for nested fields
FWIW, this nested metadata is already added by Meltano: https://github.com/meltano/meltano/blob/7cf6f2a0833127258dc885f4c095a028eab6ac06/src/meltano/core/plugin/singer/catalog.py#L415-L442

## Related

- https://gitlab.com/meltano/sdk/-/issues/285#note_750983833
- https://github.com/meltano/meltano/issues/3024
- https://github.com/meltano/meltano/issues/8807

## Links

- Slack: https://meltano.slack.com/archives/C06A1MD6A6L/p1742822266403049
- https://discuss.meltano.com/t/27124273/i-was-looking-at-some-issues-about-selecting-nested-properti#2873626a-f180-46dd-91ae-58d2b7b85654

## Summary by Sourcery

Generates standard stream metadata for nested fields, ensuring that all subfields within a schema have appropriate metadata entries.

Enhancements:
- Adds metadata entries for all subfields within a schema, ensuring that nested fields are properly represented in the metadata.
- Improves the accuracy and completeness of metadata generated for complex schemas with nested properties.
- Adds a recursive helper function to add breadcrumbs and metadata for subfields.

## Summary by Sourcery

Enhance metadata generation for Singer streams to support nested field metadata, ensuring comprehensive metadata coverage for complex schemas with nested properties.

Enhancements:
- Implement a recursive metadata generation approach to add metadata entries for nested fields in complex schemas
- Extend metadata generation to support nested object and array schemas with comprehensive breadcrumb tracking

Tests:
- Update test cases to validate metadata generation for nested schemas, including objects with multiple levels of nesting and array types

sourcery-ai[bot]: <!-- Generated by sourcery-ai[bot]: start review_guide -->

🧙 Sourcery is reviewing your pull request!

---

<details>
<summary>Tips and commands</summary>

#### Interacting with Sourcery

- **Trigger a new review:** Comment `@sourcery-ai review` on the pull request.
- **Continue discussions:** Reply directly to Sourcery's review comments.
- **Generate a GitHub issue from a review comment:** Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with `@sourcery-ai issue` to create an issue from it.
- **Generate a pull request title:** Write `@sourcery-ai` anywhere in the pull
  request title to generate a title at any time. You can also comment
  `@sourcery-ai title` on the pull request to (re-)generate the title at any time.
- **Generate a pull request summary:** Write `@sourcery-ai summary` anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment `@sourcery-ai summary` on the pull request to
  (re-)generate the summary at any time.
- **Generate reviewer's guide:** Comment `@sourcery-ai guide` on the pull
  request to (re-)generate the reviewer's guide at any time.
- **Resolve all Sourcery comments:** Comment `@sourcery-ai resolve` on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
- **Dismiss all Sourcery reviews:** Comment `@sourcery-ai dismiss` on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  `@sourcery-ai review` to trigger a new review!
- **Generate a plan of action for an issue:** Comment `@sourcery-ai plan` on
  an issue to generate a plan of action for it.

#### Customizing Your Experience

Access your [dashboard](https://app.sourcery.ai) to:
- Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
- Change the review language.
- Add, remove or edit custom review instructions.
- Adjust other review settings.

#### Getting Help

- [Contact our support team](mailto:support@sourcery.ai) for questions or feedback.
- Visit our [documentation](https://docs.sourcery.ai) for detailed guides and information.
- Keep in touch with the Sourcery team by following us on [X/Twitter](https://x.com/SourceryAI), [LinkedIn](https://www.linkedin.com/company/sourcery-ai/) or [GitHub](https://github.com/sourcery-ai).

</details>

<!-- Generated by sourcery-ai[bot]: end review_guide -->
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/meltano/sdk/pull/2924?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) Report
Attention: Patch coverage is `80.00000%` with `2 lines` in your changes missing coverage. Please review.
> Project coverage is 91.37%. Comparing base [(`1e6f06d`)](https://app.codecov.io/gh/meltano/sdk/commit/1e6f06de66d895bea233aeeea3bd5e0247020e37?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) to head [(`47f8db9`)](https://app.codecov.io/gh/meltano/sdk/commit/47f8db9cee673646c07ce5413ee3890bb4aaa62f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).
> Report is 1 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/meltano/sdk/pull/2924?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) | Patch % | Lines |
|---|---|---|
| [singer\_sdk/singerlib/catalog.py](https://app.codecov.io/gh/meltano/sdk/pull/2924?src=pr&el=tree&filepath=singer_sdk%2Fsingerlib%2Fcatalog.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano#diff-c2luZ2VyX3Nkay9zaW5nZXJsaWIvY2F0YWxvZy5weQ==) | 80.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/meltano/sdk/pull/2924?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano) |

:x: Your patch check has failed because the patch coverage (80.00%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the [target](https://docs.codecov.com/docs/commit-status#target) coverage.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2924      +/-   ##
==========================================
- Coverage   91.40%   91.37%   -0.03%     
==========================================
  Files          63       63              
  Lines        5291     5299       +8     
  Branches      676      678       +2     
==========================================
+ Hits         4836     4842       +6     
- Misses        321      322       +1     
- Partials      134      135       +1     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/meltano/sdk/pull/2924?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=meltano).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
codspeed-hq[bot]: <!-- __CODSPEED_PERFORMANCE_REPORT_COMMENT__ -->
<!-- __CODSPEED_INSTRUMENTATION_PERFORMANCE_REPORT_COMMENT__ -->

## [CodSpeed Performance Report](https://codspeed.io/meltano/sdk/branches/edgarrmondragon%2Ffix%2Fnested-metadata)

### Merging #2924 will **not alter performance**

<sub>Comparing <code>edgarrmondragon/fix/nested-metadata</code> (47f8db9) with <code>main</code> (1e6f06d)</sub>



### Summary

`✅ 7` untouched benchmarks  




edgarrmondragon: @sourcery-ai review
edgarrmondragon: @sourcery-ai review
edgarrmondragon: > FWIW, this nested metadata is already added by Meltano: https://github.com/meltano/meltano/blob/7cf6f2a0833127258dc885f4c095a028eab6ac06/src/meltano/core/plugin/singer/catalog.py#L415-L442

Even then, it'd be nice not rely on Meltano to enrich their metadata.

Patch:
diff --git a/singer_sdk/singerlib/catalog.py b/singer_sdk/singerlib/catalog.py
index f235c47a..4082c16f 100644
--- a/singer_sdk/singerlib/catalog.py
+++ b/singer_sdk/singerlib/catalog.py
@@ -188,13 +188,25 @@ class MetadataMapping(dict[Breadcrumb, AnyMetadata]):
             selected_by_default=selected_by_default,
         )
 
+        def _add_subfield_metadata(
+            properties: dict,
+            breadcrumb: Breadcrumb = (),
+        ) -> None:
+            """Add breadcrumbs and metadata for subfields."""
+            for field_name, field_schema in properties.items():
+                key = (*breadcrumb, "properties", field_name)
+                mapping[key] = Metadata(inclusion=Metadata.InclusionType.AVAILABLE)
+
+                if field_schema.get("properties"):
+                    _add_subfield_metadata(field_schema["properties"], key)
+
         if schema:
             root.inclusion = Metadata.InclusionType.AVAILABLE
 
             if schema_name:
                 root.schema_name = schema_name
 
-            for field_name in schema.get("properties", {}):
+            for field_name, field_schema in schema.get("properties", {}).items():
                 if (key_properties and field_name in key_properties) or (
                     valid_replication_keys and field_name in valid_replication_keys
                 ):
@@ -202,7 +214,10 @@ class MetadataMapping(dict[Breadcrumb, AnyMetadata]):
                 else:
                     entry = Metadata(inclusion=Metadata.InclusionType.AVAILABLE)
 
-                mapping["properties", field_name] = entry
+                breadcrumb = ("properties", field_name)
+                mapping[breadcrumb] = entry
+
+                _add_subfield_metadata(field_schema.get("properties", {}), breadcrumb)
 
         mapping[()] = root
 


Test patch:
diff --git a/tests/singerlib/test_catalog.py b/tests/singerlib/test_catalog.py
index 41efd806..91d59a1a 100644
--- a/tests/singerlib/test_catalog.py
+++ b/tests/singerlib/test_catalog.py
@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+import typing as t
+
 import pytest
 
 from singer_sdk._singerlib import (
@@ -11,6 +13,9 @@ from singer_sdk._singerlib import (
     StreamMetadata,
 )
 
+if t.TYPE_CHECKING:
+    from singer_sdk.singerlib.catalog import Breadcrumb
+
 METADATA_ARRAY = [
     {
         "breadcrumb": [],
@@ -193,16 +198,18 @@ def test_catalog_parsing():
 
 
 @pytest.mark.parametrize(
-    "schema,key_properties,replication_method,valid_replication_keys,schema_name",
+    "schema,key_properties,replication_method,valid_replication_keys,schema_name,breadcrumbs",
     [
-        (
+        pytest.param(
             {"properties": {"id": {"type": "integer"}}, "type": "object"},
             ["id"],
             "FULL_TABLE",
             None,
             None,
+            {(), ("properties", "id")},
+            id="simple_integer_id_full_table",
         ),
-        (
+        pytest.param(
             {
                 "properties": {
                     "first_name": {"type": "string"},
@@ -215,8 +222,15 @@ def test_catalog_parsing():
             "INCREMENTAL",
             ["updated_at"],
             "users",
+            {
+                (),
+                ("properties", "first_name"),
+                ("properties", "last_name"),
+                ("properties", "updated_at"),
+            },
+            id="users_incremental_with_datetime",
         ),
-        (
+        pytest.param(
             {
                 "properties": {
                     "first_name": {"type": "string"},
@@ -229,13 +243,74 @@ def test_catalog_parsing():
             "FULL_TABLE",
             None,
             None,
+            {
+                (),
+                ("properties", "first_name"),
+                ("properties", "last_name"),
+                ("properties", "group"),
+            },
+            id="users_full_table_with_group",
         ),
-        (
-            {},
-            [],
+        pytest.param(
+            {
+                "properties": {
+                    "id": {"type": "string"},
+                    "user": {
+                        "type": "object",
+                        "properties": {
+                            "name": {"type": "string"},
+                            "age": {"type": "integer"},
+                            "address": {
+                                "type": "object",
+                                "properties": {
+                                    "street": {"type": "string"},
+                                    "city": {"type": "string"},
+                                    "state": {"type": "string"},
+                                },
+                            },
+                        },
+                    },
+                },
+                "type": "object",
+            },
+            ["id"],
+            "FULL_TABLE",
             None,
             None,
+            {
+                (),
+                ("properties", "id"),
+                ("properties", "user"),
+                ("properties", "user", "properties", "name"),
+                ("properties", "user", "properties", "age"),
+                ("properties", "user", "properties", "address"),
+                ("properties", "user", "properties", "address", "properties", "street"),
+                ("properties", "user", "properties", "address", "properties", "city"),
+                ("properties", "user", "properties", "address", "properties", "state"),
+            },
+            id="nested_user_object_full_table",
+        ),
+        pytest.param({}, [], None, None, None, {()}, id="empty_schema"),
+        pytest.param(
+            {
+                "properties": {
+                    "id": {"type": "integer"},
+                    "nested_array": {
+                        "type": "array",
+                        "items": {
+                            "type": "object",
+                            "properties": {"name": {"type": "string"}},
+                        },
+                    },
+                },
+                "type": "object",
+            },
+            [],
+            "FULL_TABLE",
+            None,
             None,
+            {(), ("properties", "id"), ("properties", "nested_array")},
+            id="nested_array_full_table",
         ),
     ],
 )
@@ -245,6 +320,7 @@ def test_standard_metadata(
     replication_method: str | None,
     valid_replication_keys: list[str] | None,
     schema_name: str | None,
+    breadcrumbs: set[Breadcrumb],
 ):
     """Validate generated metadata."""
     metadata = MetadataMapping.get_standard_metadata(
@@ -272,3 +348,5 @@ def test_standard_metadata(
         rk_metadata = metadata["properties", rk]
         assert rk_metadata.inclusion == Metadata.InclusionType.AUTOMATIC
         assert rk_metadata.selected is None
+
+    assert set(metadata) == breadcrumbs