Repository: meltano/meltano

Issue:
`metadata` and other extras of kind `object` not honored when using inheritance
Consider the following inheritance scheme:

```yaml
plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    select:
    - raw_github-issues.*
    metadata:
      raw_github-issues:
        replication-method: INCREMENTAL
        replication-key: updated_at
  - name: tap-postgres--inherited
    inherit_from: tap-postgres
    select:
    - raw_github-commits.*
    metadata:
      raw_github-commits:
        replication-method: INCREMENTAL
        replication-key: commit_timestamp
```

The expectation is that `tap-postgres--inherited` uses its own metadata object to patch the Singer catalog, and the output of `meltano config` would suggest that everything is working as expected:

```console
$ meltano config --extras tap-postgres--inherited
{
  "_catalog": null,
  "_state": null,
  "_load_schema": "tap_postgres__inherited",
  "_select": [
    "raw_github-commits.*"
  ],
  "_metadata": {
    "raw_github-issues": {
      "replication-method": "INCREMENTAL",
      "replication-key": "updated_at"
    }
  },
  "_schema": {},
  "_select_filter": [],
  "_metadata.raw_github-issues.replication-method": "INCREMENTAL",
  "_metadata.raw_github-issues.replication-key": "updated_at",
  "_metadata.raw_github-commits.replication-method": "INCREMENTAL",
  "_metadata.raw_github-commits.replication-key": "commit_timestamp"
}
```

> **Note**
> The bug seems to be that `_metadata` above does not contain everything that starts with `_metadata` prefix 🤔

However, invoking `tap-postgres--inherited` fails as its own metadata is being ignored:

```
Traceback (most recent call last):
  File "/Users/edgarramirez/Code/meltano-personal/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(main())
  File "/Users/edgarramirez/Code/meltano-personal/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 435, in main
    raise exc
  File "/Users/edgarramirez/Code/meltano-personal/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 432, in main
    main_impl()
  File "/Users/edgarramirez/Code/meltano-personal/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 421, in main_impl
    do_sync(conn_config, args.catalog.to_dict() if args.catalog else args.properties,
  File "/Users/edgarramirez/Code/meltano-personal/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 293, in do_sync
    sync_method_for_streams(streams, state, default_replication_method)
  File "/Users/edgarramirez/Code/meltano-personal/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 107, in sync_method_for_streams
    raise Exception(f"Unrecognized replication_method {replication_method}")
Exception: Unrecognized replication_method None
```

Removing the `metadata` section from the _parent_ plugin solves the problem, but this not how things should work.


Discussion:
@edgarrmondragon thanks for opening the issue, since this is related to https://github.com/meltano/meltano/issues/3322 and still urgent I'm assigning to you and putting the urgency high label on it.

Resolving patch:
diff --git a/src/meltano/core/settings_service.py b/src/meltano/core/settings_service.py
--- a/src/meltano/core/settings_service.py
+++ b/src/meltano/core/settings_service.py
@@ -374,12 +374,14 @@ def get_with_metadata(  # noqa: WPS210, WPS615
                 value = expanded_value
 
         if setting_def:
-            if (
-                setting_def.kind == SettingKind.OBJECT
-                and metadata["source"] is SettingValueStore.DEFAULT
+            # Expand flattened config values if the root value is the default
+            # or inherited empty object.
+            if setting_def.kind == SettingKind.OBJECT and (
+                metadata["source"]
+                in {SettingValueStore.DEFAULT, SettingValueStore.INHERITED}
             ):
                 object_value = {}
-                object_source = SettingValueStore.DEFAULT
+                object_source = metadata["source"]
                 for setting_key in [  # noqa: WPS335
                     setting_def.name,
                     *setting_def.aliases,


Test patch:
diff --git a/tests/meltano/core/plugin/test_plugin_settings.py b/tests/meltano/core/plugin/test_plugin_settings.py
--- a/tests/meltano/core/plugin/test_plugin_settings.py
+++ b/tests/meltano/core/plugin/test_plugin_settings.py
@@ -876,6 +876,24 @@ def test_extra_object(
             SettingValueStore.MELTANO_ENV,
         )
 
+        subject.project.active_environment = None
+
+        inherited = project_add_service.add(
+            PluginType.TRANSFORMS,
+            "tap-mock-transform--inherited",
+            inherit_from=transform.name,
+        )
+        inherited_subject = plugin_settings_service_factory(inherited)
+        inherited_subject.set("_vars", {"new": "from_inheriting"})
+
+        assert inherited_subject.get_with_source("_vars") == (
+            {
+                "other": "from_meltano_yml",
+                "new": "from_inheriting",
+            },
+            SettingValueStore.MELTANO_YML,
+        )
+
     def test_find_setting_raises_with_conflicting(
         self, tap, plugin_settings_service_factory, monkeypatch
     ):