Repository: meltano/meltano

Issue:
`metadata` and other extras of kind `object` not honored when using environments
Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3400

Originally created by @tbriggs2 on 2022-04-14 20:18:50

---


### What is the current *bug* behavior?
If the metadata extra is defined in the base configuration of an inherited plugin it is not respected - the metadata from the base configuration of the parent plugin is used instead.



### What is the expected *correct* behavior?
The metadata extra from the inherited plugin definition should be used.



### Steps to reproduce
Using the attached [meltano.yml](http://gitlab.com/meltano/meltano/uploads/3ddeb9658b37693e2edfe65519e970c2/meltano.yml) run

`meltano --environment=prod run tap-mysql2 target-jsonl`

SomeTable2 will get replicated but with replication-method=LOG_BASED. If instead you run

`meltano --environment=dev run tap-mysql2 target-jsonl`

SomeTable2 will be replicated using replication-method=FULL_TABLE as expected. The point being that this appears to be a problem specifically with metadata defined on the base inherited plugin only.

Note also that if you change the metadata in the base parent plugin to not use a wildcard, i.e.

```   
    metadata:
      SchemaName-SomeTable:
        replication-method: LOG_BASED
```

The pipeline will fail with an error about the replication method not being defined for SomeTable2. I think that's relevant because it shows that the wildcard isn't overriding the inherited plugin configuration.

### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team


Discussion:
[View 2 previous comments from the original issue on GitLab](https://gitlab.com/meltano/meltano/-/issues/3400)
From https://gitlab.com/meltano/meltano/-/issues/3400#note_915746214:

> Yes, I can confirm that the parent's `metadata` is used, **when it's defined**. If it's undefined, the child plugin's `metadata` is used.
Seems to be happening to this user [in Slack](https://meltano.slack.com/archives/C01TCRBBJD7/p1654785151254499):

> after upgrading, the workflows that were running fine so far now fail with `Extractor failed: Unrecognized replication_method None`. I did meltano config `tap-postgres--orchest-dashboard set _metadata '*' replication-method FULL_TABLE` again just in case, still fails (which makes sense, because nothing changed in meltano.yml).
@edgarrmondragon They suggest it used to work before upgrading though, are you sure this is the same issue?
@DouweM I'm almost sure this bug was introduced in a 1.10x.0 release and I noticed they're using inheritance. I asked them to confirm if they upgraded from `1.105.0`.

I also tested on `main` and I can confirm the same behavior: `metadata` in child plugin is only honored if parent's is clear.
@edgarrmondragon Thanks, I've put this in the current iteration!

Resolving patch:
diff --git a/src/meltano/core/environment.py b/src/meltano/core/environment.py
--- a/src/meltano/core/environment.py
+++ b/src/meltano/core/environment.py
@@ -15,7 +15,7 @@
 TEnv = TypeVar("TEnv")
 
 
-class NoActiveEnvironment(Exception):
+class NoActiveEnvironment(Exception):  # noqa: N818
     """Exception raised when invocation has no active environment."""
 
 
@@ -92,7 +92,7 @@ def get_orphan_settings(
         Returns:
             List of orphan settings.
         """
-        return SettingDefinition.from_missing(existing, self.config)
+        return SettingDefinition.from_missing(existing, self.config_with_extras)
 
 
 class EnvironmentConfig(Canonical):
diff --git a/src/meltano/core/setting_definition.py b/src/meltano/core/setting_definition.py
--- a/src/meltano/core/setting_definition.py
+++ b/src/meltano/core/setting_definition.py
@@ -194,6 +194,14 @@ def __init__(
 
         self._verbatim.add("value")
 
+    def __repr__(self) -> str:
+        """Return string representation.
+
+        Returns:
+            String representation of this setting.
+        """
+        return f"<SettingDefinition {self.name} ({self.kind})>"
+
     @classmethod
     def from_missing(cls, defs: list[SettingDefinition], config: dict, **kwargs):
         """Create SettingDefinition instances for missing settings.
diff --git a/src/meltano/core/settings_store.py b/src/meltano/core/settings_store.py
--- a/src/meltano/core/settings_store.py
+++ b/src/meltano/core/settings_store.py
@@ -28,7 +28,7 @@
 logger = logging.getLogger(__name__)
 
 
-class ConflictingSettingValueException(Exception):
+class ConflictingSettingValueException(Exception):  # noqa: N818
     """Occurs when a setting has multiple conflicting values via aliases."""
 
     def __init__(self, setting_names):
@@ -50,15 +50,14 @@ def __str__(self) -> str:
         return f"Conflicting values for setting found in: {self.setting_names}"
 
 
-class MultipleEnvVarsSetException(Exception):
+class MultipleEnvVarsSetException(Exception):  # noqa: N818
     """Occurs when a setting value is set via multiple environment variable names."""
 
     def __init__(self, names):
         """Instantiate the error.
 
         Args:
-            setting_names: the name/aliases where conflicting values are set
-
+            names: the name/aliases where conflicting values are set
         """
         self.names = names
         super().__init__(names)
@@ -312,7 +311,10 @@ def get(
 
         Raises:
             StoreNotSupportedError: if setting_def not passed.
-            ConflictingSettingValueException: if multiple conflicting values for the same setting are provided.
+            ConflictingSettingValueException: if multiple conflicting values for the
+                same setting are provided.
+            MultipleEnvVarsSetException: if multiple environment variables are set for
+                the same setting.
 
         Returns:
             A tuple the got value and a dictionary containing metadata.
@@ -328,14 +330,13 @@ def get(
             except KeyError:
                 pass
         if len(vals_with_metadata) > 1:
-            if not reduce(eq, (val for val, _ in vals_with_metadata)):
+            if not reduce(eq, (val for val, _ in vals_with_metadata)):  # noqa: WPS504
                 raise ConflictingSettingValueException(
                     [metadata["env_var"] for _, metadata in vals_with_metadata]
                 )
-            else:
-                raise MultipleEnvVarsSetException(
-                    [metadata["env_var"] for _, metadata in vals_with_metadata]
-                )
+            raise MultipleEnvVarsSetException(
+                [metadata["env_var"] for _, metadata in vals_with_metadata]
+            )
         return vals_with_metadata[0] if vals_with_metadata else (None, {})
 
     def setting_env_vars(self, *args, **kwargs) -> dict:
@@ -594,10 +595,12 @@ def get(
         for key in keys:
             try:
                 value = flat_config[key]
-                self.log(f"Read key '{key}' from `meltano.yml`: {value!r}")
-                vals_with_metadata.append((value, {"key": key, "expandable": True}))
             except KeyError:
-                pass
+                continue
+
+            self.log(f"Read key '{key}' from `meltano.yml`: {value!r}")
+            vals_with_metadata.append((value, {"key": key, "expandable": True}))
+
         if len(vals_with_metadata) > 1 and not reduce(
             eq, (val for val, _ in vals_with_metadata)
         ):
@@ -1205,11 +1208,16 @@ def get(
         for source in self.sources:
             try:
                 manager = self.manager_for(source)
+            except StoreNotSupportedError:
+                continue
+
+            try:
                 value, metadata = manager.get(name, setting_def=setting_def, **kwargs)
-                found_source = source
             except StoreNotSupportedError:
                 continue
 
+            found_source = source
+
             if value is not None:
                 break
 


Test patch:
diff --git a/tests/meltano/core/plugin/test_plugin_settings.py b/tests/meltano/core/plugin/test_plugin_settings.py
--- a/tests/meltano/core/plugin/test_plugin_settings.py
+++ b/tests/meltano/core/plugin/test_plugin_settings.py
@@ -797,6 +797,7 @@ def test_extra(self, subject, tap, monkeypatch, env_var):
 
     def test_extra_object(
         self,
+        active_environment,
         subject,
         monkeypatch,
         env_var,
@@ -836,25 +837,43 @@ def test_extra_object(
             SettingValueStore.MELTANO_YML,
         )
 
-        subject.set("_vars", {"other": "from_meltano_yml"})
+        subject.set(
+            "_vars",
+            {"other": "from_meltano_yml"},
+            store=SettingValueStore.MELTANO_YML,
+        )
 
         assert subject.get_with_source("_vars") == (
             {"var": "from_default", "other": "from_meltano_yml"},
             SettingValueStore.MELTANO_YML,
         )
 
-        monkeypatch.setenv(env_var(subject, "_vars.var"), "from_env")
-
-        assert subject.get_with_source("_vars") == (
-            {"var": "from_env", "other": "from_meltano_yml"},
-            SettingValueStore.ENV,
-        )
+        with monkeypatch.context() as patch:
+            patch.setenv(env_var(subject, "_vars.var"), "from_env")
+            assert subject.get_with_source("_vars") == (
+                {"var": "from_env", "other": "from_meltano_yml"},
+                SettingValueStore.ENV,
+            )
 
-        monkeypatch.setenv(env_var(subject, "_vars"), '{"var": "from_env"}')
+        with monkeypatch.context() as patch:
+            patch.setenv(env_var(subject, "_vars"), '{"var": "from_env"}')
+            assert subject.get_with_source("_vars") == (
+                {"var": "from_env"},
+                SettingValueStore.ENV,
+            )
 
+        subject.set(
+            "_vars",
+            {"dev_setting": "from_dev_env"},
+            store=SettingValueStore.MELTANO_ENV,
+        )
         assert subject.get_with_source("_vars") == (
-            {"var": "from_env"},
-            SettingValueStore.ENV,
+            {
+                "var": "from_default",
+                "other": "from_meltano_yml",
+                "dev_setting": "from_dev_env",
+            },
+            SettingValueStore.MELTANO_ENV,
         )
 
     def test_find_setting_raises_with_conflicting(