Repository: meltano/meltano

Issue:
Update environment variable inheritance to be in line with currently xfail-ing tests
Environment variable inheritance should behave as illustrated by the `STACKED` variables in [this issue description](https://gitlab.com/meltano/meltano/-/issues/3173). There is a test written for this behavior in `tests/meltano/core/test_environment_variables.py` which is currently marked to `xfail`. Inheritance behavior should be updated so that this test passes.

## Related: 

- Exploratory issue:
  - https://github.com/meltano/meltano/issues/3100
  - https://github.com/meltano/meltano/pull/5938
- Also found:
  - https://github.com/meltano/meltano/issues/5982
- Xfail (expected-to-fail) tests to fix:  https://github.com/meltano/meltano/commit/17ce6c65d2ea3f4b5851528ec15de45e22843be8



Discussion:
@cjohnhanson - Thanks for logging this.

Can you say a bit more of the currently observed behavior you seeing on inheritance/stacking?

For instance, is it something of a name-conflict issue, or are higher-level values simply unavailable during evaluation at lower levels of the tree?


@aaronsteers what's the weight on this and is this something that can be done for 2.0? 
@aaronsteers here's the current behavior, basically environment-level `env` takes precedence and doesn't inherit from variable definition in other "layers". See the `STACKED=3` output line here:
```bash
~/Projects/project-meltano/env-vars/env-vars-3 via 🐍 v3.9.12 (env-vars) on ☁️  cody@codyjhanson.com 
❯ export STACKED=1

~/Projects/project-meltano/env-vars/env-vars-3 via 🐍 v3.9.12 (env-vars) on ☁️  cody@codyjhanson.com 
❯ cat meltano.yml
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: meltano.yml
       │ Size: 667 B
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ version: 1
   2   │ default_environment: dev
   3   │ project_id: 044adc22-cf5b-4c51-9bef-46b512b848fe
   4   │ send_anonymous_usage_stats: false
   5   │ env: "${STACKED}2"
   6   │ plugins:
   7   │   utilities:
   8   │   - name: codys-test-plugin
   9   │     namespace: codys_test_plugin
  10   │     executable: pwd
  11   │     env:
  12   │       STACKED: "${STACKED}4"
  13   │     settings:
  14   │     - name: cody_from
  15   │       env: CODY_FROM
  16   │       kind: string
  17   │       aliases: [cody_from_alias]
  18   │       value: default_value
  19   │ 
  20   │ environments:
  21   │ - name: dev
  22   │   env:
  23   │     STACKED: "${STACKED}3"
  24   │   config:
  25   │     plugins:
  26   │       utilities:
  27   │         - name: codys-test-plugin
  28   │           env:
  29   │             STACKED: "${STACKED}5"
  30   │           config:
  31   │             cody_from_alias: "coming_from_alias"
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

~/Projects/project-meltano/env-vars/env-vars-3 via 🐍 v3.9.12 (env-vars) on ☁️  cody@codyjhanson.com 
❯ meltano invoke --print-var STACKED codys-test-plugin
2022-06-03T21:12:48.422490Z [info     ] Environment 'dev' is active
STACKED=3
/Users/codyhanson/Projects/project-meltano/env-vars/env-vars-3

```
@tayloramurphy @aaronsteers this might be overly optimistic, but my best guess on weight for this would be an 4, I think it would just involve correcting the behavior in `PluginContext`. I think I could knock this out this weekend, particularly if @edgarrmondragon is taking over https://github.com/meltano/meltano/issues/3375
@aaronsteers -- just to confirm, the scope of this and #5982 do _not_ currently include the implementation outlined in #6253, correct? We're going to leave that as tech debt for now (potentially pending further discussion) and move forward with the more short-term solution of directly updating this logic in the each of the various places where env variable expansion happens? I think there may have been some offline synchronous discussion around this that didn't make it into these issues, so just wanted to make sure the approach hasn't changed.

Resolving patch:
diff --git a/src/meltano/core/plugin_invoker.py b/src/meltano/core/plugin_invoker.py
--- a/src/meltano/core/plugin_invoker.py
+++ b/src/meltano/core/plugin_invoker.py
@@ -23,6 +23,9 @@
 from .plugin.settings_service import PluginSettingsService
 from .project import Project
 from .project_plugins_service import ProjectPluginsService
+from .project_settings_service import ProjectSettingsService
+from .settings_service import FeatureFlags
+from .utils import expand_env_vars
 from .venv_service import VenvService, VirtualEnv
 
 logger = get_logger(__name__)
@@ -321,10 +324,63 @@ def env(self):
         Returns:
             Dictionary of environment variables.
         """
+        project_settings_service = ProjectSettingsService(
+            self.project, config_service=self.plugins_service.config_service
+        )
+        with self.settings_service.feature_flag(
+            FeatureFlags.STRICT_ENV_VAR_MODE, raise_error=False
+        ) as strict_env_var_mode:
+
+            # Expand root env w/ os.environ
+            expanded_project_env = expand_env_vars(
+                project_settings_service.env,
+                os.environ,
+                raise_if_missing=strict_env_var_mode,
+            )
+            expanded_project_env.update(
+                expand_env_vars(
+                    self.settings_service.project.dotenv_env,
+                    os.environ,
+                    raise_if_missing=strict_env_var_mode,
+                )
+            )
+            # Expand active env w/ expanded root env
+            expanded_active_env = (
+                expand_env_vars(
+                    self.settings_service.project.active_environment.env,
+                    expanded_project_env,
+                    raise_if_missing=strict_env_var_mode,
+                )
+                if self.settings_service.project.active_environment
+                else {}
+            )
+
+            # Expand root plugin env w/ expanded active env
+            expanded_root_plugin_env = expand_env_vars(
+                self.settings_service.plugin.env,
+                expanded_active_env,
+                raise_if_missing=strict_env_var_mode,
+            )
+
+            # Expand active env plugin env w/ expanded root plugin env
+            expanded_active_env_plugin_env = (
+                expand_env_vars(
+                    self.settings_service.environment_plugin_config.env,
+                    expanded_root_plugin_env,
+                    raise_if_missing=strict_env_var_mode,
+                )
+                if self.settings_service.environment_plugin_config
+                else {}
+            )
+
         env = {
+            **expanded_project_env,
             **self.project.dotenv_env,
             **self.settings_service.env,
             **self.plugin_config_env,
+            **expanded_root_plugin_env,
+            **expanded_active_env,
+            **expanded_active_env_plugin_env,
         }
 
         # Ensure Meltano venv is not inherited
diff --git a/src/meltano/core/utils/__init__.py b/src/meltano/core/utils/__init__.py
--- a/src/meltano/core/utils/__init__.py
+++ b/src/meltano/core/utils/__init__.py
@@ -38,7 +38,12 @@ class NotFound(Exception):
     """Occurs when an element is not found."""
 
     def __init__(self, name, obj_type=None):
-        """Create a new exception."""
+        """Create a new exception.
+
+        Args:
+            name: the name of the element that is not found
+            obj_type: the type of element
+        """
         if obj_type is None:
             super().__init__(f"{name} was not found.")
         else:
@@ -46,7 +51,14 @@ def __init__(self, name, obj_type=None):
 
 
 def click_run_async(func):
-    """Small decorator to allow click invoked functions to leverage `asyncio.run` and be declared as async."""
+    """Small decorator to allow click invoked functions to leverage `asyncio.run` and be declared as async.
+
+    Args:
+        func: the function to run async
+
+    Returns:
+        A function which runs the given function async
+    """
 
     @functools.wraps(func)
     def wrapper(*args, **kwargs):  # noqa: WPS430
@@ -67,7 +79,7 @@ def compose(*fs: Callable[[Any], Any]):
         compose(f, g)(x) == f(g(x))
         ```
 
-    Return:
+    Returns:
         The composition of the provided unary functions, which itself is a unary function.
     """
     return functools.reduce(lambda f, g: lambda x: f(g(x)), compact(fs), lambda x: x)
@@ -77,6 +89,12 @@ def compose(*fs: Callable[[Any], Any]):
 def slugify(s):
     """Normalize strings into something URL-friendly.
 
+    Args:
+        s: the string to slugify
+
+    Returns:
+        The string as a slug
+
     >>> slugify("[Some] _ Article's Title--")
     'some-articles-title'
     """
@@ -147,6 +165,16 @@ def merge(src, dest):
 def nest(d: dict, path: str, value=None, maxsplit=-1, force=False):
     """Create a hierarchical dictionary path and return the leaf dict.
 
+    Args:
+        d: the dictionary to operate on
+        path: the dot-delimited path to operate on
+        value: the value to set at the given path
+        maxsplit: maximum number of splits to split path by
+        force: if true, write an empty dict
+
+    Returns:
+        The leaf element of the dict
+
     Examples:
         >>> d = dict()
         >>> test = nest(d, "foo.bar.test")
@@ -204,6 +232,14 @@ def flatten(d: dict, reducer: str | Callable = "tuple", **kwargs):
     """Flatten a dictionary with `dot` and `env_var` reducers.
 
     Wrapper arround `flatten_dict.flatten`.
+
+    Args:
+        d: the dict to flatten
+        reducer: the reducer to flatten with
+        **kwargs: additional kwargs to pass to flatten_dict.flatten
+
+    Returns:
+        the flattened dict
     """
 
     def dot_reducer(*xs):
@@ -220,6 +256,14 @@ def dot_reducer(*xs):
 
 
 def compact(xs: Iterable) -> Iterable:
+    """Remove None values from an iterable.
+
+    Args:
+        xs: the iterable to operate on
+
+    Returns:
+        The iterable with Nones removed
+    """
     return (x for x in xs if x is not None)
 
 
@@ -245,7 +289,14 @@ def truthy(val: str) -> bool:
 
 
 def coerce_datetime(d: date | datetime) -> datetime | None:
-    """Add a `time` component to `d` if it is missing."""
+    """Add a `time` component to `d` if it is missing.
+
+    Args:
+        d: the date or datetime to add the time to
+
+    Returns:
+        The resulting datetime
+    """
     if d is None:
         return None
 
@@ -371,12 +422,17 @@ def __init__(self, env_var: str):
         self.env_var = env_var
 
     def __str__(self) -> str:
-        """Return the error as a string."""
+        """Return the error as a string."""  # noqa: DAR201
         return f"{self.env_var} referenced but not set."
 
 
 def expand_env_vars(raw_value, env: dict, raise_if_missing: bool = False):
-    if not isinstance(raw_value, str):
+    if isinstance(raw_value, dict):
+        return {
+            key: expand_env_vars(val, env, raise_if_missing)
+            for key, val in raw_value.items()
+        }
+    elif not isinstance(raw_value, str):
         return raw_value
 
     # find viable substitutions
@@ -422,7 +478,15 @@ def uniques_in(original):
 
 # https://gist.github.com/cbwar/d2dfbc19b140bd599daccbe0fe925597#gistcomment-2845059
 def human_size(num, suffix="B"):
-    """Return human-readable file size."""
+    """Return human-readable file size.
+
+    Args:
+        num: the number to convert
+        suffix: the suffix to append to the resulting file size
+
+    Returns:
+        File size in human-readable format
+    """
     magnitude = int(math.floor(math.log(num, 1024)))
     val = num / math.pow(1024, magnitude)
 
@@ -456,7 +520,7 @@ def format_exception(exception: BaseException) -> str:
     """Get the exception with its traceback in the standard format it would have been printed with.
 
     Args:
-        The exception value to be turned into a string.
+        exception: The exception value to be turned into a string.
 
     Returns:
         A string that shows the exception object as it would have been printed had it been raised


Test patch:
diff --git a/tests/meltano/core/test_environment_variables.py b/tests/meltano/core/test_environment_variables.py
--- a/tests/meltano/core/test_environment_variables.py
+++ b/tests/meltano/core/test_environment_variables.py
@@ -256,90 +256,90 @@ def test_env_var_resolution(
             ]
         )
 
-    @pytest.mark.xfail
-    def test_environment_variable_inheritance(self, cli_runner, project, monkeypatch):
-        # This test will be resolved to pass as part of
-        # this issue: https://github.com/meltano/meltano/issues/5983
-        monkeypatch.setenv("STACKED", "1")
-        with project.meltano_update() as meltanofile:
-            meltanofile.update(
-                {
-                    "env": "${STACKED}2",
-                    "plugins": {
-                        "utilities": [
-                            {
-                                "name": "test-environment-inheritance",
-                                "namespace": "test_environment_inheritance",
-                                "executable": "pwd",
-                                "env": "${STACKED}4",
-                            }
-                        ],
-                    },
-                    "environments": [
+
+def test_environment_variable_inheritance(cli_runner, project, monkeypatch):
+    # This test will be resolved to pass as part of
+    # this issue: https://github.com/meltano/meltano/issues/5983
+    monkeypatch.setenv("STACKED", "1")
+    with project.meltano_update() as meltanofile:
+        meltanofile.update(
+            {
+                "env": {"STACKED": "${STACKED}2"},
+                "plugins": {
+                    "utilities": [
                         {
-                            "name": "dev",
-                            "env": {"STACKED": "${STACKED}3"},
-                            "config": {
-                                "plugins": {
-                                    "utilities": [
-                                        {
-                                            "name": "test-environment-inheritance",
-                                            "env": {"STACKED": "${STACKED}5"},
-                                        }
-                                    ]
-                                }
-                            },
+                            "name": "test-environment-inheritance",
+                            "namespace": "test_environment_inheritance",
+                            "executable": "pwd",
+                            "env": {"STACKED": "${STACKED}4"},
                         }
                     ],
                 },
-            )
-        result = cli_runner.invoke(
-            cli,
-            [
-                "invoke",
-                "--print-var",
-                "STACKED",
-                "test-environment-inheritance",
-            ],
+                "environments": [
+                    {
+                        "name": "dev",
+                        "env": {"STACKED": "${STACKED}3"},
+                        "config": {
+                            "plugins": {
+                                "utilities": [
+                                    {
+                                        "name": "test-environment-inheritance",
+                                        "env": {"STACKED": "${STACKED}5"},
+                                    }
+                                ]
+                            }
+                        },
+                    }
+                ],
+            },
         )
-        assert_cli_runner(result)
-        assert result.stdout.strip() == "STACKED=12345"
+    result = cli_runner.invoke(
+        cli,
+        [
+            "invoke",
+            "--print-var",
+            "STACKED",
+            "test-environment-inheritance",
+        ],
+    )
+    assert_cli_runner(result)
+    assert result.stdout.strip() == "STACKED=12345"
 
-    @pytest.mark.xfail
-    def test_environment_variable_inheritance_meltano_env_only(
-        self, cli_runner, project, monkeypatch
-    ):
-        # This test will be resolved to pass as part of
-        # this issue: https://github.com/meltano/meltano/issues/5983
-        monkeypatch.setenv("STACKED", "1")
-        with project.meltano_update() as meltanofile:
-            meltanofile.update(
-                {
-                    "plugins": {
-                        "utilities": [
-                            {
-                                "name": "test-environment-inheritance",
-                                "namespace": "test_environment_inheritance",
-                                "executable": "pwd",
-                            }
-                        ],
-                    },
-                    "environments": [
+
+def test_environment_variable_inheritance_meltano_env_only(
+    cli_runner, project, monkeypatch
+):
+    # This test will be resolved to pass as part of
+    # this issue: https://github.com/meltano/meltano/issues/5983
+    monkeypatch.setenv("STACKED", "1")
+    with project.meltano_update() as meltanofile:
+        meltanofile.update(
+            {
+                "plugins": {
+                    "utilities": [
                         {
-                            "name": "dev",
-                            "env": {"STACKED": "${STACKED}2"},
-                        },
+                            "name": "test-environment-inheritance",
+                            "namespace": "test_environment_inheritance",
+                            "executable": "pwd",
+                        }
                     ],
                 },
-            )
-        result = cli_runner.invoke(
-            cli,
-            [
-                "invoke",
-                "--print-var",
-                "STACKED",
-                "test-environment-inheritance",
-            ],
+                "environments": [
+                    {
+                        "name": "dev",
+                        "env": {"STACKED": "${STACKED}2"},
+                    },
+                ],
+            },
         )
-        assert_cli_runner(result)
-        assert result.stdout.strip() == "STACKED=12"
+    result = cli_runner.invoke(
+        cli,
+        [
+            "invoke",
+            "--print-var",
+            "STACKED",
+            "test-environment-inheritance",
+        ],
+    )
+    assert_cli_runner(result)
+    assert result.stdout.strip() == "STACKED=12"
diff --git a/tests/meltano/core/test_utils.py b/tests/meltano/core/test_utils.py
--- a/tests/meltano/core/test_utils.py
+++ b/tests/meltano/core/test_utils.py
@@ -2,7 +2,7 @@
 
 import pytest  # noqa: F401
 
-from meltano.core.utils import flatten, nest, pop_at_path, set_at_path
+from meltano.core.utils import expand_env_vars, flatten, nest, pop_at_path, set_at_path
 
 
 def test_nest():
@@ -79,3 +79,35 @@ def test_flatten():
     expected_flat = {"_update.orchestrate/dags/meltano.py": False}
     result = flatten(example_config, "dot")
     assert result == expected_flat
+
+
+def test_expand_env_vars():
+    assert (
+        expand_env_vars("${ENV_VAR}_suffix", {"ENV_VAR": "substituted"})
+        == "substituted_suffix"
+    )
+
+
+def test_expand_env_vars_nested():
+    input_dict = {
+        "some_key": 12,
+        "some_var": "${ENV_VAR_1}",
+        "nested": {
+            "${THIS_DOES_NOT_EXPAND}": "another_val",
+            "another_layer": {"nested_var": "${ENV_VAR_2}"},
+        },
+        "another_top_level_var": "${ENV_VAR_2}",
+    }
+    env = {"ENV_VAR_1": "substituted_1", "ENV_VAR_2": "substituted_2"}
+
+    expected_output = {
+        "some_key": 12,
+        "some_var": "substituted_1",
+        "nested": {
+            "${THIS_DOES_NOT_EXPAND}": "another_val",
+            "another_layer": {"nested_var": "substituted_2"},
+        },
+        "another_top_level_var": "substituted_2",
+    }
+
+    assert expand_env_vars(input_dict, env) == expected_output