Repository: meltano/meltano

Issue:
Allow defining required environment variables
Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3426

Originally created by @pnadolny13 on 2022-04-20 18:09:54

---

Its common for projects to have a .env file with user or development secrets which is git ignored, along with a sample of that .env that is checked into git for users to know what environment variables to set when they clone the project.

Meltano could allow you to set these as env vars as required so it can fail if they arent set instead of relying on users to read the sample file and export them properly. They can be set at the environment level to allow for variation based on environment.

Squared Project Use Case:

For ELT development the Squared project has isolated environments inside Snowflake that are configured based on some local settings: 
- the [USER_PREFIX](https://gitlab.com/meltano/squared/-/blob/master/data/environments.meltano.yml#L116) set in the meltano.yml which is propagated throughout to create the isolation
- the snowflake password thats stored once in the [.env](https://gitlab.com/meltano/squared/-/blob/master/data/.env.template#L18) so it can be shared by dbt/target-snowflake/great expectations.

The setup instructions tell the user to edit the meltano.yml and set the USER_PREFIX to their name (i.e. pnadolny) but thats not ideal because theres always a diff when trying to commit something. The other option is to remove it from the meltano.yml and set it in the .env file, which works but it not ideal because someone can easily forget to add it and without any validation the dbt configuration ends up writing to `USERDEV._PROD.TABLE_NAME` instead of `USERDEV.PNADOLNY_PROD.TABLE_NAME` because USERDEV is null and is not validated.



```yaml
environments:
- name: userdev
  config:
    plugins:
      extractors:
        ...
  required_env_vars:
    - USER_PREFIX
  env:
    OTHER_HARD_CODED_ENVS: foo
```

```bash
# .env file
USER_PREFIX=pnadolny
```


Discussion:
[View 2 previous comments from the original issue on GitLab](https://gitlab.com/meltano/meltano/-/issues/3426)
## Feature flag: `MELTANO_FF_ENV_VAR_STRICT_MODE`

We could likely solve this with a new 'strict mode' feature flag that fails execution if an environment variable reference is not available. 

By taking a feature flag approach, any users who really want to silently take an empty string instead of a failure could do so. (Primarily for legacy compatibility reasons, I would think.)

But eventually the default behavior should be: if you reference an environment variable explicitly, that variable shouldn't be null.

## Explicit null handling

Meltano has a custom parsing logic that looks for the bash env var syntax `${varname}` or `$varname`. We could easily extend this logic so that the bash convention for `:-` signifies when we should coalesce to another value rather than failing.

As described here: https://unix.stackexchange.com/a/62337/487180

```yml
environments:
  prod:
    env:
      SAMPLE1: ${DOESNT_EXIST}.   ## hard failure: "Required environment variable `DOESNT_EXIST` doesn't exist."
      SAMPLE2: ${DOESNT_EXIST:-}.   ## returns empty string ('')
      SAMPLE3: ${DOESNT_EXIST:-default}.   ## returns 'default'
```


Resolving patch:
diff --git a/src/meltano/core/project_settings_service.py b/src/meltano/core/project_settings_service.py
--- a/src/meltano/core/project_settings_service.py
+++ b/src/meltano/core/project_settings_service.py
@@ -5,7 +5,11 @@
 from dotenv import dotenv_values
 
 from meltano.core.setting_definition import SettingDefinition
-from meltano.core.settings_service import SettingsService, SettingValueStore
+from meltano.core.settings_service import (
+    FeatureFlags,
+    SettingsService,
+    SettingValueStore,
+)
 from meltano.core.utils import expand_env_vars as do_expand_env_vars
 from meltano.core.utils import nest_object
 
@@ -41,10 +45,15 @@ def __init__(self, *args, config_service: ConfigService = None, **kwargs):
             # Update this with `self.project.dotenv_env`, `self.env`, etc. to expand
             # other environment variables in the Environment's `env`.
             expandable_env = {**self.project.env}
-            environment_env = {
-                var: do_expand_env_vars(value, expandable_env)
-                for var, value in self.project.active_environment.env.items()
-            }
+            with self.feature_flag(
+                FeatureFlags.STRICT_ENV_VAR_MODE, raise_error=False
+            ) as strict_env_var_mode:
+                environment_env = {
+                    var: do_expand_env_vars(
+                        value, expandable_env, raise_if_missing=strict_env_var_mode
+                    )
+                    for var, value in self.project.active_environment.env.items()
+                }
             self.env_override.update(environment_env)
 
         self.config_override = {  # noqa: WPS601
@@ -162,7 +171,6 @@ def get_with_metadata(self, name: str, *args, **kwargs):
             if ui_cfg_value is not None:
                 value = ui_cfg_value
                 metadata["source"] = SettingValueStore.ENV
-
         return value, metadata
 
     def get_from_ui_cfg(self, name: str):
diff --git a/src/meltano/core/settings_service.py b/src/meltano/core/settings_service.py
--- a/src/meltano/core/settings_service.py
+++ b/src/meltano/core/settings_service.py
@@ -32,6 +32,7 @@ class FeatureFlags(Enum):
     LOCKFILES = "lock_files"
     ENABLE_UVICORN = "enable_uvicorn"
     ENABLE_API_SCHEDULED_JOB_LIST = "enable_api_scheduled_job_list"
+    STRICT_ENV_VAR_MODE = "strict_env_var_mode"
 
     def __str__(self):
         """Return feature name.
@@ -365,10 +366,16 @@ def get_with_metadata(  # noqa: WPS210
         value, get_metadata = manager.get(name, setting_def=setting_def)
         metadata.update(get_metadata)
 
+        # Can't do conventional SettingsService.feature_flag call to check;
+        # it would result in circular dependency
+        env_var_strict_mode, _ = manager.get(
+            f"{FEATURE_FLAG_PREFIX}.{FeatureFlags.STRICT_ENV_VAR_MODE}"
+        )
         if expand_env_vars and metadata.get("expandable", False):
             metadata["expandable"] = False
-
-            expanded_value = do_expand_env_vars(value, env=expandable_env)
+            expanded_value = do_expand_env_vars(
+                value, env=expandable_env, raise_if_missing=env_var_strict_mode
+            )
 
             if expanded_value != value:
                 metadata["expanded"] = True
diff --git a/src/meltano/core/utils/__init__.py b/src/meltano/core/utils/__init__.py
--- a/src/meltano/core/utils/__init__.py
+++ b/src/meltano/core/utils/__init__.py
@@ -366,7 +366,23 @@ def set_at_path(d, path, value):
     final[tail] = value
 
 
-def expand_env_vars(raw_value, env: Dict):
+class EnvironmentVariableNotSetError(Exception):
+    """Occurs when a referenced environment variable is not set."""
+
+    def __init__(self, env_var: str):
+        """Initialize the error.
+        Args:
+            env_var: the unset environment variable name
+        """
+        super().__init__(env_var)
+        self.env_var = env_var
+
+    def __str__(self) -> str:
+        """Return the error as a string."""
+        return f"{self.env_var} referenced but not set."
+
+
+def expand_env_vars(raw_value, env: Dict, raise_if_missing: bool = False):
     if not isinstance(raw_value, str):
         return raw_value
 
@@ -394,8 +410,11 @@ def subst(match) -> str:
 
             return val
         except KeyError as e:
-            logger.debug(f"Variable '${var}' is missing from the environment.")
-            return None
+            if raise_if_missing:
+                raise EnvironmentVariableNotSetError(e.args[0])
+            else:
+                logger.debug(f"Variable '${var}' is missing from the environment.")
+                return None
 
     fullmatch = re.fullmatch(var_matcher, raw_value)
     if fullmatch:


Test patch:
diff --git a/tests/fixtures/core.py b/tests/fixtures/core.py
--- a/tests/fixtures/core.py
+++ b/tests/fixtures/core.py
@@ -80,6 +80,7 @@ def discovery():  # noqa: WPS213
                             "kind": "string",
                             "aliases": ["aliased_1", "aliased_2", "aliased_3"],
                         },
+                        {"name": "stacked_env_var", "kind": "string"},
                     ],
                     "commands": {
                         "cmd": {
diff --git a/tests/meltano/core/plugin/test_plugin_settings.py b/tests/meltano/core/plugin/test_plugin_settings.py
--- a/tests/meltano/core/plugin/test_plugin_settings.py
+++ b/tests/meltano/core/plugin/test_plugin_settings.py
@@ -14,7 +14,9 @@
 from meltano.core.project import Project
 from meltano.core.project_plugins_service import PluginAlreadyAddedException
 from meltano.core.setting import Setting
+from meltano.core.settings_service import FEATURE_FLAG_PREFIX, FeatureFlags
 from meltano.core.settings_store import ConflictingSettingValueException
+from meltano.core.utils import EnvironmentVariableNotSetError
 
 
 def test_create(session):
@@ -859,3 +861,14 @@ def test_find_setting_aliases(self, tap, plugin_settings_service_factory):
         subject = plugin_settings_service_factory(tap)
         subject.set("aliased_3", "value_3")
         assert subject.get("aliased") == "value_3"
+
+    def test_strict_env_var_mode_on_raises_error(self, subject):
+        subject.set([FEATURE_FLAG_PREFIX, str(FeatureFlags.STRICT_ENV_VAR_MODE)], True)
+        subject.set("stacked_env_var", "${NONEXISTENT_ENV_VAR}")
+        with pytest.raises(EnvironmentVariableNotSetError):
+            subject.get("stacked_env_var")
+
+    def test_strict_env_var_mode_off_no_raise_error(self, subject):
+        subject.set([FEATURE_FLAG_PREFIX, str(FeatureFlags.STRICT_ENV_VAR_MODE)], False)
+        subject.set("stacked_env_var", "${NONEXISTENT_ENV_VAR}")
+        assert subject.get("stacked_env_var") is None
diff --git a/tests/meltano/core/test_environment_variables.py b/tests/meltano/core/test_environment_variables.py
--- a/tests/meltano/core/test_environment_variables.py
+++ b/tests/meltano/core/test_environment_variables.py
@@ -257,3 +257,42 @@ def test_environment_variable_inheritance(self, cli_runner, project, monkeypatch
         )
         assert_cli_runner(result)
         assert result.stdout.strip() == "STACKED=12345"
+
+    @pytest.mark.xfail
+    def test_environment_variable_inheritance_meltano_env_only(
+        self, cli_runner, project, monkeypatch
+    ):
+        # This test will be resolved to pass as part of
+        # this issue: https://github.com/meltano/meltano/issues/5983
+        monkeypatch.setenv("STACKED", "1")
+        with project.meltano_update() as meltanofile:
+            meltanofile.update(
+                {
+                    "plugins": {
+                        "utilities": [
+                            {
+                                "name": "test-environment-inheritance",
+                                "namespace": "test_environment_inheritance",
+                                "executable": "pwd",
+                            }
+                        ],
+                    },
+                    "environments": [
+                        {
+                            "name": "dev",
+                            "env": {"STACKED": "${STACKED}2"},
+                        },
+                    ],
+                },
+            )
+        result = cli_runner.invoke(
+            cli,
+            [
+                "invoke",
+                "--print-var",
+                "STACKED",
+                "test-environment-inheritance",
+            ],
+        )
+        assert_cli_runner(result)
+        assert result.stdout.strip() == "STACKED=12"
diff --git a/tests/meltano/core/test_project_plugins_service.py b/tests/meltano/core/test_project_plugins_service.py
--- a/tests/meltano/core/test_project_plugins_service.py
+++ b/tests/meltano/core/test_project_plugins_service.py
@@ -151,7 +151,7 @@ def test_get_parent_from_lockfile(
         result_no_ff, source = subject.find_parent(tap)
         assert source == DefinitionSource.DISCOVERY
         assert result_no_ff == expected
-        assert len(result_no_ff.settings) == 12  # noqa: WPS432
+        assert len(result_no_ff.settings) == 13  # noqa: WPS432
 
         # Feature flag on means definition is indeed retrieved from lockfile
         subject.settings_service.set(FeatureFlags.LOCKFILES.setting_name, True)
diff --git a/tests/meltano/core/test_project_settings_service.py b/tests/meltano/core/test_project_settings_service.py
--- a/tests/meltano/core/test_project_settings_service.py
+++ b/tests/meltano/core/test_project_settings_service.py
@@ -7,8 +7,10 @@
 from meltano.core.settings_service import (
     EXPERIMENTAL,
     FEATURE_FLAG_PREFIX,
+    FeatureFlags,
     FeatureNotAllowedException,
 )
+from meltano.core.utils import EnvironmentVariableNotSetError
 
 
 @pytest.fixture
@@ -110,3 +112,20 @@ def should_not_run():
 
         with pytest.raises(FeatureNotAllowedException):
             should_not_run()
+
+    def test_strict_env_var_mode_on_raises_error(self, subject):
+        subject.set([FEATURE_FLAG_PREFIX, str(FeatureFlags.STRICT_ENV_VAR_MODE)], True)
+        subject.set(
+            "stacked_env_var",
+            "${NONEXISTENT_ENV_VAR}@nonexistent",
+        )
+        with pytest.raises(EnvironmentVariableNotSetError):
+            subject.get("stacked_env_var")
+
+    def test_strict_env_var_mode_off_no_raise_error(self, subject):
+        subject.set([FEATURE_FLAG_PREFIX, str(FeatureFlags.STRICT_ENV_VAR_MODE)], False)
+        subject.set(
+            "stacked_env_var",
+            "${NONEXISTENT_ENV_VAR}@nonexistent_1",
+        )
+        assert subject.get("stacked_env_var") == "@nonexistent_1"