Repository: meltano/meltano

Issue:
Add compile-time verification of built-in Meltano settings names
Via enum or constants or a similar approach, we should ensure that functionality is not dependent upon correct spelling of string settings.

This is in follow up to a bug that was discovered on https://github.com/meltano/meltano/pull/6199#issuecomment-1159235493 due to misspelling of `disable_tracking` vs `tracking_disabled`.

There'd be two parts to this:

1. Adopt the approach we are using now for naming feature flags, which ensures that mispellings are not possible without a compile-time failure.
2. Create a unit test to ensure that the `settings.yml` filed does not include anything that the enum doesn't, and vice versa.



Discussion:
@aaronsteers If an enum/constants were added for this purpose, what's to stop devs from ignoring them and using strings (with possible typos) instead? We are able to check if an enum is passed into `SettingsService.get_with_metadata`, and warn if a string is used instead, but that wouldn't work because more than just what is defined in `settings.yml` is accessed with `SettingsService.get_with_metadata`. We cannot distinguish between someone passing in a valid string name of some setting not in `settings.yml`, and someone passing in a misspelled string name of a setting from `settings.yml`.

Because of that, the ideal here (i.e. "compile-time verification") seems impossible, unless I'm missing something. An enum could still be added, but it wouldn't help in the case of something like `disable_tracking` versus `tracking_disabled` since one could just use a string instead of the enum.
@WillDaSilva - it's a good point but I was thinking we could also modify the type signature on the get method to accept/expect the enum class instead of a string. While it might still "work" by passing a string, we should be able to get mypy to fail CI checks in those cases where a string is passed directly. (And the dev would see a linter warning in their IDE.)

What do you think?

Update: I see your comment here and this is something I wasn't expecting or considering:

> more than just what is defined in settings.yml is accessed with SettingsService.get_with_metadata

So, I guess this depends on the alternate code paths and what the requirements are there.
@aaronsteers I think we're locked into accepting strings in `SettingsService.get_with_metadata` and all of the functions that ultimately call it, but maybe we're going about this the wrong way. What we ultimately want here is an assurance that we'll know if a setting name is misspelled. We can alter the first few lines of `SettingsService.get_with_metadata` like so:

```python
        try:
            setting_def = setting_def or self.find_setting(name)
        except SettingMissingError:
            warn(name)
```

This emits a warning whenever a setting uses the default value, which is true whenever a setting name as been misspelled. Running the tests with this change produces just 4 warnings:

```
tests/meltano/core/test_project_settings_service.py::TestProjectSettingsService::test_feature_flag_allowed
  /home/will/meltano/meltano2/src/meltano/core/settings_service.py:335: UserWarning: ff.allowed
    warn(name)

tests/meltano/core/test_project_settings_service.py::TestProjectSettingsService::test_feature_flag_disallowed
  /home/will/meltano/meltano2/src/meltano/core/settings_service.py:335: UserWarning: ff.disallowed
    warn(name)

tests/meltano/core/test_project_settings_service.py::TestProjectSettingsService::test_strict_env_var_mode_on_raises_error
tests/meltano/core/test_project_settings_service.py::TestProjectSettingsService::test_strict_env_var_mode_off_no_raise_error
  /home/will/meltano/meltano2/src/meltano/core/settings_service.py:335: UserWarning: stacked_env_var
    warn(name)
```

We can probably update those 4 calls to check before calling, or explicitly request the default with `source=SettingValueStore.DEFAULT`. Then we can emit a warning whenever a setting is misspelled or otherwise not found. Since our tests no longer emit any warnings as of #6293, a warning raised for this reason should be fairly noticeable, and would greatly reduce the odds of us merging in any changes with a misspelling in a setting name.

We can also go further by adding a keyword argument `strict=False`, which when `True` to raise an exception instead of a warning. In a future version of Meltano we can make a breaking change by making the keyword argument default to `True` instead.
I just checked the codebase. Most of the actual instances of string refs are using `.get(` and I see 34 results for as search for `settings_service.get("` which is the get() method and the start of a quoted string. At least a couple others like [this one](https://github.com/meltano/meltano/blob/a852de1e1a88d15a3a62356ffcc6c8f9ca8164d6/src/meltano/api/app.py#L121-L136) have dicts declared immediately above the call, with bare string values which are iterated on through multiple calls to `settings_service.get(`.

We could take a couple different directions here, such as a new named command for `get_builtin()` or similar that only takes a type of the enum class, but I do think it's worth translating these from bare strings to enums, and adding a unit test to make sure the list of known enum values are exactly the same set as we declare in `settings.yaml`. (Extras on either side should probably indicate an error.)

Even if we don't end up forcing the behavior, I think it would be valuable to have a best practice and some kind of checks to catch our typos. Since some of these are very hard to test behavior for, we could gaps today and not know it.
I considered this approach, as well as using a keyword argument to force an enum value to be provided, but they suffer from the problem of being easily ignorable. By using the approach I describe in https://github.com/meltano/meltano/issues/6234#issuecomment-1171583989 we can't ignore typos, as they result in warnings/exceptions being raised.

I can still add an enum for the builtin settings, but that would only be there for the benefit of our editor autocompletors.

Resolving patch:
diff --git a/src/meltano/cli/config.py b/src/meltano/cli/config.py
--- a/src/meltano/cli/config.py
+++ b/src/meltano/cli/config.py
@@ -37,11 +37,12 @@ def get_label(metadata) -> str:
         string describing the source of the variable's value
     """
     source = metadata["source"]
-    if "env_var" in metadata:
+    try:
         return f"from the {metadata['env_var']} variable in {source.label}"
-    else:
+    except KeyError:
         return f"from {source.label}"
 
+
 @cli.group(
     cls=InstrumentedGroup,
     invoke_without_command=True,
diff --git a/src/meltano/core/settings_service.py b/src/meltano/core/settings_service.py
--- a/src/meltano/core/settings_service.py
+++ b/src/meltano/core/settings_service.py
@@ -7,6 +7,7 @@
 from contextlib import contextmanager
 from enum import Enum
 from typing import Generator, Iterable
+from warnings import warn
 
 from meltano.core.project import Project
 from meltano.core.utils import expand_env_vars as do_expand_env_vars
@@ -51,7 +52,7 @@ def setting_name(self) -> str:
         return f"{FEATURE_FLAG_PREFIX}.{self.value}"
 
 
-class FeatureNotAllowedException(Exception):
+class FeatureNotAllowedException(Exception):  # noqa: N818
     """Occurs when a disallowed code path is run."""
 
     def __init__(self, feature):
@@ -305,7 +306,7 @@ def as_env(self, *args, **kwargs) -> dict[str, str]:
 
         return env
 
-    def get_with_metadata(  # noqa: WPS210
+    def get_with_metadata(  # noqa: WPS210, WPS615
         self,
         name: str,
         redacted=False,
@@ -415,7 +416,14 @@ def get_with_metadata(  # noqa: WPS210
                 metadata["redacted"] = True
                 value = REDACTED_VALUE
 
-        self.log(f"Got setting '{name}' with metadata: {metadata}")
+        self.log(f"Got setting {name!r} with metadata: {metadata}")
+
+        if setting_def is None and metadata["source"] is SettingValueStore.DEFAULT:
+            warn(
+                f"Unknown setting {name!r} - the default value `{value!r}` will be used",
+                RuntimeWarning,
+            )
+
         return value, metadata
 
     def get_with_source(self, *args, **kwargs):
@@ -444,7 +452,7 @@ def get(self, *args, **kwargs):
         value, _ = self.get_with_source(*args, **kwargs)
         return value
 
-    def set_with_metadata(
+    def set_with_metadata(  # noqa: WPS615
         self, path: list[str], value, store=SettingValueStore.AUTO, **kwargs
     ):
         """Set the value and metadata for a setting.


Test patch:
diff --git a/tests/meltano/core/test_project_settings_service.py b/tests/meltano/core/test_project_settings_service.py
--- a/tests/meltano/core/test_project_settings_service.py
+++ b/tests/meltano/core/test_project_settings_service.py
@@ -1,3 +1,5 @@
+import warnings
+
 import pytest
 
 from meltano.core.project_settings_service import (
@@ -129,3 +131,37 @@ def test_strict_env_var_mode_off_no_raise_error(self, subject):
             "${NONEXISTENT_ENV_VAR}@nonexistent_1",
         )
         assert subject.get("stacked_env_var") == "@nonexistent_1"
+
+    def test_warn_if_default_setting_is_used(self, subject, monkeypatch):
+        # Assert that warnings are not raised in the following cases:
+        with warnings.catch_warnings():
+            warnings.simplefilter("error")
+
+            for ff in FeatureFlags:
+                subject.get(f"{FEATURE_FLAG_PREFIX}.{ff}")
+
+            subject.get("project_id")
+            subject.get("disable_tracking")
+
+            ProjectSettingsService.config_override.pop(
+                "send_anonymous_usage_stats", None
+            )
+            subject.config_override.pop("send_anonymous_usage_stats", None)
+            subject.unset("send_anonymous_usage_stats")
+
+            monkeypatch.setenv("MELTANO_SEND_ANONYMOUS_USAGE_STATS", "True")
+            assert subject.get("send_anonymous_usage_stats")
+            monkeypatch.setenv("MELTANO_SEND_ANONYMOUS_USAGE_STATS", "False")
+            assert not subject.get("send_anonymous_usage_stats")
+            monkeypatch.delenv("MELTANO_SEND_ANONYMOUS_USAGE_STATS")
+            subject.get("send_anonymous_usage_stats")
+
+        # Assert that warnings are raised in the following cases:
+        for setting in (
+            "projectID",
+            "tracking_disabled",
+            "MELTANO_SEND_ANONYMOUS_USAGE_STATS",
+            "send_anon_usage_stats",
+        ):
+            with pytest.warns(RuntimeWarning):
+                subject.get(setting)
diff --git a/tests/meltano/core/tracking/test_tracker.py b/tests/meltano/core/tracking/test_tracker.py
--- a/tests/meltano/core/tracking/test_tracker.py
+++ b/tests/meltano/core/tracking/test_tracker.py
@@ -258,17 +258,17 @@ def test_can_track(
         platform.system() == "Windows",
         reason="Doesn't pass on windows, this is currently being tracked here https://github.com/meltano/meltano/issues/3444",
     )
-    def test_send_anonymous_usage_stats(self, project: Project):
+    def test_send_anonymous_usage_stats(self, project: Project, monkeypatch):
         clear_telemetry_settings(project)
 
-        os.environ["MELTANO_SEND_ANONYMOUS_USAGE_STATS"] = "True"
+        monkeypatch.setenv("MELTANO_SEND_ANONYMOUS_USAGE_STATS", "True")
         assert Tracker(project).send_anonymous_usage_stats is True
 
         # Ensure the env var takes priority
         ProjectSettingsService(project).set("send_anonymous_usage_stats", False)
         assert Tracker(project).send_anonymous_usage_stats is True
 
-        os.environ["MELTANO_SEND_ANONYMOUS_USAGE_STATS"] = "False"
+        monkeypatch.setenv("MELTANO_SEND_ANONYMOUS_USAGE_STATS", "False")
         assert Tracker(project).send_anonymous_usage_stats is False
 
         # Ensure the env var takes priority