Repository: meltano/meltano

Issue:
bug: Cannot set integer settings with environment variables in meltano.yml
### Meltano Version

2.15.2

### Python Version

3.8

### Bug scope

Configuration (settings parsing, validation, etc.)

### Operating System

Linux/Docker (using the image `meltano:v2.15.2-python3.8` pushed a few hours ago)

### Description

I have been using an environment variable to configure the port of my mysql tap with something like the below in my meltano.yml.

```yaml
plugins:
  extractors:
    - name: tap-mysql
      variant: transferwise
      pip_url: pipelinewise-tap-mysql
      config:
        host: '${MY_TAP_HOST}'
        port: '${MY_TAP_PORT}'
```

This use to work but appears to be broken now, presumably due to https://github.com/meltano/meltano/pull/7232. Now you'll get an error ending with

```
│ /venv/lib/python3.8/site-packages/meltano/core/setting_definition.py:374 in cast_value           │
│                                                                                                  │
│   371 │   │   │   if self.kind == SettingKind.BOOLEAN:                                           │
│   372 │   │   │   │   return utils.truthy(value)                                                 │
│   373 │   │   │   elif self.kind == SettingKind.INTEGER:                                         │
│ ❱ 374 │   │   │   │   return int(value)                                                          │
│   375 │   │   │   elif self.kind == SettingKind.OBJECT:                                          │
│   376 │   │   │   │   value = json.loads(value)                                                  │
│   377 │   │   │   │   if not isinstance(value, dict):                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: invalid literal for int() with base 10: '${MY_TAP_PORT}'
```

I realize there are other ways I could accomplish this, e.g. setting TAP_MYSQL_PORT and leave this out of meltano.yml. I mostly want to confirm if this is intended behavior and see if there are any workarounds that allow me to express something similar within meltano.yml.

### Code

_No response_


Discussion:
Thanks for raising @msg555 !

We're looking into this and we expect to have a fix within 24 hours. Agreed that this appears to be a regression related to the boolean fix in https://github.com/meltano/meltano/pull/7232.

Are you able to downgrade to the prior version in the meanwhile?

Resolving patch:
diff --git a/src/meltano/core/settings_service.py b/src/meltano/core/settings_service.py
--- a/src/meltano/core/settings_service.py
+++ b/src/meltano/core/settings_service.py
@@ -351,7 +351,8 @@ def get_with_metadata(  # noqa: WPS210, WPS615
         # Can't do conventional SettingsService.feature_flag call to check;
         # it would result in circular dependency
         strict_env_var_mode, _ = source.manager(self.project_settings_service).get(
-            f"{FEATURE_FLAG_PREFIX}.{FeatureFlags.STRICT_ENV_VAR_MODE}"
+            f"{FEATURE_FLAG_PREFIX}.{FeatureFlags.STRICT_ENV_VAR_MODE}",
+            cast_value=True,
         )
         if expand_env_vars and metadata.get("expandable", False):
             metadata["expandable"] = False
@@ -378,10 +379,7 @@ def get_with_metadata(  # noqa: WPS210, WPS615
             ):
                 object_value = {}
                 object_source = metadata["source"]
-                for setting_key in [  # noqa: WPS335
-                    setting_def.name,
-                    *setting_def.aliases,
-                ]:
+                for setting_key in (setting_def.name, *setting_def.aliases):
                     flat_config_metadata = self.config_with_metadata(
                         prefix=f"{setting_key}.",
                         redacted=redacted,
diff --git a/src/meltano/core/settings_store.py b/src/meltano/core/settings_store.py
--- a/src/meltano/core/settings_store.py
+++ b/src/meltano/core/settings_store.py
@@ -220,12 +220,18 @@ def __init__(self, settings_service: SettingsService, **kwargs):
         self.project = self.settings_service.project
 
     @abstractmethod
-    def get(self, name: str, setting_def: SettingDefinition | None = None) -> None:
+    def get(
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
+    ) -> None:
         """Abstract get method.
 
         Args:
             name: Setting name.
             setting_def: SettingDefinition instance.
+            cast_value: Whether to cast the value according to `setting_def`.
         """
 
     def set(
@@ -301,13 +307,17 @@ class ConfigOverrideStoreManager(SettingsStoreManager):
     label = "a command line flag"
 
     def get(
-        self, name: str, setting_def: SettingDefinition | None = None
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
     ) -> tuple[str, dict]:
         """Get value by name from the .env file.
 
         Args:
             name: Setting name.
             setting_def: Unused. Included to match parent class method signature.
+            cast_value: Whether to cast the value according to `setting_def`.
 
         Returns:
             A tuple the got value and an empty dictionary.
@@ -329,13 +339,17 @@ def env(self):
         """Abstract environment values property."""
 
     def get(
-        self, name: str, setting_def: SettingDefinition | None = None
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
     ) -> tuple[str, dict]:
         """Get value by name from the .env file.
 
         Args:
             name: Unused. Included to match parent class method signature.
             setting_def: SettingDefinition instance.
+            cast_value: Whether to cast the value according to `setting_def`.
 
         Raises:
             StoreNotSupportedError: if setting_def not passed.
@@ -368,7 +382,11 @@ def get(
             )
 
         value, metadata = vals_with_metadata[0] if vals_with_metadata else (None, {})
-        return cast_setting_value(value, metadata, setting_def)
+        return (
+            cast_setting_value(value, metadata, setting_def)
+            if cast_value
+            else (value, metadata)
+        )
 
     def setting_env_vars(self, *args, **kwargs) -> dict:
         """Return setting environment variables.
@@ -603,19 +621,24 @@ def ensure_supported(self, method: str = "get") -> None:
             raise StoreNotSupportedError(ProjectReadonly())
 
     def get(
-        self, name: str, setting_def: SettingDefinition | None = None
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
     ) -> tuple[str, dict]:
         """Get value by name from the system database.
 
         Args:
             name: Setting name.
             setting_def: SettingDefinition.
+            cast_value: Whether to cast the value according to `setting_def`.
 
         Returns:
             A tuple the got value and a dictionary containing metadata.
 
         Raises:
-            ConflictingSettingValueException: if multiple conflicting values for the same setting are provided.
+            ConflictingSettingValueException: if multiple conflicting values
+                for the same setting are provided.
         """
         keys = [setting_def.name, *setting_def.aliases] if setting_def else [name]
         flat_config = self.flat_config
@@ -637,7 +660,11 @@ def get(
             )
 
         value, metadata = vals_with_metadata[0] if vals_with_metadata else (None, {})
-        return cast_setting_value(value, metadata, setting_def)
+        return (
+            cast_setting_value(value, metadata, setting_def)
+            if cast_value
+            else (value, metadata)
+        )
 
     def set(
         self,
@@ -869,13 +896,17 @@ def ensure_supported(self, method: str = "get") -> None:
             raise StoreNotSupportedError("No database session provided")
 
     def get(
-        self, name: str, setting_def: SettingDefinition | None = None
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
     ) -> tuple[str, dict]:
         """Get value by name from the system database.
 
         Args:
             name: Setting name.
             setting_def: Unused. Included to match parent class method signature.
+            cast_value: Whether to cast the value according to `setting_def`.
 
         Returns:
             A tuple the got value and an empty dictionary.
@@ -1013,13 +1044,17 @@ def __init__(
         self._config_with_metadata = None
 
     def get(
-        self, name: str, setting_def: SettingDefinition | None = None
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
     ) -> tuple[str, dict]:
         """Get a Setting value by name and SettingDefinition.
 
         Args:
             name: Setting name.
             setting_def: SettingDefinition.
+            cast_value: Whether to cast the value according to `setting_def`.
 
         Returns:
             A tuple containing the got value and accompanying metadata dictionary.
@@ -1087,13 +1122,17 @@ class DefaultStoreManager(SettingsStoreManager):
     label = "the default"
 
     def get(
-        self, name: str, setting_def: SettingDefinition | None = None
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
     ) -> tuple[str, dict]:
         """Get a Setting value by name and SettingDefinition.
 
         Args:
             name: Setting name.
             setting_def: SettingDefinition.
+            cast_value: Whether to cast the value according to `setting_def`.
 
         Returns:
             A tuple containing the got value and accompanying metadata dictionary.
@@ -1246,14 +1285,20 @@ def auto_store(  # noqa: WPS231 # Too complex
         return None
 
     def get(
-        self, name: str, setting_def: SettingDefinition | None = None, **kwargs
+        self,
+        name: str,
+        setting_def: SettingDefinition | None = None,
+        cast_value: bool = False,
+        **kwargs,
     ) -> tuple[str, dict]:
         """Get a Setting value by name and SettingDefinition.
 
         Args:
             name: Setting name.
-            setting_def: SettingDefinition. If none is passed, one will be discovered using `self.find_setting(name)`.
-            kwargs: Additional keword arguments to pass to `manager.get()`
+            setting_def: SettingDefinition. If none is passed, one will be
+                discovered using `self.find_setting(name)`.
+            cast_value: Whether to cast the value according to `setting_def`.
+            kwargs: Additional keword arguments to pass to `manager.get()`.
 
         Returns:
             A tuple containing the got value and accompanying metadata dictionary.
@@ -1271,7 +1316,12 @@ def get(
                 continue
 
             try:
-                value, metadata = manager.get(name, setting_def=setting_def, **kwargs)
+                value, metadata = manager.get(
+                    name,
+                    setting_def=setting_def,
+                    cast_value=cast_value,
+                    **kwargs,
+                )
             except StoreNotSupportedError:
                 continue
 
@@ -1287,7 +1337,11 @@ def get(
             metadata["auto_store"] = auto_store
             metadata["overwritable"] = auto_store.can_overwrite(found_source)
 
-        return cast_setting_value(value, metadata, setting_def)
+        return (
+            cast_setting_value(value, metadata, setting_def)
+            if cast_value
+            else (value, metadata)
+        )
 
     def set(self, name: str, path: list[str], value, setting_def=None) -> dict:
         """Set a Setting by name, path and (optionally) SettingDefinition.


Test patch:
diff --git a/tests/meltano/core/test_project_settings_service.py b/tests/meltano/core/test_project_settings_service.py
--- a/tests/meltano/core/test_project_settings_service.py
+++ b/tests/meltano/core/test_project_settings_service.py
@@ -197,3 +197,22 @@ def test_fully_missing_env_var_setting_is_none(
         with pytest.warns(RuntimeWarning, match="Unknown setting 'port'"):
             subject.set("port", "${UNSET_PORT_ENV_VAR}")
         assert subject.get("port") is None
+
+    def test_env_var_settings_expanded_before_cast(
+        self, subject: ProjectSettingsService
+    ):
+        name = "database_max_retries"  # Using this because it's an int setting
+        setting_def = subject.find_setting(name)
+
+        subject.set(name, setting_def.value, setting_def=setting_def)
+        assert subject.get(name) == setting_def.value
+
+        SettingValueStore.MELTANO_YML.manager(subject).set(
+            name, [name], "$DB_MAX_RETRIES_TEST", setting_def=setting_def
+        )
+
+        subject.set([FEATURE_FLAG_PREFIX, str(FeatureFlags.STRICT_ENV_VAR_MODE)], False)
+        assert subject.get(name) is None
+
+        subject.env_override["DB_MAX_RETRIES_TEST"] = "7"
+        assert subject.get(name) == 7