Repository: meltano/meltano

Issue:
bug: Configuration values for `hidden`-type settings from environment variables are always `string` implicitly 
### Meltano Version

2.16.1

### Python Version

3.7

### Bug scope

Configuration (settings parsing, validation, etc.)

### Operating System

Linux - Ubuntu 22.04

### Description

### Overview
When providing configuration for a plugin from environment variables (shell or `.env`), if the corresponding setting is of `kind` `hidden`, (in `meltano.yml` or plugin `.lock` file), the value is always handled as a string (as if the setting were of `kind` `string`). 

In the case of [`tap-auth0`](https://hub.meltano.com/extractors/tap-auth0/), trying to supply an override for the default value of `job_poll_interval_ms` as an environment variable (`TAP_AUTH0_JOB_POLL_INTERVAL_MS`) causes configuration schema validation to fail (see traceback).

### Fix
Since the Meltano UI is no longer supported, I don't know how much value updating the setting definition to replace `hidden` as a `kind` with `hidden` as a separate boolean flag holds for Meltano (not sure if it is used anywhere else other than for the UI).

If `hidden` is to be deprecated or removed eventually, a nice interim solution would be to infer the type from the setting default value (if present). 

Happy to open a PR for either of these.

### Code

`meltano.yml`
```
version: 1
send_anonymous_usage_stats: true
project_id: tap-auth0
default_environment: test
environments:
- name: test
plugins:
  extractors:
  - name: tap-auth0
    namespace: tap_auth0
    pip_url: -e .
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: client_id
      kind: password
    - name: client_secret
      kind: password
    - name: domain
    - name: job_poll_interval_ms
      kind: hidden
    - name: job_poll_max_count
      kind: hidden
```

`.env`
```
# required settings
TAP_AUTH0_CLIENT_ID=<client id>
TAP_AUTH0_CLIENT_SECRET=<client secret>
TAP_AUTH0_DOMAIN=<domain>

# optional integer setting
TAP_AUTH0_JOB_POLL_INTERVAL_MS=5000
```

```
$ meltano invoke tap-auth0
Traceback (most recent call last):
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/bin/tap-auth0", line 8, in <module>
    sys.exit(TapAuth0.cli())
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/singer_sdk/tap_base.py", line 497, in cli
    validate_config=validate_config,
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/singer_sdk/tap_base.py", line 73, in __init__
    validate_config=validate_config,
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/singer_sdk/plugin_base.py", line 121, in __init__
    self._validate_config(raise_errors=validate_config)
  File "/home/reuben/Documents/taps/tap-auth0/.meltano/extractors/tap-auth0/venv/lib/python3.7/site-packages/singer_sdk/plugin_base.py", line 257, in _validate_config
    raise ConfigValidationError(summary)
singer_sdk.exceptions.ConfigValidationError: Config validation failed: '5000' is not of type 'integer', 'null'
JSONSchema was: {'type': 'object', 'properties': {'client_id': {'type': ['string'], 'description': 'App client ID'}, 'client_secret': {'type': ['string'], 'description': 'App client secret'}, 'domain': {'type': ['string'], 'description': 'Tenant domain'}, 'job_poll_interval_ms': {'type': ['integer', 'null'], 'default': 2000, 'description': 'Job poll interval (ms)'}, 'job_poll_max_count': {'type': ['integer', 'null'], 'default': 10, 'description': 'Maximum job poll count'}, 'stream_maps': {'type': ['object', 'null'], 'properties': {}, 'description': 'Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html).'}, 'stream_map_config': {'type': ['object', 'null'], 'properties': {}, 'description': 'User-defined config values to be used within map expressions.'}, 'flattening_enabled': {'type': ['boolean', 'null'], 'description': "'True' to enable schema flattening and automatically expand nested properties."}, 'flattening_max_depth': {'type': ['integer', 'null'], 'description': 'The max depth to flatten schemas.'}}, 'required': ['client_id', 'client_secret', 'domain']}
```
```



Discussion:
@ReubenFrankel These issues are related:

* https://github.com/meltano/hub/issues/24
* https://github.com/meltano/sdk/issues/347

I hate that we have `kind: hidden`. It really needs to be just about the datatype. 

I think the proposal in https://github.com/meltano/hub/issues/24#issuecomment-1195835961 would work for me. Specifically:

```yaml
- name: pg_password
  kind: string
  sensitive: true
  hidden: true
```
@tayloramurphy Ah yes, I knew that `sensitive` issue was lying around somewhere. Do you think that makes sense for the above use-case though? The setting isn't really "sensitive" - we just want to hide the field in our UI (hence `kind: hidden`).

From https://github.com/meltano/sdk/issues/347:

> ## Example
> ### `oauth_credentials.access_token` Setting
>  Configuration for an `oauth_credentials.access_token` setting should be both hidden from the user, and stored encrypted.
>
>  In this case we need
>
>  kind: password
>  hidden: true
>  ### `oauth_credentials.scope` Setting
>  Configuration for an `oauth_credentials.scope` setting is not sensitive, but does not need to be shown to the user for configuration.
>  
>  In this case we need
>  
>  kind: string
>  hidden: true

@ReubenFrankel good call out. I don't have any opposition to having `hidden: true` be an additional key. I would agree that sensitive / hidden have different meanings and use cases. I updated my example to include it as well.
@tayloramurphy So `sensitive` is more-or-less a proposal to replace `kind: password`? I guess we haven't run into a problem with that in particular since most passwords resolve to a string anyway. I can definitely see a potential use-case for a non-string sensitive credential like a PIN or code, which currently is not possible due to conflict between `kind: integer` and `kind: password`.

Would you accept a PR to add new `sensitive` and `hidden` fields as drop-in replacements for the corresponding kinds (which could then be deprecated)?

`sensitive` I feel would be quite an involved change if it ties into a enhancement relating to how setting values are stored (i.e. requiring encryption/decryption) and displayed to the user from the CLI (e.g. `meltano config <plugin> list` or `meltano config set --interactive`). If this is the case, perhaps it's better to split the two changes up for now.
@ReubenFrankel let's keep the PR's small since changes like this may have larger effects than we intend. Pulling out `hidden` as a separate field would be a good first iteration and would inform us about what we're likely to run into for `password` vs `sensitive`. 

Also, we should do this in a backwards compatible way until we release 3.0 (which we're planning for same time as Python 3.7 EOL). cc @WillDaSilva 

Resolving patch:
diff --git a/src/meltano/core/plugin/base.py b/src/meltano/core/plugin/base.py
--- a/src/meltano/core/plugin/base.py
+++ b/src/meltano/core/plugin/base.py
@@ -4,6 +4,7 @@
 
 import logging
 import re
+from collections import defaultdict
 
 import yaml
 
@@ -13,7 +14,7 @@
 from meltano.core.job_state import STATE_ID_COMPONENT_DELIMITER
 from meltano.core.plugin.command import Command
 from meltano.core.plugin.requirements import PluginRequirement
-from meltano.core.setting_definition import SettingDefinition, YAMLEnum
+from meltano.core.setting_definition import SettingDefinition, SettingKind, YAMLEnum
 from meltano.core.utils import NotFound, find_named
 
 logger = logging.getLogger(__name__)
@@ -762,7 +763,7 @@ class StandalonePlugin(Canonical):
 
     def __init__(
         self,
-        plugin_type: PluginType,
+        plugin_type: str,
         name: str,
         namespace: str,
         variant: str | None = None,
@@ -804,7 +805,7 @@ def __init__(
             extras: Additional attributes to set on the plugin.
         """
         super().__init__(
-            plugin_type=plugin_type,
+            plugin_type=PluginType(plugin_type),
             name=name,
             namespace=namespace,
             variant=variant,
@@ -824,6 +825,34 @@ def __init__(
             extras=extras,
         )
 
+        deprecated_kind_replacements = {
+            SettingKind.HIDDEN: "hidden: true",
+        }
+
+        settings_by_deprecated_kind = defaultdict(list)
+
+        for s in self.settings:
+            if s.kind in deprecated_kind_replacements:
+                settings_by_deprecated_kind[s.kind].append(s)
+
+        if settings_by_deprecated_kind:
+            for kind, settings in settings_by_deprecated_kind.items():
+                logger.warning(
+                    f"`kind: {kind}` is deprecated for setting definitions"
+                    + (
+                        f" in favour of `{deprecated_kind_replacements[kind]}`"
+                        if deprecated_kind_replacements[kind]
+                        else ""
+                    )
+                    + ", and is currently in use by the following settings of "
+                    + f"{self.plugin_type.singular} '{self.name}': "
+                    + ", ".join([f"'{s.name}'" for s in settings])
+                    + ". "
+                    + "Please open an issue or pull request to update the plugin "
+                    + "definition on Meltano Hub at "
+                    + f"https://github.com/meltano/hub/blob/main/_data/meltano/{self.plugin_type}/{self.name}/{self.variant}.yml.",  # noqa: E501
+                )
+
     @classmethod
     def from_variant(
         cls: type[StandalonePlugin],
diff --git a/src/meltano/core/setting_definition.py b/src/meltano/core/setting_definition.py
--- a/src/meltano/core/setting_definition.py
+++ b/src/meltano/core/setting_definition.py
@@ -161,7 +161,7 @@ def __init__(
         aliases: list[str] | None = None,
         env: str | None = None,
         env_aliases: list[str] | None = None,
-        kind: SettingKind | None = None,
+        kind: str | None = None,
         value=None,
         label: str | None = None,
         documentation: str | None = None,
@@ -172,6 +172,7 @@ def __init__(
         placeholder: str | None = None,
         protected: bool | None = None,
         env_specific: bool | None = None,
+        hidden: bool | None = None,
         custom: bool = False,
         value_processor=None,
         value_post_processor=None,
@@ -196,6 +197,7 @@ def __init__(
             placeholder: A placeholder value for this setting.
             protected: A protected setting cannot be changed from the UI.
             env_specific: Flag for environment-specific setting.
+            hidden: Hidden setting.
             custom: Custom setting flag.
             value_processor: Used with `kind: object` to pre-process the keys
                 in a particular way.
@@ -208,13 +210,16 @@ def __init__(
         options = options or []
         oauth = oauth or {}
 
+        kind = SettingKind(kind) if kind else None
+        hidden = hidden or kind is SettingKind.HIDDEN or None
+
         super().__init__(
             # Attributes will be listed in meltano.yml in this order:
             name=name,
             aliases=aliases,
             env=env,
             env_aliases=env_aliases,
-            kind=SettingKind(kind) if kind else None,
+            kind=kind,
             value=value,
             label=label,
             documentation=documentation,
@@ -225,6 +230,7 @@ def __init__(
             placeholder=placeholder,
             protected=protected,
             env_specific=env_specific,
+            hidden=hidden,
             value_processor=value_processor,
             value_post_processor=value_post_processor,
             _custom=custom,
diff --git a/src/meltano/core/settings_service.py b/src/meltano/core/settings_service.py
--- a/src/meltano/core/settings_service.py
+++ b/src/meltano/core/settings_service.py
@@ -590,7 +590,7 @@ def definitions(self, extras=None) -> t.Iterable[dict]:
             self._setting_defs = [
                 setting
                 for setting in self.setting_definitions
-                if setting.kind != SettingKind.HIDDEN or self.show_hidden
+                if not setting.hidden or self.show_hidden
             ]
 
         if extras is not None:


Test patch: