Repository: meltano/meltano

Issue:
bug:  Regression in `meltano remove`
### Meltano Version

2.3.0

### Python Version

NA

### Bug scope

CLI (options, error messages, logging, etc.)

### Operating System

NA

### Description

Meltano `2.3.0` introduced a regression on `meltano remove` where it complains that the plugin is missing the `settings_with_extras` attribute.

The regression was most likely introduced in https://github.com/meltano/meltano/pull/6267 somewhere in the use of a feature flag:

https://github.com/meltano/meltano/blob/23a9099cefe148cb560d25e7ae335447858948df/src/meltano/core/plugin/settings_service.py#L63-L65

I still haven't figured out the root cause but might be related with trying to check for feature flags in the `PluginSettingsService`.

Other commands may be similarly affected, but at least `invoke` and `run` seem to be working fine.

### Code


<details><summary>Traceback</summary>

```
Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/behavior/canonical.py", line 138, in __getattr__
    value = self._dict[attr]
KeyError: 'settings_with_extras'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/__init__.py", line 66, in _run_cli
    cli(obj={"project": None})
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/cli.py", line 35, in main
    return super().main(*args, windows_expand_args=False, **kwargs)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/utils.py", line 540, in invoke
    super().invoke(ctx)  # noqa: WPS608
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/params.py", line 18, in decorate
    return func(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/params.py", line 53, in decorate
    func(project, *args, **kwargs)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/remove.py", line 32, in remove
    remove_plugins(project, plugins)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/remove.py", line 39, in remove_plugins
    num_removed, total = remove_service.remove_plugins(
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_remove_service.py", line 60, in remove_plugins
    removal_managers = self.remove_plugin(plugin)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_remove_service.py", line 87, in remove_plugin
    DbRemoveManager(plugin, self.project),
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_location_remove.py", line 86, in __init__
    self.plugins_settings_service = PluginSettingsService(project, plugin)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin/settings_service.py", line 63, in __init__
    with self.feature_flag(
  File "/Users/edgarramirez/.pyenv/versions/3.9.13/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 657, in feature_flag
    allowed = self.get(f"{FEATURE_FLAG_PREFIX}.{feature}") or False
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 444, in get
    value, _ = self.get_with_source(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 431, in get_with_source
    value, metadata = self.get_with_metadata(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 333, in get_with_metadata
    setting_def = setting_def or self.find_setting(name)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 598, in find_setting
    for setting in self.definitions()
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 568, in definitions
    for setting in self.setting_definitions
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin/settings_service.py", line 136, in setting_definitions
    settings = self.plugin.settings_with_extras
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/behavior/canonical.py", line 143, in __getattr__
    raise AttributeError(attr) from err
AttributeError: settings_with_extras

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/__init__.py", line 74, in _run_cli
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: settings_with_extras
settings_with_extras
```

</details>




Discussion:
Thanks for opening @edgarrmondragon. @aaronsteers I'm assigning to Iteration 10 as a fixing regressions should be a high priority.
As noted in #6656 [here](https://github.com/meltano/meltano/pull/6656#discussion_r953147703) this may be fixed by that PR. Will verify the fixed behaviour once #6656 merges 👍
@kgpayne No luck. #6656 didn't fix this. Seeing:

<details><summary>Traceback</summary>

```console
$ meltano remove extractor tap-gitlab
2022-08-24T16:31:17.519641Z [info     ] Environment 'dev' is active
2022-08-24T16:31:18.183907Z [debug    ] Creating engine <meltano.core.project.Project object at 0x104d652d0>@sqlite:////Users/edgarramirez/edgar/meltano-dataops/.meltano/meltano.db

Removing extractor 'tap-gitlab'...

2022-08-24T16:31:19.861278Z [debug    ] settings_with_extras           
Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/behavior/canonical.py", line 177, in __getattr__
    value = self._dict[attr]
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.10/lib/python3.10/site-packages/ruamel/yaml/comments.py", line 927, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'settings_with_extras'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/__init__.py", line 66, in _run_cli
    cli(obj={"project": None})
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/cli.py", line 38, in main
    return super().main(*args, windows_expand_args=False, **kwargs)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/utils.py", line 541, in invoke
    super().invoke(ctx)  # noqa: WPS608
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.10/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/utils.py", line 559, in invoke
    super().invoke(ctx)  # noqa: WPS608
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.10/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.10/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/params.py", line 29, in decorate
    return func(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/params.py", line 78, in decorate
    func(project, *args, **kwargs)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.10/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/remove.py", line 34, in remove
    remove_plugins(project, plugins)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/remove.py", line 41, in remove_plugins
    num_removed, total = remove_service.remove_plugins(
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_remove_service.py", line 63, in remove_plugins
    removal_managers = self.remove_plugin(plugin)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_remove_service.py", line 90, in remove_plugin
    DbRemoveManager(plugin, self.project),
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_location_remove.py", line 88, in __init__
    self.plugins_settings_service = PluginSettingsService(project, plugin)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin/settings_service.py", line 65, in __init__
    with self.feature_flag(
  File "/Users/edgarramirez/.pyenv/versions/3.10.4/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 658, in feature_flag
    allowed = self.get(f"{FEATURE_FLAG_PREFIX}.{feature}") or False
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 442, in get
    value, _ = self.get_with_source(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 429, in get_with_source
    value, metadata = self.get_with_metadata(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 322, in get_with_metadata
    setting_def = setting_def or self.find_setting(name)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 596, in find_setting
    for setting in self.definitions()
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 566, in definitions
    for setting in self.setting_definitions
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin/settings_service.py", line 138, in setting_definitions
    settings = self.plugin.settings_with_extras
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/behavior/canonical.py", line 182, in __getattr__
    raise AttributeError(attr) from err
AttributeError: settings_with_extras

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/__init__.py", line 74, in _run_cli
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: settings_with_extras
settings_with_extras
```

</details>


I think the fix is to replicate the patch Cody did on `PluginInvoker.env` in the construtor of `PluginSettingsService`, i.e.

```diff
- with self.feature_flag(
+ with project_settings_service.feature_flag(
```

@kgpayne @edgarrmondragon -- agreed with @edgarrmondragon's suggestion. Feature flags are generally at the project level and available only via the `project_settings_service`. The fix would be in [this line](https://github.com/meltano/meltano/blob/0dfe5367383da6c42a4f2d17634ab1a511b6d2d3/src/meltano/core/plugin/settings_service.py#L65)

I'm realizing that it's not very intuitive to have access to the `feature_flag` method in the `SettingsService` base class. I'm not sure what all changes it would entail, but it might be worth moving that method into `ProjectSettingsService` only, if possible to avoid this in the future.
> I'm realizing that it's not very intuitive to have access to the `feature_flag` method in the `SettingsService` base class. I'm not sure what all changes it would entail, but it might be worth moving that method into `ProjectSettingsService` only, if possible to avoid this in the future.

@cjohnhanson Thanks for jumping in! Yeah, it makes sense to implement feature flags exclusively at the project settings level. I'm almost sure it wouldn't cause any issues.
@kgpayne a good way to prevent future regressions would be to add a `meltano remove` at the end of this integration test:

https://github.com/meltano/meltano/blob/40a8066c3d3b5ad403bc08a4e51d195ffdd1e02a/docs/example-library/meltano-objects/index.md?plain=1#L1

Resolving patch:
diff --git a/src/meltano/core/plugin/settings_service.py b/src/meltano/core/plugin/settings_service.py
--- a/src/meltano/core/plugin/settings_service.py
+++ b/src/meltano/core/plugin/settings_service.py
@@ -62,7 +62,7 @@ def __init__(
 
         environment_env = {}
         if self.project.active_environment:
-            with self.feature_flag(
+            with project_settings_service.feature_flag(
                 FeatureFlags.STRICT_ENV_VAR_MODE, raise_error=False
             ) as strict_env_var_mode:
                 environment_env = {


Test patch: