Resolve ambiguity in current supported ways to set environment variables inside the plugin env via meltano.yml

# Meltano Configuration

Configuration management is a central part of the value provided to users of Meltano. Intention:

> Meltano is responsible for managing the configuration of all of a project's plugins. It knows what settings are supported by each plugin, and how and when different types of plugins expect to be fed that configuration.

## Use Cases

### Discovery

Discovery is how Meltano "knows what settings are supported by each plugin, and how and when different types of plugins expect to be fed that configuration". This is done by:

- Defining the settings available/expected to be configured for a given plugin.
- Defining the environment variables available/expected to be configured for a given plugin via the `env` settings [key](https://gitlab.com/meltano/meltano/-/blob/master/schema/discovery.schema.json#L653).
- Defining default values for settings via the `value` [key](https://gitlab.com/meltano/meltano/-/blob/master/schema/discovery.schema.json#L598). Note, default values can contain templated string references to environment variables (e.g. `$MELTANO_PROJECT_ROOT/utilities/great_expectations`).
	- Which environment is `MELTANO_PROJECT_ROOT` expected to be in? The Meltano runtime or the plugin runtime? 🤯 
- Defining env var maps to predictable names via the `env_aliases` [key](https://gitlab.com/meltano/meltano/-/blob/master/schema/discovery.schema.json#L661) for reuse across types of plugins (now marked as deprecated 🤦‍♂️). (e.g. `TAP_POSTGRESS_HOST` > `PG_ADDRESS`).
- Defining settings aliases via the `aliases` [key](https://gitlab.com/meltano/meltano/-/blob/master/schema/discovery.schema.json#L581) as alternative names by which the setting can be configured in `meltano.yml` and the CLI. This is often done to 'prettify' or make uniform with other plugin variants (e.g. `postgres_host` > `host`). **Note:* Meltano does not currently look for env vars of the form `<plugin name>_<alias name>` by default (though sunch env vars can be specified as `env_aliases` manually), nor does it appear to check or warn of clashes as the result of settings referenced in multiple ways (by its `name`, `aliases` or `env_aliases`).

In addition to the behaviours implemented as part of discovery, Meltano has 2 other builtin mechanisms for handling/discovering settings:

- [Pipeline environment variables](https://docs.meltano.com/guide/integration#pipeline-environment-variables) define settings that are passed from the environment of one plugin to the next in a pipeline (e.g. `extractor>loader>transformer`).
- [Plugin extras](https://docs.meltano.com/guide/configuration#plugin-extras) define settings used to configure the behaviour of specific plugin types _before_ and _after_ 3rd party code is invoked. This works by passing `plugin_extra` settings for use by all subclasses of [`BasePlugin`](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/base.py#L246) (e.g. extras available to the [`SingerTap`](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/tap.py#L104) allow users to [override](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/tap.py#L302) the `catalog.json` path used during Tap invocation).

### Configuration

Configuration is how end-users populate discovered settings with their own values. Config is done by:

- Using the `meltano config ...` CLI and `meltano.yml` to configure the above settings by name.
- Using environment variables in the Meltano runtime to configure the above settings via a `<PLUGIN NAME>_<SETTING_NAME>` env var naming convention. Environment variables can either be `export`'d into the Meltano runtime or coded in a `.env` file automatically discovered by Meltano.
-  Using values stored in the Meltano [system database](https://docs.meltano.com/concepts/project#system-database) to configure settings by name (also available through the UI).
-  Using the `env:` key of the [environments](https://docs.meltano.com/concepts/environments) feature to inject arbitrary environment variables into a plugins environment. These env vars will also be passed to subsequent stages in a pipeline according to the [pipeline environment variables](https://docs.meltano.com/guide/integration#pipeline-environment-variables) feature.
	- How are duplicates/clashes handled?
-  Setting env vars in the Meltano runtime environment matching names specified in setting `env:` and `env_aliases:` discovery keys (that do not have to conform to the `<PLUGIN NAME>_<SETTING_NAME>` conventions) to inject values into the plugins environment. **Note:** this is now an undesired behaviour, as per [#2984](https://gitlab.com/meltano/meltano/-/issues/2984).
-  Referencing provided [pipeline environment variables](https://docs.meltano.com/guide/integration#pipeline-environment-variables) in plugin tools code (e.g. the [provided profiles.yml](https://gitlab.com/meltano/files-dbt/-/blob/master/bundle/transform/profile/profiles.yml))
-  Using `plugin_extras` to configure the pre and post invocation behaviours of plugins.
-  Mapping a setting to an environment variable by adding a `my_setting: ${MY_ENV_VAR}` mapping under the `config:` block in `meltano.yml` or via the CLI using `meltano config <plugin> set my_setting "${MY_ENV_VAR}"`.
	- Which environment is `MY_ENV_VAR` expected to be in? The Meltano runtime or the plugin runtime? 🤯 

## FAQ's

- What is the difference between `config` and `settings`?

	Settings are specified as part of plugin definitions for the purposes of discovery. Config is the mechanism by which the fields described as settings are populated with values.

	Whilst `settings` usually only appear in `discovery.yml`, custom plugins added in `meltano.yml` places `settings` and `config` in the same view, which is completely baffling.

- How does `config` specified in `meltano.yml` arrive in files inside a plugins runtime? e.g. Singers' `config.json` or dbts' `profiles.yml`?

	For Singer the `SingerPlugin` class `src/meltano/core/plugin/singer/base.py` has a method (decorated as a `before_configure` Hook) for [writing a valid config.json file]() and another for [removing the file after](). Config arrives via an instance of the `PluginSettingsService` instantiated with both project and plugin.
	
	For dbt (specifically `profiles.yml` the approach is slightly different. dbt supports the retrieval of env vars within its config files (using jinja templating syntax). Therefore we need only match the env vars specified in the [provided profiles.yml](https://gitlab.com/meltano/files-dbt/-/blob/master/bundle/transform/profile/profiles.yml) to those available inside the Transformers environment. To smooth over this injection of env vars, Meltano makes available all the env vars provided to the Loader available to the Transformer as well, as part of the [pipeline environment feature](https://docs.meltano.com/guide/integration#pipeline-environment-variables). Currently we prescribe env vars of the form `PG_*` in the file bundle, but any matching env var name could be used (e.g. `TARGET_POSTGRES_*` to match the Loader env var convention).

- How do we determine what `settings`, specified in discovery, should be environment variables vs. which should be injected into files?

	This is implemented using the `env:` and `env_aliases:` keys in `discovery.yml`. Details as per Douwes comment [here]:
	
	> The behaviour around `env` and `env_aliases` is implemented in [`SettingsService.setting_env_vars`](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/settings_service.py#L63), which takes a `for_writing` boolean (which is `False` by default). Writing here means writing into the plugin's execution env, not-writing means reading from Meltano's execution env to populate a setting (that may then be written into the plugin execution env later). That `for_writing` flag is passed to [`SettingDefinition.env_vars`](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/setting_definition.py#L174) as `include_custom`. Following the logic there shows that when reading from the env, only the auto generated `<prefix>_<name>` is checked, and when writing into the env, env and env_aliases are used as well. I think the logic is that we want Meltano users to always use `<prefix>_<name>` consistently, but the plugin itself may expect a different env var.
	
	This description does not appear to explain the case described by [#2984](https://gitlab.com/meltano/meltano/-/issues/2984), which sparked our efforts to deprecate `env:` and `env_aliases:` in the first place 🤷‍♂️

## Proposed Next Steps

- To resolve  [#2984](https://gitlab.com/meltano/meltano/-/issues/2984) I believe we should remove the ability of the `SettingsService` to retrieve for injection any env vars _other than_ those matching the `<plugin name>_<setting name>` convention from the Meltano runtime environment. This meets Douwes intent that:

	> we want Meltano users to always use `<prefix>_<name>` consistently, but the plugin itself may expect a different env var."

- Implement a check to determine if any arbitrary env vars configured in `environments.[].env` do not clash with `env:` and `env_aliases:` defined in discovery. Warn or error if clashes are found.

- Provide guidance/docs on the usage on `env:` and `env_aliases:` keys in discovery. AJ suggests:
	
	> General plugins vs Singer plugins: General plugins must get some of there config from env vars; Singer plugins should not need env vars - but some do. Following from this:

	> - It's a healthy pattern for General plugins to declare `env:` entries to pass along settings to the plugin context (but probably not `env_aliases:`).
	> - It's a discouraged pattern for Singer plugins to declare `env:` entries. (We'll support if needed, but we should keep an eye on them and this should be the exception rather than the rule.)
	> - To avoid confusion around which env entries are needed for Singer plugins to function - versus which are convenient aliases, I think we should remove `env_alias:` entries from Singer plugins, replacing them with env: entries and an explanatory comment with issue link perhaps.

### Description V1

Quick summary of recent findings:

1. `env_aliases` / `env` - These are the same functionally. (New info to everyone!) - these fit the description as recently added in the JSON Schema for `env:`:
    - > `An environment variable that will be initialized with the setting's value whenever the plugin is invoked. This is used to inject environment variables into the plugin's execution context. If the named environment variable is already defined at runtime, Meltano will use the passed value from context instead of the configured value.`
    - Essentially a two-way read and write to/from the env variable.
2. The `env_alias` and `env` entries also have a special role in seeding `dbt_profiles.yml`. Specifically `PG_ADDRESS`, etc. I think (to confirm?) this is part of the reason dbt cannot be run on its own: because if the target wasn't invoked, those settings don't exist.
3. Settings aliases - newly discovered! Similar to `env_aliases` but at the setting level.
    - To be confirmed: if these are compatible with env injection in the form `<PLUGIN_NAME>_<SETTING_ALIAS>`. More in #3209.
4. `config` vs `settings`: `settings` are the plugin definition (optionally with default values) and `config` is one project's settings values.
5. When a user wants to map a setting to an environment variable, the recommended way of doing so is probably to add a `my_setting: ${MY_ENV_VAR}` mapping under the `config:` block.
6. General plugins vs Singer plugins: General plugins _must_ get some of there config from env vars; Singer plugins should not need env vars - but some do. Following from this:
    - It's a healthy pattern for General plugins to declare `env:` entries to pass along settings to the plugin context (but probably not `env_aliases:`).
    - It's a discouraged pattern for Singer plugins to declare `env:` entries. (We'll support if needed, but we should keep an eye on them and this should be the exception rather than the rule.)
    - To avoid confusion around which env entries are needed for Singer plugins to function - versus which are convenient aliases, I think we should remove `env_alias:` entries from Singer plugins, replacing them with `env:` entries and an explanatory comment with issue link perhaps.

------------------------

As per office hours 2022/02/02, the current mechanism of declaring an environment variable to be populated within plugin `settings` using `env:` and `env_alias:` causes confusion to even experienced Meltano users.

From the jsonschema:

```json
"env": {
    "type": "string",
    "description": "An environment variable that will be initialized with the setting's value whenever the plugin is invoked. This is used to inject environment variables into the plugin's execution context. If the named environment variable is already defined at runtime, Meltano will use the passed value from context instead of the configured value.",
    "examples": [
        "DBT_PROFILES_DIR",
        "GE_HOME"
    ]
}

"env_aliases": {
    "type": "array",
    "description": "Deprecated. Use to delegate alternative environment variables for overriding this setting's value",
    "examples": [
        "TAP_ACCOUNT_ID",
        "TAP_ACCOUNT_IDENTIFIER"
    ],
    "items": {
        "type": "string"
    }
}
```

Specifically, from `env`,:

> If the named environment variable is already defined at runtime, Meltano will use the passed value from context instead of the configured value.

This causes unexpected behaviours for users not aware of this dual behaviour. We propose that, in stead of relying on `env:` for both purposes (sourcing values to inject _and_ determining the target env var name to inject into), we simply remove the sourcing behaviour. This would mean Meltano would no longer search its runtime environment for matching env var names to retrieve values, effectively making any settings with `env:` specified an "environment variable type setting" (as suggested by @tayloramurphy [here](https://gitlab.com/meltano/meltano/-/merge_requests/2505#note_829519083)). Sourcing of values would only be possible via the standard mechanisms; explicit `config:` or env vars in the Meltano runtime of the form `<plugin name>_<setting name>`.

`env_aliases` might be kept to allow one setting to be mapped to multiple env vars. E.g.

```yaml
settings:
- name: aws_profile
  env: AWS_PROFILE
  env_aliases: [AWS_DEFUALT_PROFILE]
```

or both could be replaced with an array-type `env_vars:` e.g.:

```yaml
settings:
- name: aws_profile
  env_vars: [AWS_PROFILE, AWS_DEFUALT_PROFILE]
```

It is also worth considering if these settings should be injected as keys in config files either. Not doing so would further increase the specificity of the settings function as "an environment variable created from a Meltano setting".

### Possible Challenges

- Users can also set environment variables directly using the `env:` key in the `--environment` feature. This produces two ways to achieve the same outcome, and may cause conflicts and confusion to users. This can be handled with a resolution order (preferring settings above environment variables) and/or by warning/erroring if the same env var is specified by the two methods.

Story points: 4.0