Repository: meltano/meltano

Issue:
bug: `meltano config <plugin>` ignores `MELTANO_ENVIRONMENT` environment variable
### Meltano Version

2.15.0

### Python Version

3.11

### Bug scope

CLI (options, error messages, logging, etc.)

### Operating System

macOS Monterey 12.6

### Description

We run meltano using environment variables for as much configuration as possible. I have found what I think is a bug, in that I have the meltano environment set via an environment variable, but the `meltano config <plugin>` command ignores it.

Relevant sections of `.env`:
```shell
MELTANO_ENVIRONMENT=dev
```

and `meltano.yml`
```yaml
include_paths:
- ./environments/dev.meltano.yml
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git@34f135b45a87b468f3375c72f9d451ae6525fae7
  - name: tap-s3-mongo
    inherit_from: tap-spreadsheets-anywhere
```

and from `environments/dev.meltano.yml`
```yaml
environments:
  - name: dev
    config:
      plugins:
        loaders:
          - name: target-redshift-mongo
            config:
              schema_mapping:
                payments:
                  target_schema: payments
                profiles:
                  target_schema: customers
```

Expected behavior: I expect that running `meltano config target-redshift-mongo` here (with meltano environment set to `dev` via environment variables) includes the schema mapping defined for this plugin in the dev environment.

Observed behavior: It does not include the schema mapping. But, if I pass `--environment=dev` explicitly as a command-line argument, it does include it.

```shell
# note: MELTANO_ENVIRONMENT=dev is set in .env
$ meltano config target-redshift-mongo
2023-02-07T12:43:14.151935Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
{
  "host": "<SECRET>",
  "port": <SECRET>,
  "dbname": "<SECRET>",
  "user": "<SECRET>",
  "password": "<SECRET>",
  "s3_bucket": "<SECRET>",
  "default_target_schema": "<SECRET>",
  "copy_options": "EMPTYASNULL BLANKSASNULL TRIMBLANKS TRUNCATECOLUMNS TIMEFORMAT 'auto' COMPUPDATE OFF STATUPDATE OFF",
  "batch_size_rows": 100,
  "flush_all_streams": false,
  "parallelism": 0,
  "max_parallelism": 16,
  "disable_table_cache": false,
  "add_metadata_columns": true,
  "hard_delete": false,
  "data_flattening_max_level": 0,
  "primary_key_required": false,
  "validate_records": false,
  "skip_updates": false,
  "slices": 1
}
```

```shell
# note: MELTANO_ENVIRONMENT=dev is set in .env
$ meltano --environment=dev config target-redshift-mongo
2023-02-07T12:45:17.010451Z [info     ] Environment 'dev' is active
{
  "host": "<SECRET>",
  "port": <SECRET>,
  "dbname": "<SECRET>",
  "user": "<SECRET>",
  "password": "<SECRET>",
  "s3_bucket": "<SECRET>",
  "default_target_schema": "<SECRET>",
  "copy_options": "EMPTYASNULL BLANKSASNULL TRIMBLANKS TRUNCATECOLUMNS TIMEFORMAT 'auto' COMPUPDATE OFF STATUPDATE OFF",
  "batch_size_rows": 100,
  "flush_all_streams": false,
  "parallelism": 0,
  "max_parallelism": 16,
  "schema_mapping": {
    "payments": {
      "target_schema": "payments"
    },
    "profiles": {
      "target_schema": "customers"
    }
  },
  "disable_table_cache": false,
  "add_metadata_columns": true,
  "hard_delete": false,
  "data_flattening_max_level": 0,
  "primary_key_required": false,
  "validate_records": false,
  "skip_updates": false,
  "slices": 1
}
```

If I change the `default_environment` in `meltano.yml` (so that it's not the same as the environment I have defined in `.env`) I get the same behavior.

### Code

_No response_


Discussion:
Thanks for the detailed report @menzenski

I can confirm that the `MELTANO_ENVIRONMENT` env var is not being used by `meltano` as an alternative to `--environment` when set via `.env`, but it should be. For it to not work like that would both be inconsistent, and also go against an example we have in our documentation.

Resolving patch:
diff --git a/src/meltano/cli/cli.py b/src/meltano/cli/cli.py
--- a/src/meltano/cli/cli.py
+++ b/src/meltano/cli/cli.py
@@ -15,7 +15,7 @@
 from meltano.core.behavior.versioned import IncompatibleVersionError
 from meltano.core.error import EmptyMeltanoFileException
 from meltano.core.logging import LEVELS, setup_logging
-from meltano.core.project import Project, ProjectNotFound
+from meltano.core.project import PROJECT_ENVIRONMENT_ENV, Project, ProjectNotFound
 from meltano.core.project_settings_service import ProjectSettingsService
 from meltano.core.tracking import Tracker
 from meltano.core.tracking.contexts import CliContext
@@ -51,11 +51,7 @@ def main(self, *args, **kwargs) -> NoReturn:
     "--log-config", type=str, help="Path to a python logging yaml config file."
 )
 @click.option("-v", "--verbose", count=True, help="Not used.")
-@click.option(
-    "--environment",
-    envvar="MELTANO_ENVIRONMENT",
-    help="Meltano environment name.",
-)
+@click.option("--environment", help="Meltano environment name.")
 @click.option(
     "--no-environment", is_flag=True, default=False, help="Don't use any environment."
 )
@@ -111,23 +107,18 @@ def cli(  # noqa: C901,WPS231
         if project.readonly:
             logger.debug("Project is read-only.")
 
-        # detect active environment
-        selected_environment = None
-        is_default_environment = False
-        if no_environment or (environment and environment.lower() == "null"):
-            logger.info("No environment is active")
-        elif environment:
-            selected_environment = environment
-        elif project_setting_service.get("default_environment"):
-            selected_environment = project_setting_service.get("default_environment")
-            is_default_environment = True
-        ctx.obj["selected_environment"] = selected_environment
-        ctx.obj["is_default_environment"] = is_default_environment
+        (
+            ctx.obj["selected_environment"],
+            ctx.obj["is_default_environment"],
+        ) = detect_selected_environment(
+            cli_environment=environment,
+            cli_no_environment=no_environment,
+            project=project,
+            project_settings_service=project_setting_service,
+        )
         ctx.obj["project"] = project
         ctx.obj["tracker"] = Tracker(project)
-        ctx.obj["tracker"].add_contexts(
-            CliContext.from_click_context(ctx)
-        )  # backfill the `cli` CliContext
+        ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx))
     except ProjectNotFound:
         ctx.obj["project"] = None
     except EmptyMeltanoFileException:
@@ -143,3 +134,41 @@ def cli(  # noqa: C901,WPS231
             "For more details, visit https://docs.meltano.com/guide/installation#upgrading-meltano-version"
         )
         sys.exit(3)
+
+
+def detect_selected_environment(
+    *,
+    cli_environment: str | None,
+    cli_no_environment: bool,
+    project: Project,
+    project_settings_service: ProjectSettingsService,
+) -> tuple[str | None, bool]:
+    """Detect the Meltano environment selected (but not yet activated).
+
+    Precedence is:
+    1. The `--environment` CLI option
+    2. Env var from the shell
+    3. Env var from `.env`
+    4. Default environment from `meltano.yml`
+    5. `None`
+
+    Args:
+        cli_environment: The `--environment` option value from the CLI.
+        no_environment: The `--no-environment` option value from the CLI.
+        project: The Meltano project.
+        project_setting_service: A project settings service for the project.
+
+    Returns:
+        The selected environment, and whether it is the default environment.
+    """
+    environment = cli_environment or os.environ.get(
+        PROJECT_ENVIRONMENT_ENV,
+        project.dotenv_env.get(PROJECT_ENVIRONMENT_ENV, None),
+    )
+    if cli_no_environment or (environment and environment.lower() == "null"):
+        logger.info("No Meltano environment was selected")
+    elif environment:
+        return environment, False
+    elif project_settings_service.get("default_environment"):
+        return project_settings_service.get("default_environment"), True
+    return None, False
diff --git a/src/meltano/cli/utils.py b/src/meltano/cli/utils.py
--- a/src/meltano/cli/utils.py
+++ b/src/meltano/cli/utils.py
@@ -602,7 +602,7 @@ def enact_environment_behavior(
     ctx: click.Context,
 ) -> None:
     """Activate the environment in the specified way."""
-    if behavior is None:
+    if behavior is None or "selected_environment" not in ctx.obj:
         return
     if behavior is CliEnvironmentBehavior.environment_optional_use_default:
         activate_environment(ctx, ctx.obj["project"], required=False)


Test patch:
diff --git a/tests/meltano/cli/test_cli.py b/tests/meltano/cli/test_cli.py
--- a/tests/meltano/cli/test_cli.py
+++ b/tests/meltano/cli/test_cli.py
@@ -8,18 +8,20 @@
 from time import perf_counter_ns
 
 import click
+import mock
 import pytest
 import yaml
 from structlog.stdlib import get_logger
 
 import meltano
 from asserts import assert_cli_runner
+from fixtures.cli import MeltanoCliRunner
 from fixtures.utils import cd
 from meltano.cli import cli, handle_meltano_error
 from meltano.cli.utils import CliError
 from meltano.core.error import EmptyMeltanoFileException, MeltanoError
 from meltano.core.logging.utils import setup_logging
-from meltano.core.project import PROJECT_READONLY_ENV, Project
+from meltano.core.project import PROJECT_ENVIRONMENT_ENV, PROJECT_READONLY_ENV, Project
 from meltano.core.project_settings_service import ProjectSettingsService
 
 ANSI_RE = re.compile(r"\033\[[;?0-9]*[a-zA-Z]")
@@ -82,9 +84,7 @@ def test_activate_project_readonly_env(
         assert Project._default.readonly
 
     @pytest.mark.order(2)
-    def test_activate_project_readonly_dotenv(
-        self, project, cli_runner, pushd, monkeypatch
-    ):
+    def test_activate_project_readonly_dotenv(self, project, cli_runner, pushd):
         ProjectSettingsService(project).set("project_readonly", True)
 
         assert Project._default is None
@@ -94,6 +94,43 @@ def test_activate_project_readonly_dotenv(
 
         assert Project._default.readonly
 
+    def test_environment_precedence(
+        self,
+        project: Project,
+        pushd,
+        cli_runner: MeltanoCliRunner,
+        monkeypatch: pytest.MonkeyPatch,
+    ):
+        pushd(project.root)
+        monkeypatch.delenv(PROJECT_ENVIRONMENT_ENV, raising=False)
+        environment_names = {
+            name: f"env_set_from_{name}" for name in ("dotenv", "cli_option", "env_var")
+        }
+        with mock.patch(
+            "meltano.core.project.Project.dotenv_env",
+            new_callable=mock.PropertyMock,
+            return_value={PROJECT_ENVIRONMENT_ENV: environment_names["dotenv"]},
+        ):
+            args = ("invoke", "tap-mock")
+            results = {
+                "dotenv": cli_runner.invoke(cli, args),
+                "cli_option": cli_runner.invoke(
+                    cli,
+                    (f"--environment={environment_names['cli_option']}", *args),
+                ),
+                "env_var": cli_runner.invoke(
+                    cli,
+                    args,
+                    env={PROJECT_ENVIRONMENT_ENV: environment_names["env_var"]},
+                ),
+            }
+        for source, name in environment_names.items():
+            assert results[source].exit_code
+            assert (
+                results[source].exception.args[0]
+                == f"Environment {name!r} was not found."
+            )
+
     def test_version(self, cli_runner):
         cli_version = cli_runner.invoke(cli, ["--version"])