Repository: dbt-labs/dbt-core
Pull request: Create and protect dbt engine environment variable namespace via prefix `DBT_ENGINE`
Resolves #11340

### Problem

Any time we added a new dbt engine environment variable using the `DBT_` environment variable (like `DBT_SAMPLE`) we were potentially _breaking_ projects that already has a custom environment variable for their own purposes which the same name. We _need_ to be able to add new environment variables _without_ breaking people's projects.

### Solution

Create a namespace that can _only_ contain dbt engine environment variables. Specifically, the prefix `DBT_ENGINE` is now _reserved_. People can still _set_ these environment variables, but only ones which are defined as existing.

### Checklist

- [X] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me.
- [X] I have run this code in development, and it appears to resolve the stated issue.
- [X] This PR includes tests, or tests are not required or relevant for this PR.
- [X] This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
- [X] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions.


Problem statement:
Deprecate unconstrained user space of environment variables
# Problem
The current environment variable landscape is that:
1. All core defined env vars are prefixed with `DBT_`
2. Core users can create custom environment variables with any name
3. Cloud users must create env vars with the `DBT_` prefix

The issue is that whenever we add a new core defined env var (1), it is potentially breaking for existing projects in both core and cloud. If the new env var defined in core collides with a custom environment variable the user has, then the project may break. We saw this recently with sample mode. We added a new environment variable, `DBT_SAMPLE`, that requires a specific structure for the associated value when set. Separately, a project had defined `DBT_SAMPLE` as a truthy value for their uses, and thus when they used it, it caused a parsing error.

# Solution
A reserved name space for dbt defined environment variables. All new dbt defined environment variables (not custom environment variables) will be prefixed with `_DBT_`. The use of custom `_DBT_` environment variables will be deprecated, which is currently only possible in situation (2).

# Acceptance criteria
- There is a tracked allow list of `_DBT_` environment variables
- The existence of custom `_DBT_` environment variables raises a deprecation warning

Hints from issue discussion:

Commit message: Get the env vars from the invocation context in `validate_engine_env_vars`

Patch to review:
diff --git a/.changes/unreleased/Features-20250703-175341.yaml b/.changes/unreleased/Features-20250703-175341.yaml
new file mode 100644
index 00000000000..a1048342772
--- /dev/null
+++ b/.changes/unreleased/Features-20250703-175341.yaml
@@ -0,0 +1,6 @@
+kind: Features
+body: Create constrained namespace for dbt engine env vars
+time: 2025-07-03T17:53:41.406701-05:00
+custom:
+  Author: QMalcolm
+  Issue: "11340"
diff --git a/core/dbt/cli/params.py b/core/dbt/cli/params.py
index 92089eaffe6..a7c43677357 100644
--- a/core/dbt/cli/params.py
+++ b/core/dbt/cli/params.py
@@ -1,4 +1,5 @@
 from pathlib import Path
+from typing import Any, Callable, List
 
 import click
 
@@ -24,48 +25,65 @@
     "type": tuple,
 }
 
+# Record of env vars associated with options
+KNOWN_ENV_VARS: List[str] = []
+
+
+def _create_option_and_track_env_var(
+    *args: Any, **kwargs: Any
+) -> Callable[[click.decorators.FC], click.decorators.FC]:
+    global KNOWN_ENV_VARS
+
+    envvar = kwargs.get("envvar", None)
+    if isinstance(envvar, str):
+        KNOWN_ENV_VARS.append(envvar)
+
+    return click.option(*args, **kwargs)
+
+
 # --- The actual option definitions --- #
-add_package = click.option(
+add_package = _create_option_and_track_env_var(
     "--add-package",
     help="Add a package to current package spec, specify it as package-name@version. Change the source with --source flag.",
     envvar=None,
     type=Package(),
 )
-args = click.option(
+
+args = _create_option_and_track_env_var(
     "--args",
     envvar=None,
     help="Supply arguments to the macro. This dictionary will be mapped to the keyword arguments defined in the selected macro. This argument should be a YAML string, eg. '{my_variable: my_value}'",
     type=YAML(),
 )
 
-browser = click.option(
+browser = _create_option_and_track_env_var(
     "--browser/--no-browser",
     envvar=None,
     help="Wether or not to open a local web browser after starting the server",
     default=True,
 )
 
-cache_selected_only = click.option(
+cache_selected_only = _create_option_and_track_env_var(
     "--cache-selected-only/--no-cache-selected-only",
     envvar="DBT_CACHE_SELECTED_ONLY",
     help="At start of run, populate relational cache only for schemas containing selected nodes, or for all schemas of interest.",
 )
 
-clean_project_files_only = click.option(
+clean_project_files_only = _create_option_and_track_env_var(
     "--clean-project-files-only / --no-clean-project-files-only",
     envvar="DBT_CLEAN_PROJECT_FILES_ONLY",
     help="If disabled, dbt clean will delete all paths specified in clean-paths, even if they're outside the dbt project.",
     default=True,
 )
 
-compile_docs = click.option(
+compile_docs = _create_option_and_track_env_var(
     "--compile/--no-compile",
     envvar=None,
     help="Whether or not to run 'dbt compile' as part of docs generation",
     default=True,
 )
 
-compile_inject_ephemeral_ctes = click.option(
+compile_inject_ephemeral_ctes = _create_option_and_track_env_var(
     "--inject-ephemeral-ctes/--no-inject-ephemeral-ctes",
     envvar=None,
     help="Internal flag controlling injection of referenced ephemeral models' CTEs during `compile`.",
@@ -73,21 +91,21 @@
     default=True,
 )
 
-config_dir = click.option(
+config_dir = _create_option_and_track_env_var(
     "--config-dir",
     envvar=None,
     help="Print a system-specific command to access the directory that the current dbt project is searching for a profiles.yml. Then, exit. This flag renders other debug step flags no-ops.",
     is_flag=True,
 )
 
-debug = click.option(
+debug = _create_option_and_track_env_var(
     "--debug/--no-debug",
     "-d/ ",
     envvar="DBT_DEBUG",
     help="Display debug logging during dbt execution. Useful for debugging and making bug reports.",
 )
 
-debug_connection = click.option(
+debug_connection = _create_option_and_track_env_var(
     "--connection",
     envvar=None,
     help="Test the connection to the target database independent of dependency checks.",
@@ -95,13 +113,13 @@
 )
 
 # flag was previously named DEFER_MODE
-defer = click.option(
+defer = _create_option_and_track_env_var(
     "--defer/--no-defer",
     envvar="DBT_DEFER",
     help="If set, resolve unselected nodes by deferring to the manifest within the --state directory.",
 )
 
-defer_state = click.option(
+defer_state = _create_option_and_track_env_var(
     "--defer-state",
     envvar="DBT_DEFER_STATE",
     help="Override the state directory for deferral only.",
@@ -114,7 +132,7 @@
     ),
 )
 
-deprecated_defer = click.option(
+deprecated_defer = _create_option_and_track_env_var(
     "--deprecated-defer",
     envvar="DBT_DEFER_TO_STATE",
     help="Internal flag for deprecating old env var.",
@@ -122,14 +140,14 @@
     hidden=True,
 )
 
-deprecated_favor_state = click.option(
+deprecated_favor_state = _create_option_and_track_env_var(
     "--deprecated-favor-state",
     envvar="DBT_FAVOR_STATE_MODE",
     help="Internal flag for deprecating old env var.",
 )
 
 # Renamed to --export-saved-queries
-deprecated_include_saved_query = click.option(
+deprecated_include_saved_query = _create_option_and_track_env_var(
     "--include-saved-query/--no-include-saved-query",
     envvar="DBT_INCLUDE_SAVED_QUERY",
     help="Include saved queries in the list of resources to be selected for build command",
@@ -137,7 +155,7 @@
     hidden=True,
 )
 
-deprecated_print = click.option(
+deprecated_print = _create_option_and_track_env_var(
     "--deprecated-print/--deprecated-no-print",
     envvar="DBT_NO_PRINT",
     help="Internal flag for deprecating old env var.",
@@ -146,7 +164,7 @@
     callback=lambda ctx, param, value: not value,
 )
 
-deprecated_state = click.option(
+deprecated_state = _create_option_and_track_env_var(
     "--deprecated-state",
     envvar="DBT_ARTIFACT_STATE_PATH",
     help="Internal flag for deprecating old env var.",
@@ -160,21 +178,21 @@
     ),
 )
 
-empty = click.option(
+empty = _create_option_and_track_env_var(
     "--empty/--no-empty",
     envvar="DBT_EMPTY",
     help="If specified, limit input refs and sources to zero rows.",
     is_flag=True,
 )
 
-empty_catalog = click.option(
+empty_catalog = _create_option_and_track_env_var(
     "--empty-catalog",
     help="If specified, generate empty catalog.json file during the `dbt docs generate` command.",
     default=False,
     is_flag=True,
 )
 
-event_time_end = click.option(
+event_time_end = _create_option_and_track_env_var(
     "--event-time-end",
     envvar="DBT_EVENT_TIME_END",
     help="If specified, the end datetime dbt uses to filter microbatch model inputs (exclusive).",
@@ -182,7 +200,7 @@
     default=None,
 )
 
-event_time_start = click.option(
+event_time_start = _create_option_and_track_env_var(
     "--event-time-start",
     envvar="DBT_EVENT_TIME_START",
     help="If specified, the start datetime dbt uses to filter microbatch model inputs (inclusive).",
@@ -190,7 +208,7 @@
     default=None,
 )
 
-exclude = click.option(
+exclude = _create_option_and_track_env_var(
     "--exclude",
     envvar=None,
     type=tuple,
@@ -199,7 +217,7 @@
     help="Specify the nodes to exclude.",
 )
 
-exclude_resource_type = click.option(
+exclude_resource_type = _create_option_and_track_env_var(
     "--exclude-resource-types",
     "--exclude-resource-type",
     envvar="DBT_EXCLUDE_RESOURCE_TYPES",
@@ -226,7 +244,7 @@
     default=(),
 )
 
-export_saved_queries = click.option(
+export_saved_queries = _create_option_and_track_env_var(
     "--export-saved-queries/--no-export-saved-queries",
     envvar="DBT_EXPORT_SAVED_QUERIES",
     help="Export saved queries within the 'build' command, otherwise no-op",
@@ -234,20 +252,20 @@
     hidden=True,
 )
 
-fail_fast = click.option(
+fail_fast = _create_option_and_track_env_var(
     "--fail-fast/--no-fail-fast",
     "-x/ ",
     envvar="DBT_FAIL_FAST",
     help="Stop execution on first failure.",
 )
 
-favor_state = click.option(
+favor_state = _create_option_and_track_env_var(
     "--favor-state/--no-favor-state",
     envvar="DBT_FAVOR_STATE",
     help="If set, defer to the argument provided to the state flag for resolving unselected nodes, even if the node(s) exist as a database object in the current environment.",
 )
 
-full_refresh = click.option(
+full_refresh = _create_option_and_track_env_var(
     "--full-refresh",
     "-f",
     envvar="DBT_FULL_REFRESH",
@@ -255,7 +273,7 @@
     is_flag=True,
 )
 
-host = click.option(
+host = _create_option_and_track_env_var(
     "--host",
     envvar="DBT_HOST",
     help="host to serve dbt docs on",
@@ -263,7 +281,7 @@
     default="127.0.0.1",
 )
 
-indirect_selection = click.option(
+indirect_selection = _create_option_and_track_env_var(
     "--indirect-selection",
     envvar="DBT_INDIRECT_SELECTION",
     help="Choose which tests to select that are adjacent to selected resources. Eager is most inclusive, cautious is most exclusive, and buildable is in between. Empty includes no tests at all.",
@@ -271,40 +289,40 @@
     default="eager",
 )
 
-inline = click.option(
+inline = _create_option_and_track_env_var(
     "--inline",
     envvar=None,
     help="Pass SQL inline to dbt compile and show",
 )
 
-inline_direct = click.option(
+inline_direct = _create_option_and_track_env_var(
     "--inline-direct",
     envvar=None,
     help="Internal flag to pass SQL inline to dbt show. Do not load the entire project or apply templating.",
     hidden=True,
 )
 
-introspect = click.option(
+introspect = _create_option_and_track_env_var(
     "--introspect/--no-introspect",
     envvar="DBT_INTROSPECT",
     help="Whether to scaffold introspective queries as part of compilation",
     default=True,
 )
 
-lock = click.option(
+lock = _create_option_and_track_env_var(
     "--lock",
     envvar=None,
     help="Generate the package-lock.yml file without install the packages.",
     is_flag=True,
 )
 
-log_cache_events = click.option(
+log_cache_events = _create_option_and_track_env_var(
     "--log-cache-events/--no-log-cache-events",
     help="Enable verbose logging for relational cache events to help when debugging.",
     envvar="DBT_LOG_CACHE_EVENTS",
 )
 
-log_format = click.option(
+log_format = _create_option_and_track_env_var(
     "--log-format",
     envvar="DBT_LOG_FORMAT",
     help="Specify the format of logging to the console and the log file. Use --log-format-file to configure the format for the log file differently than the console.",
@@ -312,7 +330,7 @@
     default="default",
 )
 
-log_format_file = click.option(
+log_format_file = _create_option_and_track_env_var(
     "--log-format-file",
     envvar="DBT_LOG_FORMAT_FILE",
     help="Specify the format of logging to the log file by overriding the default value and the general --log-format setting.",
@@ -320,7 +338,7 @@
     default="debug",
 )
 
-log_level = click.option(
+log_level = _create_option_and_track_env_var(
     "--log-level",
     envvar="DBT_LOG_LEVEL",
     help="Specify the minimum severity of events that are logged to the console and the log file. Use --log-level-file to configure the severity for the log file differently than the console.",
@@ -328,7 +346,7 @@
     default="info",
 )
 
-log_level_file = click.option(
+log_level_file = _create_option_and_track_env_var(
     "--log-level-file",
     envvar="DBT_LOG_LEVEL_FILE",
     help="Specify the minimum severity of events that are logged to the log file by overriding the default value and the general --log-level setting.",
@@ -336,7 +354,7 @@
     default="debug",
 )
 
-log_file_max_bytes = click.option(
+log_file_max_bytes = _create_option_and_track_env_var(
     "--log-file-max-bytes",
     envvar="DBT_LOG_FILE_MAX_BYTES",
     help="Configure the max file size in bytes for a single dbt.log file, before rolling over. 0 means no limit.",
@@ -345,7 +363,7 @@
     hidden=True,
 )
 
-log_path = click.option(
+log_path = _create_option_and_track_env_var(
     "--log-path",
     envvar="DBT_LOG_PATH",
     help="Configure the 'log-path'. Only applies this setting for the current run. Overrides the 'DBT_LOG_PATH' if it is set.",
@@ -353,16 +371,16 @@
     type=click.Path(resolve_path=True, path_type=Path),
 )
 
-macro_debugging = click.option(
+macro_debugging = _create_option_and_track_env_var(
     "--macro-debugging/--no-macro-debugging",
     envvar="DBT_MACRO_DEBUGGING",
     hidden=True,
 )
 
-models = click.option(*model_decls, **select_attrs)  # type: ignore[arg-type]
+models = _create_option_and_track_env_var(*model_decls, **select_attrs)  # type: ignore[arg-type]
 
 # This less standard usage of --output where output_path below is more standard
-output = click.option(
+output = _create_option_and_track_env_var(
     "--output",
     envvar=None,
     help="Specify the output format: either JSON or a newline-delimited list of selectors, paths, or names",
@@ -370,7 +388,7 @@
     default="selector",
 )
 
-output_keys = click.option(
+output_keys = _create_option_and_track_env_var(
     "--output-keys",
     envvar=None,
     help=(
@@ -383,7 +401,7 @@
     default=[],
 )
 
-output_path = click.option(
+output_path = _create_option_and_track_env_var(
     "--output",
     "-o",
     envvar=None,
@@ -392,14 +410,14 @@
     default=None,
 )
 
-partial_parse = click.option(
+partial_parse = _create_option_and_track_env_var(
     "--partial-parse/--no-partial-parse",
     envvar="DBT_PARTIAL_PARSE",
     help="Allow for partial parsing by looking for and writing to a pickle file in the target directory. This overrides the user configuration file.",
     default=True,
 )
 
-partial_parse_file_diff = click.option(
+partial_parse_file_diff = _create_option_and_track_env_var(
     "--partial-parse-file-diff/--no-partial-parse-file-diff",
     envvar="DBT_PARTIAL_PARSE_FILE_DIFF",
     help="Internal flag for whether to compute a file diff during partial parsing.",
@@ -407,7 +425,7 @@
     default=True,
 )
 
-partial_parse_file_path = click.option(
+partial_parse_file_path = _create_option_and_track_env_var(
     "--partial-parse-file-path",
     envvar="DBT_PARTIAL_PARSE_FILE_PATH",
     help="Internal flag for path to partial_parse.manifest file.",
@@ -416,21 +434,21 @@
     type=click.Path(exists=True, dir_okay=False, resolve_path=True),
 )
 
-print = click.option(
+print = _create_option_and_track_env_var(
     "--print/--no-print",
     envvar="DBT_PRINT",
     help="Output all {{ print() }} macro calls.",
     default=True,
 )
 
-populate_cache = click.option(
+populate_cache = _create_option_and_track_env_var(
     "--populate-cache/--no-populate-cache",
     envvar="DBT_POPULATE_CACHE",
     help="At start of run, use `show` or `information_schema` queries to populate a relational cache, which can speed up subsequent materializations.",
     default=True,
 )
 
-port = click.option(
+port = _create_option_and_track_env_var(
     "--port",
     envvar=None,
     help="Specify the port number for the docs server",
@@ -438,7 +456,7 @@
     type=click.INT,
 )
 
-printer_width = click.option(
+printer_width = _create_option_and_track_env_var(
     "--printer-width",
     envvar="DBT_PRINTER_WIDTH",
     help="Sets the width of terminal output",
@@ -446,13 +464,13 @@
     default=80,
 )
 
-profile = click.option(
+profile = _create_option_and_track_env_var(
     "--profile",
     envvar="DBT_PROFILE",
     help="Which existing profile to load. Overrides setting in dbt_project.yml.",
 )
 
-profiles_dir = click.option(
+profiles_dir = _create_option_and_track_env_var(
     "--profiles-dir",
     envvar="DBT_PROFILES_DIR",
     help="Which directory to look in for the profiles.yml file. If not set, dbt will look in the current working directory first, then HOME/.dbt/",
@@ -463,7 +481,7 @@
 # `dbt debug` uses this because it implements custom behaviour for non-existent profiles.yml directories
 # `dbt deps` does not load a profile at all
 # `dbt init` will write profiles.yml if it doesn't yet exist
-profiles_dir_exists_false = click.option(
+profiles_dir_exists_false = _create_option_and_track_env_var(
     "--profiles-dir",
     envvar="DBT_PROFILES_DIR",
     help="Which directory to look in for the profiles.yml file. If not set, dbt will look in the current working directory first, then HOME/.dbt/",
@@ -471,7 +489,7 @@
     type=click.Path(exists=False),
 )
 
-project_dir = click.option(
+project_dir = _create_option_and_track_env_var(
     "--project-dir",
     envvar="DBT_PROJECT_DIR",
     help="Which directory to look in for the dbt_project.yml file. Default is the current working directory and its parents.",
@@ -479,16 +497,16 @@
     type=click.Path(exists=True),
 )
 
-quiet = click.option(
+quiet = _create_option_and_track_env_var(
     "--quiet/--no-quiet",
     "-q",
     envvar="DBT_QUIET",
     help="Suppress all non-error logging to stdout. Does not affect {{ print() }} macro calls.",
 )
 
-raw_select = click.option(*select_decls, **select_attrs)  # type: ignore[arg-type]
+raw_select = _create_option_and_track_env_var(*select_decls, **select_attrs)  # type: ignore[arg-type]
 
-record_timing_info = click.option(
+record_timing_info = _create_option_and_track_env_var(
     "--record-timing-info",
     "-r",
     envvar=None,
@@ -496,7 +514,7 @@
     type=click.Path(exists=False),
 )
 
-resource_type = click.option(
+resource_type = _create_option_and_track_env_var(
     "--resource-types",
     "--resource-type",
     envvar="DBT_RESOURCE_TYPES",
@@ -524,7 +542,7 @@
     default=(),
 )
 
-sample = click.option(
+sample = _create_option_and_track_env_var(
     "--sample",
     envvar="DBT_SAMPLE",
     help="Run in sample mode with given SAMPLE_WINDOW spec, such that ref/source calls are sampled by the sample window.",
@@ -537,29 +555,29 @@
 # Most CLI arguments should use the combined `select` option that aliases `--models` to `--select`.
 # However, if you need to split out these separators (like `dbt ls`), use the `models` and `raw_select` options instead.
 # See https://github.com/dbt-labs/dbt-core/pull/6774#issuecomment-1408476095 for more info.
-select = click.option(*select_decls, *model_decls, **select_attrs)  # type: ignore[arg-type]
+select = _create_option_and_track_env_var(*select_decls, *model_decls, **select_attrs)  # type: ignore[arg-type]
 
-selector = click.option(
+selector = _create_option_and_track_env_var(
     "--selector",
     envvar=None,
     help="The selector name to use, as defined in selectors.yml",
 )
 
-send_anonymous_usage_stats = click.option(
+send_anonymous_usage_stats = _create_option_and_track_env_var(
     "--send-anonymous-usage-stats/--no-send-anonymous-usage-stats",
     envvar="DBT_SEND_ANONYMOUS_USAGE_STATS",
     help="Send anonymous usage stats to dbt Labs.",
     default=True,
 )
 
-show = click.option(
+show = _create_option_and_track_env_var(
     "--show",
     envvar=None,
     help="Show a sample of the loaded data in the terminal",
     is_flag=True,
 )
 
-show_limit = click.option(
+show_limit = _create_option_and_track_env_var(
     "--limit",
     envvar=None,
     help="Limit the number of results returned by dbt show",
@@ -567,7 +585,7 @@
     default=5,
 )
 
-show_output_format = click.option(
+show_output_format = _create_option_and_track_env_var(
     "--output",
     envvar=None,
     help="Output format for dbt compile and dbt show",
@@ -575,7 +593,7 @@
     default="text",
 )
 
-show_resource_report = click.option(
+show_resource_report = _create_option_and_track_env_var(
     "--show-resource-report/--no-show-resource-report",
     default=False,
     envvar="DBT_SHOW_RESOURCE_REPORT",
@@ -588,14 +606,14 @@
 # This will need to be communicated as a change to the community!
 #
 # N.B. This flag is only used for testing, hence it's hidden from help text.
-single_threaded = click.option(
+single_threaded = _create_option_and_track_env_var(
     "--single-threaded/--no-single-threaded",
     envvar="DBT_SINGLE_THREADED",
     default=False,
     hidden=True,
 )
 
-show_all_deprecations = click.option(
+show_all_deprecations = _create_option_and_track_env_var(
     "--show-all-deprecations/--no-show-all-deprecations",
     envvar=None,
     help="By default, each type of a deprecation warning is only shown once. Use this flag to show all deprecation warning instances.",
@@ -603,7 +621,7 @@
     default=False,
 )
 
-skip_profile_setup = click.option(
+skip_profile_setup = _create_option_and_track_env_var(
     "--skip-profile-setup",
     "-s",
     envvar=None,
@@ -611,7 +629,7 @@
     is_flag=True,
 )
 
-source = click.option(
+source = _create_option_and_track_env_var(
     "--source",
     envvar=None,
     help="Source to download page from, must be one of hub, git, or local. Defaults to hub.",
@@ -619,7 +637,7 @@
     default="hub",
 )
 
-state = click.option(
+state = _create_option_and_track_env_var(
     "--state",
     envvar="DBT_STATE",
     help="Unless overridden, use this state directory for both state comparison and deferral.",
@@ -632,42 +650,42 @@
     ),
 )
 
-static = click.option(
+static = _create_option_and_track_env_var(
     "--static",
     help="Generate an additional static_index.html with manifest and catalog built-in.",
     default=False,
     is_flag=True,
 )
 
-static_parser = click.option(
+static_parser = _create_option_and_track_env_var(
     "--static-parser/--no-static-parser",
     envvar="DBT_STATIC_PARSER",
     help="Use the static parser.",
     default=True,
 )
 
-store_failures = click.option(
+store_failures = _create_option_and_track_env_var(
     "--store-failures",
     envvar="DBT_STORE_FAILURES",
     help="Store test results (failing rows) in the database",
     is_flag=True,
 )
 
-target = click.option(
+target = _create_option_and_track_env_var(
     "--target",
     "-t",
     envvar="DBT_TARGET",
     help="Which target to load for the given profile",
 )
 
-target_path = click.option(
+target_path = _create_option_and_track_env_var(
     "--target-path",
     envvar="DBT_TARGET_PATH",
     help="Configure the 'target-path'. Only applies this setting for the current run. Overrides the 'DBT_TARGET_PATH' if it is set.",
     type=click.Path(),
 )
 
-threads = click.option(
+threads = _create_option_and_track_env_var(
     "--threads",
     envvar=None,
     help="Specify number of threads to use while executing models. Overrides settings in profiles.yml.",
@@ -675,41 +693,41 @@
     type=click.INT,
 )
 
-upgrade = click.option(
+upgrade = _create_option_and_track_env_var(
     "--upgrade",
     envvar=None,
     help="Upgrade packages to the latest version.",
     is_flag=True,
 )
 
-use_colors = click.option(
+use_colors = _create_option_and_track_env_var(
     "--use-colors/--no-use-colors",
     envvar="DBT_USE_COLORS",
     help="Specify whether log output is colorized in the console and the log file. Use --use-colors-file/--no-use-colors-file to colorize the log file differently than the console.",
     default=True,
 )
 
-use_colors_file = click.option(
+use_colors_file = _create_option_and_track_env_var(
     "--use-colors-file/--no-use-colors-file",
     envvar="DBT_USE_COLORS_FILE",
     help="Specify whether log file output is colorized by overriding the default value and the general --use-colors/--no-use-colors setting.",
     default=True,
 )
 
-use_experimental_parser = click.option(
+use_experimental_parser = _create_option_and_track_env_var(
     "--use-experimental-parser/--no-use-experimental-parser",
     envvar="DBT_USE_EXPERIMENTAL_PARSER",
     help="Enable experimental parsing features.",
 )
 
-use_fast_test_edges = click.option(
+use_fast_test_edges = _create_option_and_track_env_var(
     "--use-fast-test-edges/--no-use-fast-test-edges",
     envvar="DBT_USE_FAST_TEST_EDGES",
     default=False,
     hidden=True,
 )
 
-vars = click.option(
+vars = _create_option_and_track_env_var(
     "--vars",
     envvar=None,
     help="Supply variables to the project. This argument overrides variables defined in your dbt_project.yml file. This argument should be a YAML string, eg. '{my_variable: my_value}'",
@@ -727,7 +745,7 @@ def _version_callback(ctx, _param, value):
     ctx.exit()
 
 
-version = click.option(
+version = _create_option_and_track_env_var(
     "--version",
     "-V",
     "-v",
@@ -739,14 +757,14 @@ def _version_callback(ctx, _param, value):
     is_flag=True,
 )
 
-version_check = click.option(
+version_check = _create_option_and_track_env_var(
     "--version-check/--no-version-check",
     envvar="DBT_VERSION_CHECK",
     help="If set, ensure the installed dbt version matches the require-dbt-version specified in the dbt_project.yml file (if any). Otherwise, allow them to differ.",
     default=True,
 )
 
-warn_error = click.option(
+warn_error = _create_option_and_track_env_var(
     "--warn-error",
     envvar="DBT_WARN_ERROR",
     help="If dbt would normally warn, instead raise an exception. Examples include --select that selects nothing, deprecations, configurations with no associated models, invalid test configurations, and missing sources/refs in tests.",
@@ -754,7 +772,7 @@ def _version_callback(ctx, _param, value):
     is_flag=True,
 )
 
-warn_error_options = click.option(
+warn_error_options = _create_option_and_track_env_var(
     "--warn-error-options",
     envvar="DBT_WARN_ERROR_OPTIONS",
     default="{}",
@@ -763,14 +781,14 @@ def _version_callback(ctx, _param, value):
     type=WarnErrorOptionsType(),
 )
 
-write_json = click.option(
+write_json = _create_option_and_track_env_var(
     "--write-json/--no-write-json",
     envvar="DBT_WRITE_JSON",
     help="Whether or not to write the manifest.json and run_results.json files to the target directory",
     default=True,
 )
 
-upload_artifacts = click.option(
+upload_artifacts = _create_option_and_track_env_var(
     "--upload-to-artifacts-ingest-api/--no-upload-to-artifacts-ingest-api",
     envvar="DBT_UPLOAD_TO_ARTIFACTS_INGEST_API",
     help="Whether or not to upload the artifacts to the dbt Cloud API",
diff --git a/core/dbt/cli/requires.py b/core/dbt/cli/requires.py
index dc2b3796a07..35eb9a92a12 100644
--- a/core/dbt/cli/requires.py
+++ b/core/dbt/cli/requires.py
@@ -3,7 +3,7 @@
 import time
 import traceback
 from functools import update_wrapper
-from typing import Optional
+from typing import Dict, Optional
 
 from click import Context
 
@@ -17,6 +17,7 @@
 from dbt.context.providers import generate_runtime_macro_context
 from dbt.context.query_header import generate_query_header_context
 from dbt.deprecations import show_deprecations_summary
+from dbt.env_vars import KNOWN_ENGINE_ENV_VARS, validate_engine_env_vars
 from dbt.events.logging import setup_event_logger
 from dbt.events.types import (
     ArtifactUploadError,
@@ -56,6 +57,17 @@
 from dbt_common.utils import cast_dict_to_dict_of_strings
 
 
+def _cross_propagate_engine_env_vars(env_dict: Dict[str, str]) -> None:
+    for env_var in KNOWN_ENGINE_ENV_VARS:
+        if env_var.old_name is not None:
+            # If the old name is in the env dict, and not the new name, set the new name based on the old name
+            if env_var.old_name in env_dict and env_var.name not in env_dict:
+                env_dict[env_var.name] = env_dict[env_var.old_name]
+            # If the new name is in the env dict, override the old name with it
+            elif env_var.name in env_dict:
+                env_dict[env_var.old_name] = env_dict[env_var.name]
+
+
 def preflight(func):
     def wrapper(*args, **kwargs):
         ctx = args[0]
@@ -69,7 +81,9 @@ def wrapper(*args, **kwargs):
 
         # Must be set after record/replay is set up so that the env can be
         # recorded or replayed if needed.
-        get_invocation_context()._env = get_env()
+        env_dict = get_env()
+        _cross_propagate_engine_env_vars(env_dict)
+        get_invocation_context()._env = env_dict
 
         # Flags
         flags = Flags(ctx)
@@ -108,6 +122,9 @@ def wrapper(*args, **kwargs):
         # Adapter management
         ctx.with_resource(adapter_management())
 
+        # Validate engine env var restricted name space
+        validate_engine_env_vars()
+
         return func(*args, **kwargs)
 
     return update_wrapper(wrapper, func)
diff --git a/core/dbt/deprecations.py b/core/dbt/deprecations.py
index 5803d71194c..60bbc5352d8 100644
--- a/core/dbt/deprecations.py
+++ b/core/dbt/deprecations.py
@@ -204,6 +204,11 @@ class ModelParamUsageDeprecation(DBTDeprecation):
     _event = "ModelParamUsageDeprecation"
 
 
+class EnvironmentVariableNamespaceDeprecation(DBTDeprecation):
+    _name = "environment-variable-namespace-deprecation"
+    _event = "EnvironmentVariableNamespaceDeprecation"
+
+
 def renamed_env_var(old_name: str, new_name: str):
     class EnvironmentVariableRenamed(DBTDeprecation):
         _name = f"environment-variable-renamed:{old_name}"
@@ -285,6 +290,7 @@ def show_deprecations_summary() -> None:
     ModelParamUsageDeprecation(),
     WEOInlcudeExcludeDeprecation(),
     SourceOverrideDeprecation(),
+    EnvironmentVariableNamespaceDeprecation(),
 ]
 
 deprecations: Dict[str, DBTDeprecation] = {d.name: d for d in deprecations_list}
diff --git a/core/dbt/env_vars.py b/core/dbt/env_vars.py
new file mode 100644
index 00000000000..3344fd8dda3
--- /dev/null
+++ b/core/dbt/env_vars.py
@@ -0,0 +1,61 @@
+from dataclasses import dataclass
+from typing import List, Optional
+
+from dbt.cli import params
+from dbt.deprecations import warn
+from dbt_common.constants import ENGINE_ENV_PREFIX
+from dbt_common.context import get_invocation_context
+
+# These are env vars that are not in the params module, but are still allowed to be set.
+# New additions to this list should use the new naming scheme, unless they are being added because
+# they already existed, but we didn't know about them previously.
+# TODO: Should at least some of these become (undocumented) cli param options?
+_ADDITIONAL_ENGINE_ENV_VARS: List[str] = [
+    "DBT_INVOCATION_ENV",
+    "DBT_RECORDED_FILE_PATH",
+    "DBT_TEST_STATE_MODIFIED",  # TODO: This is testing related, should we do this differently?
+    "DBT_PACKAGE_HUB_URL",
+    "DBT_DOWNLOAD_DIR",
+    "DBT_PP_FILE_DIFF_TEST",  # TODO: This is testing related, should we do this differently?
+    "DBT_PP_TEST",  # TODO: This is testing related, should we do this differently?
+]
+
+
+@dataclass(frozen=True)
+class EngineEnvVar:
+    name: str
+    old_name: Optional[str] = None
+
+
+def _create_engine_env_var(name: str) -> EngineEnvVar:
+    if name.startswith(ENGINE_ENV_PREFIX):
+        return EngineEnvVar(name=name)
+    elif name.startswith("DBT"):
+        return EngineEnvVar(name=name.replace("DBT", f"{ENGINE_ENV_PREFIX}"), old_name=name)
+    else:
+        raise RuntimeError(
+            f"Invalid environment variable: {name}, this will only happen if we add a new option to dbt that has an envvar that doesn't start with DBT_ or {ENGINE_ENV_PREFIX}"
+        )
+
+
+# Here we are creating a set of all known engine env vars. This is used in this moduleto create an allow list of dbt
+# engine env vars. We also use it in the cli flags module to cross propagate engine env vars with their old non-engine prefixed names.
+KNOWN_ENGINE_ENV_VARS: set[EngineEnvVar] = {
+    _create_engine_env_var(envvar)
+    for envvar in [*params.KNOWN_ENV_VARS, *_ADDITIONAL_ENGINE_ENV_VARS]
+}
+_ALLOWED_ENV_VARS: set[str] = {envvar.name for envvar in KNOWN_ENGINE_ENV_VARS}
+
+
+def validate_engine_env_vars() -> None:
+    """
+    Validate that any set environment variables that begin with the engine prefix are allowed.
+    """
+    env_vars = get_invocation_context()._env
+    for env_var in env_vars.keys():
+        if env_var.startswith(ENGINE_ENV_PREFIX) and env_var not in _ALLOWED_ENV_VARS:
+            warn(
+                "environment-variable-namespace-deprecation",
+                env_var=env_var,
+                reserved_prefix=ENGINE_ENV_PREFIX,
+            )
diff --git a/core/dbt/events/types.py b/core/dbt/events/types.py
index d72eac04b63..6758bc689f4 100644
--- a/core/dbt/events/types.py
+++ b/core/dbt/events/types.py
@@ -722,6 +722,15 @@ def message(self) -> str:
         return line_wrap_message(deprecation_tag(description))
 
 
+class EnvironmentVariableNamespaceDeprecation(WarnLevel):
+    def code(self) -> str:
+        return "D036"
+
+    def message(self) -> str:
+        description = f"Found custom environment variable `{self.env_var}` in the environment. The prefix `{self.reserved_prefix}` reserved for dbt engine environment variables. Custom environment variables with the prefix `{self.reserved_prefix}` may cause collisions and runtime errors."
+        return line_wrap_message(deprecation_tag(description))
+
+
 # =======================================================
 # I - Project parsing
 # =======================================================
diff --git a/core/setup.py b/core/setup.py
index 15a0b53ad3d..f73629c55c2 100644
--- a/core/setup.py
+++ b/core/setup.py
@@ -72,9 +72,9 @@
         "dbt-extractor>=0.5.0,<=0.6",
         "dbt-semantic-interfaces>=0.8.3,<0.9",
         # Minor versions for these are expected to be backwards-compatible
-        "dbt-common>=1.25.1,<2.0",
+        "dbt-common>=1.26.0,<2.0",
         "dbt-adapters>=1.15.2,<2.0",
-        "dbt-protos>=1.0.335,<2.0",
+        "dbt-protos>=1.0.337,<2.0",
         "pydantic<3",
         # ----
         # Expect compatibility with all new versions of these packages, so lower bounds only.
diff --git a/tests/functional/cli/test_requires.py b/tests/functional/cli/test_requires.py
new file mode 100644
index 00000000000..23e023d5a91
--- /dev/null
+++ b/tests/functional/cli/test_requires.py
@@ -0,0 +1,46 @@
+import os
+
+import pytest
+from pytest_mock import MockerFixture
+
+from dbt.events.types import JinjaLogInfo
+from dbt.tests.util import run_dbt
+from tests.utils import EventCatcher
+
+model_one_sql = """
+    {{ log("DBT_ENGINE_SHOW_RESOURCE_REPORT: " ~ env_var('DBT_ENGINE_SHOW_RESOURCE_REPORT', default="0"), info=True) }}
+    {{ log("DBT_SHOW_RESOURCE_REPORT: " ~ env_var('DBT_SHOW_RESOURCE_REPORT', default="0"), info=True) }}
+    select 1 as fun
+"""
+
+
+class TestOldEngineEnvVarPropagation:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {"model_one.sql": model_one_sql}
+
+    @pytest.mark.parametrize(
+        "set_old,set_new, expect",
+        [(False, False, 0), (True, False, False), (False, True, True), (True, True, True)],
+    )
+    def test_engine_env_var_propagation(
+        self, project, mocker: MockerFixture, set_old: bool, set_new: bool, expect: bool
+    ):
+        # Of note, the default value for DBT_PARTIAL_PARSE is True
+        if set_old:
+            mocker.patch.dict(os.environ, {"DBT_SHOW_RESOURCE_REPORT": "False"})
+        if set_new:
+            mocker.patch.dict(os.environ, {"DBT_ENGINE_SHOW_RESOURCE_REPORT": "True"})
+
+        event_catcher = EventCatcher(event_to_catch=JinjaLogInfo)
+        run_dbt(["parse", "--no-partial-parse"], callbacks=[event_catcher.catch])
+
+        assert len(event_catcher.caught_events) == 2
+
+        for event in event_catcher.caught_events:
+            if event.data.msg.startswith("DBT_ENGINE_SHOW_RESOURCE_REPORT"):
+                assert event.data.msg.endswith(f"{expect}")
+            elif event.data.msg.startswith("DBT_SHOW_RESOURCE_REPORT"):
+                assert event.data.msg.endswith(f"{expect}")
+            else:
+                assert False, "Unexpected log message"
diff --git a/tests/functional/deprecations/test_deprecations.py b/tests/functional/deprecations/test_deprecations.py
index 589c020a1d1..bbc3c835d5a 100644
--- a/tests/functional/deprecations/test_deprecations.py
+++ b/tests/functional/deprecations/test_deprecations.py
@@ -16,6 +16,7 @@
     CustomOutputPathInSourceFreshnessDeprecation,
     DeprecationsSummary,
     DuplicateYAMLKeysDeprecation,
+    EnvironmentVariableNamespaceDeprecation,
     GenericJSONSchemaValidationDeprecation,
     ModelParamUsageDeprecation,
     PackageRedirectDeprecation,
@@ -583,3 +584,22 @@ def test_select_usage(self, project):
         assert len(event_catcher.caught_events) == 0
         dbtRunner(callbacks=[event_catcher.catch]).invoke(["ls", "--select", "some_model"])
         assert len(event_catcher.caught_events) == 0
+
+
+class TestEnvironmentVariableNamespaceDeprecation:
+    @mock.patch.dict(
+        os.environ,
+        {
+            "DBT_ENGINE_PARTIAL_PARSE": "False",
+            "DBT_ENGINE_MY_CUSTOM_ENV_VAR_FOR_TESTING": "True",
+        },
+    )
+    def test_environment_variable_namespace_deprecation(self):
+        event_catcher = EventCatcher(event_to_catch=EnvironmentVariableNamespaceDeprecation)
+
+        run_dbt(["parse", "--show-all-deprecations"], callbacks=[event_catcher.catch])
+        assert len(event_catcher.caught_events) == 1
+        assert (
+            "DBT_ENGINE_MY_CUSTOM_ENV_VAR_FOR_TESTING"
+            == event_catcher.caught_events[0].data.env_var
+        )
diff --git a/tests/unit/test_env_vars.py b/tests/unit/test_env_vars.py
new file mode 100644
index 00000000000..02e232e128a
--- /dev/null
+++ b/tests/unit/test_env_vars.py
@@ -0,0 +1,49 @@
+import os
+from unittest import mock
+
+from dbt.deprecations import EnvironmentVariableNamespaceDeprecation as EVND
+from dbt.deprecations import active_deprecations
+from dbt.env_vars import KNOWN_ENGINE_ENV_VARS, validate_engine_env_vars
+from dbt.events.types import EnvironmentVariableNamespaceDeprecation
+from dbt.tests.util import safe_set_invocation_context
+from dbt_common.events.event_manager_client import add_callback_to_manager
+from tests.utils import EventCatcher
+
+
+@mock.patch.dict(
+    os.environ,
+    {
+        "DBT_ENGINE_PARTIAL_PARSE": "False",
+        "DBT_ENGINE_MY_CUSTOM_ENV_VAR_FOR_TESTING": "True",
+    },
+)
+def test_validate_engine_env_vars():
+    safe_set_invocation_context()
+    event_catcher = EventCatcher(event_to_catch=EnvironmentVariableNamespaceDeprecation)
+    add_callback_to_manager(event_catcher.catch)
+
+    validate_engine_env_vars()
+    # If it's zero, then we _failed_ to notice the deprecation instance (and we should look why the custom engine env var wasn't noticed)
+    # If it's more than one, then we're getting too many deprecation instances (and we should check what the other env vars identified were)
+    assert active_deprecations[EVND().name] == 1
+    assert (
+        "DBT_ENGINE_MY_CUSTOM_ENV_VAR_FOR_TESTING" == event_catcher.caught_events[0].data.env_var
+    )
+
+
+def test_engine_env_vars_with_old_names_has_not_increased():
+    engine_env_vars_with_old_names = sum(
+        1 for env_var in KNOWN_ENGINE_ENV_VARS if env_var.old_name is not None
+    )
+
+    # This failing means we either:
+    # 1. incorrectly created a new engine environment variable without using the `DBT_ENGINE` prefix
+    # 2. we've identified, and added, an existing but previously unknown engine env var to the _ADDITIONAL_ENGINE_ENV_VARS list.
+    # 3. we've _removed_ an existing engine env var with an old name (unlikely)
+    #
+    # In the case of (1), we should correct the new engine environent variable name
+    # In the case of (2), we should increase the number here.
+    # In the case of (3), we should decrease the number here.
+    assert (
+        engine_env_vars_with_old_names == 65
+    ), "We've added a new engine env var _without_ using the new naming scheme"
diff --git a/tests/unit/test_events.py b/tests/unit/test_events.py
index 0462a949c49..323ba7ff393 100644
--- a/tests/unit/test_events.py
+++ b/tests/unit/test_events.py
@@ -171,6 +171,7 @@ def test_event_codes(self):
     core_types.PropertyMovedToConfigDeprecation(key="", key_path="", file=""),
     core_types.WEOIncludeExcludeDeprecation(found_include=True, found_exclude=True),
     core_types.ModelParamUsageDeprecation(),
+    core_types.EnvironmentVariableNamespaceDeprecation(env_var="", reserved_prefix=""),
     # E - DB Adapter ======================
     adapter_types.AdapterEventDebug(),
     adapter_types.AdapterEventInfo(),


Code review comments:
diff_hunk: @@ -0,0 +1,61 @@
+from dataclasses import dataclass
+from typing import List, Optional
+
+from dbt.cli import params
+from dbt.deprecations import warn
+from dbt_common.constants import ENGINE_ENV_PREFIX
+from dbt_common.context import get_invocation_context
+
+# These are env vars that are not in the params module, but are still allowed to be set.
+# New additions to this list should use the new naming scheme, unless they are being added because
+# they already existed, but we didn't know about them previously.
+# TODO: Should at least some of these become (undocumented) cli param options?
+_ADDITIONAL_ENGINE_ENV_VARS: List[str] = [
+    "DBT_INVOCATION_ENV",
+    "DBT_RECORDED_FILE_PATH",
+    "DBT_TEST_STATE_MODIFIED",  # TODO: This is testing related, should we do this differently?
+    "DBT_PACKAGE_HUB_URL",
+    "DBT_DOWNLOAD_DIR",
+    "DBT_PP_FILE_DIFF_TEST",  # TODO: This is testing related, should we do this differently?
+    "DBT_PP_TEST",  # TODO: This is testing related, should we do this differently?
+]
+
+
+@dataclass(frozen=True)
+class EngineEnvVar:
+    name: str
+    old_name: Optional[str] = None
+
+
+def _create_engine_env_var(name: str) -> EngineEnvVar:
+    if name.startswith(ENGINE_ENV_PREFIX):
+        return EngineEnvVar(name=name)
+    elif name.startswith("DBT"):
+        return EngineEnvVar(name=name.replace("DBT", f"{ENGINE_ENV_PREFIX}"), old_name=name)
+    else:
+        raise RuntimeError(
+            f"Invalid environment variable: {name}, this will only happen if we add a new option to dbt that has an envvar that doesn't start with DBT_ or {ENGINE_ENV_PREFIX}"
path: core/dbt/env_vars.py
text: @user1:
Sounds like this reflects an issue with the dbt engine itself. Should this be a DbtInternalError instead?

diff_hunk: @@ -722,6 +722,15 @@ def message(self) -> str:
         return line_wrap_message(deprecation_tag(description))
 
 
+class EnvironmentVariableNamespaceDeprecation(WarnLevel):
+    def code(self) -> str:
+        return "D036"
+
+    def message(self) -> str:
+        description = f"Found custom environment variable `{self.env_var}` in the environment. The prefix `{self.reserved_prefix}` reserved for dbt engine environment variables. Custom environment variables with the prefix `{self.reserved_prefix}` may cause collisions and runtime errors."
path: core/dbt/events/types.py
text: @user1:
nit: "The prefix `{self.reserved_prefix}` _is_ reserved..."

diff_hunk: @@ -0,0 +1,61 @@
+from dataclasses import dataclass
+from typing import List, Optional
+
+from dbt.cli import params
+from dbt.deprecations import warn
+from dbt_common.constants import ENGINE_ENV_PREFIX
+from dbt_common.context import get_invocation_context
+
+# These are env vars that are not in the params module, but are still allowed to be set.
+# New additions to this list should use the new naming scheme, unless they are being added because
+# they already existed, but we didn't know about them previously.
+# TODO: Should at least some of these become (undocumented) cli param options?
+_ADDITIONAL_ENGINE_ENV_VARS: List[str] = [
+    "DBT_INVOCATION_ENV",
+    "DBT_RECORDED_FILE_PATH",
+    "DBT_TEST_STATE_MODIFIED",  # TODO: This is testing related, should we do this differently?
+    "DBT_PACKAGE_HUB_URL",
+    "DBT_DOWNLOAD_DIR",
+    "DBT_PP_FILE_DIFF_TEST",  # TODO: This is testing related, should we do this differently?
+    "DBT_PP_TEST",  # TODO: This is testing related, should we do this differently?
+]
+
+
+@dataclass(frozen=True)
+class EngineEnvVar:
+    name: str
+    old_name: Optional[str] = None
+
+
+def _create_engine_env_var(name: str) -> EngineEnvVar:
path: core/dbt/env_vars.py
text: @user1:
nit: perhaps this would be better-suited as a post_init method on EngineEnvVar

@author:
I think I'm actually gonna make this one an `__init__` instead of `__post_init__`. The reason being we're partially determining whether the passed in argument should be the `name` or the `old_name`, and that feels more `__init__`-y