Repository: dbt-labs/dbt-core

Issue:
[CT-767] [Feature] Supporting on-the-fly configuration of `target-path`
### Is this your first time opening an issue?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)

### Describe the Feature

A flag (like `--target-path=` or `--target-dir=`) to customize the [`target-path` variable](https://docs.getdbt.com/reference/project-configs/target-path) used by `dbt run` (and others) temporarily. If there is already a `target-path` set in the configuration file, this option will **override** it for the same `run`.

### Describe alternatives you've considered

An environment variable (like `DBT_TARGET_PATH`) would also be an alternative. The benefits include the ease of implementation (not needing to add the precise config option to each subcommand that needs to deal with `target-path`). But it would reduce the chances of discovery (since `--help` won't show it by default).

### Who will this benefit?

Anyone who has more than 1 DBT subprocess running at any given time. Our use case involves spawning multiple DBT subprocesses, and sometimes they clash and override each other's `run_results.json` files.

### Are you interested in contributing this feature?

Yes

### Anything else?

For more context, please see [this thread](https://getdbt.slack.com/archives/CBSQTAPLG/p1655767766784039) in DBT's slack channel.


Discussion:
@isidentical Thanks for opening!

This feels related to https://github.com/dbt-labs/dbt-core/issues/5132, in that there's a larger desire to override "path" configs set in `dbt_project.yml`. Similar to ["global" configs](https://docs.getdbt.com/reference/global-configs), I'd envision the order being:

1. CLI flag (`--target-path`)
2. env var (`DBT_TARGET_PATH`)
3. Config in version-controlled file (`dbt_project.yml`)

It's already possible to achieve the precedence in 2+3 by explicitly passing an environment variable into `dbt_project.yml`:
```yml
# dbt_project.yml
target-path: "{{ env_var('TARGET_PATH', 'target') }}"
```
```
$ TARGET_PATH=whatever dbt run
$ ls -1 whatever
compiled
graph.gpickle
manifest.json
partial_parse.msgpack
run
run_results.json
```
I think that sort of env var override should be the default/implicit behavior, too.

So, I'm not strictly opposed to the one-off addition in #5402. But what I'd really love is a consistent approach that:
- achieves this consistently for all "runtime" path configs (at least both `log-path` + `target-path`)
- explicitly enables setting these via CLI flag or `DBT_`-prefixed env var, following logic in [the `flags` module](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/flags.py)

Would you be interested in attempting that in #5402?
Also: We should find some way to square config that's set in `dbt_project.yml` versus `profiles.yml`, in cases where it makes sense to unify both under "global" configs. In the past, config that's relevant to _project directory files_ tended to go in the project file, but there's also a compelling proposal to support `profiles.yml` in the same directory by default: https://github.com/dbt-labs/dbt-core/issues/5411

That's out of the scope of this issue/PR, but it felt like a relevant place to leave the thought for now.
Thanks a lot for your answer @jtcohen6! Two quick questions before I start the implementation:

> I think that sort of env var override should be the default/implicit behavior, too.

> explicitly enables setting these via CLI flag or DBT_-prefixed env var, following logic in [the flags module](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/flags.py)

Should #5402 initially aim for adding only the environment variables (which would solve the issue for us / and for the other people who have the use cases I've shared) or should it add both the env vars and the cli flags?

 For the environment variables, are the following namings make sense or should we consider `DBT_CONFIG_` prefix?
- `DBT_TARGET_PATH`
- `DBT_LOG_PATH`
I'd much prefer to thread these through the same centralized `flags` logic linked above, rather than one-off calls to `os.getenv` nested deep in the `dbt_project.yml` parsing logic. At that point, it's probably not much additional work to support both CLI flags and env vars.

The `DBT_` prefix feels appropriate, and consistent with all other global configs.

Resolving patch:
diff --git a/core/dbt/config/project.py b/core/dbt/config/project.py
--- a/core/dbt/config/project.py
+++ b/core/dbt/config/project.py
@@ -15,7 +15,7 @@
 import hashlib
 import os
 
-from dbt import deprecations
+from dbt import flags, deprecations
 from dbt.clients.system import resolve_path_from_base
 from dbt.clients.system import path_exists
 from dbt.clients.system import load_file_contents
@@ -142,6 +142,13 @@ def _all_source_paths(
 T = TypeVar("T")
 
 
+def flag_or(flag: Optional[T], value: Optional[T], default: T) -> T:
+    if flag is None:
+        return value_or(value, default)
+    else:
+        return flag
+
+
 def value_or(value: Optional[T], default: T) -> T:
     if value is None:
         return default
@@ -356,9 +363,9 @@ def create_project(self, rendered: RenderComponents) -> "Project":
 
         docs_paths: List[str] = value_or(cfg.docs_paths, all_source_paths)
         asset_paths: List[str] = value_or(cfg.asset_paths, [])
-        target_path: str = value_or(cfg.target_path, "target")
+        target_path: str = flag_or(flags.TARGET_PATH, cfg.target_path, "target")
         clean_targets: List[str] = value_or(cfg.clean_targets, [target_path])
-        log_path: str = value_or(cfg.log_path, "logs")
+        log_path: str = flag_or(flags.LOG_PATH, cfg.log_path, "logs")
         packages_install_path: str = value_or(cfg.packages_install_path, "dbt_packages")
         # in the default case we'll populate this once we know the adapter type
         # It would be nice to just pass along a Quoting here, but that would
diff --git a/core/dbt/flags.py b/core/dbt/flags.py
--- a/core/dbt/flags.py
+++ b/core/dbt/flags.py
@@ -37,6 +37,8 @@
 QUIET = None
 NO_PRINT = None
 CACHE_SELECTED_ONLY = None
+TARGET_PATH = None
+LOG_PATH = None
 
 _NON_BOOLEAN_FLAGS = [
     "LOG_FORMAT",
@@ -44,6 +46,8 @@
     "PROFILES_DIR",
     "INDIRECT_SELECTION",
     "EVENT_BUFFER_SIZE",
+    "TARGET_PATH",
+    "LOG_PATH",
 ]
 
 _NON_DBT_ENV_FLAGS = ["DO_NOT_TRACK"]
@@ -71,6 +75,8 @@
     "QUIET": False,
     "NO_PRINT": False,
     "CACHE_SELECTED_ONLY": False,
+    "TARGET_PATH": None,
+    "LOG_PATH": None,
 }
 
 
@@ -121,6 +127,7 @@ def set_from_args(args, user_config):
     global WRITE_JSON, PARTIAL_PARSE, USE_COLORS, STORE_FAILURES, PROFILES_DIR, DEBUG, LOG_FORMAT
     global INDIRECT_SELECTION, VERSION_CHECK, FAIL_FAST, SEND_ANONYMOUS_USAGE_STATS
     global PRINTER_WIDTH, WHICH, LOG_CACHE_EVENTS, EVENT_BUFFER_SIZE, QUIET, NO_PRINT, CACHE_SELECTED_ONLY
+    global TARGET_PATH, LOG_PATH
 
     STRICT_MODE = False  # backwards compatibility
     # cli args without user_config or env var option
@@ -148,6 +155,8 @@ def set_from_args(args, user_config):
     QUIET = get_flag_value("QUIET", args, user_config)
     NO_PRINT = get_flag_value("NO_PRINT", args, user_config)
     CACHE_SELECTED_ONLY = get_flag_value("CACHE_SELECTED_ONLY", args, user_config)
+    TARGET_PATH = get_flag_value("TARGET_PATH", args, user_config)
+    LOG_PATH = get_flag_value("LOG_PATH", args, user_config)
 
     _set_overrides_from_env()
 
diff --git a/core/dbt/main.py b/core/dbt/main.py
--- a/core/dbt/main.py
+++ b/core/dbt/main.py
@@ -651,6 +651,22 @@ def _add_common_arguments(*subparsers):
             settings in profiles.yml.
             """,
         )
+        sub.add_argument(
+            "--target-path",
+            required=False,
+            help="""
+            Configure the 'target-path'. Only applies this setting for the
+            current run. Overrides the 'DBT_TARGET_PATH' if it is set.
+            """,
+        )
+        sub.add_argument(
+            "--log-path",
+            required=False,
+            help="""
+            Configure the 'log-path'. Only applies this setting for the
+            current run. Overrides the 'DBT_LOG_PATH' if it is set.
+            """,
+        )
         _add_version_check(sub)
 
 


Test patch:
diff --git a/test/integration/075_custom_target_path/models/do_nothing_1.sql b/test/integration/075_custom_target_path/models/do_nothing_1.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/075_custom_target_path/models/do_nothing_1.sql
@@ -0,0 +1 @@
+select 1 as x
diff --git a/test/integration/075_custom_target_path/models/do_nothing_2.sql b/test/integration/075_custom_target_path/models/do_nothing_2.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/075_custom_target_path/models/do_nothing_2.sql
@@ -0,0 +1 @@
+select 1 as y
diff --git a/test/integration/075_custom_target_path/models/do_nothing_3.sql b/test/integration/075_custom_target_path/models/do_nothing_3.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/075_custom_target_path/models/do_nothing_3.sql
@@ -0,0 +1 @@
+select 1 as z
diff --git a/test/integration/075_custom_target_path/test_target_path.py b/test/integration/075_custom_target_path/test_target_path.py
new file mode 100644
--- /dev/null
+++ b/test/integration/075_custom_target_path/test_target_path.py
@@ -0,0 +1,44 @@
+import os
+from unittest import mock
+from test.integration.base import DBTIntegrationTest, use_profile
+
+
+class TestTargetPathFromProjectConfig(DBTIntegrationTest):
+    @property
+    def project_config(self):
+        return {"config-version": 2, "target-path": "project_target"}
+
+    @property
+    def schema(self):
+        return "target_path_tests_075"
+
+    @property
+    def models(self):
+        return "models"
+
+    @use_profile("postgres")
+    def test_postgres_overriden_target_path(self):
+        results = self.run_dbt(args=["run"])
+        self.assertFalse(os.path.exists("./target"))
+        self.assertTrue(os.path.exists("./project_target"))
+
+
+class TestTargetPathOverridenEnv(TestTargetPathFromProjectConfig):
+    @use_profile("postgres")
+    def test_postgres_overriden_target_path(self):
+        with mock.patch.dict(os.environ, {"DBT_TARGET_PATH": "env_target"}):
+            results = self.run_dbt(args=["run"])
+        self.assertFalse(os.path.exists("./target"))
+        self.assertFalse(os.path.exists("./project_target"))
+        self.assertTrue(os.path.exists("./env_target"))
+
+
+class TestTargetPathOverridenEnvironment(TestTargetPathFromProjectConfig):
+    @use_profile("postgres")
+    def test_postgres_overriden_target_path(self):
+        with mock.patch.dict(os.environ, {"DBT_TARGET_PATH": "env_target"}):
+            results = self.run_dbt(args=["run", "--target-path", "cli_target"])
+        self.assertFalse(os.path.exists("./target"))
+        self.assertFalse(os.path.exists("./project_target"))
+        self.assertFalse(os.path.exists("./env_target"))
+        self.assertTrue(os.path.exists("./cli_target"))
diff --git a/test/unit/test_flags.py b/test/unit/test_flags.py
--- a/test/unit/test_flags.py
+++ b/test/unit/test_flags.py
@@ -250,3 +250,16 @@ def test__flags(self):
         os.environ.pop('DBT_CACHE_SELECTED_ONLY')
         delattr(self.args, 'cache_selected_only')
         self.user_config.cache_selected_only = False
+
+        # target_path/log_path
+        flags.set_from_args(self.args, self.user_config)
+        self.assertIsNone(flags.LOG_PATH)
+        os.environ['DBT_LOG_PATH'] = 'a/b/c'
+        flags.set_from_args(self.args, self.user_config)
+        self.assertEqual(flags.LOG_PATH, 'a/b/c')
+        setattr(self.args, 'log_path', 'd/e/f')
+        flags.set_from_args(self.args, self.user_config)
+        self.assertEqual(flags.LOG_PATH, 'd/e/f')
+        # cleanup
+        os.environ.pop('DBT_LOG_PATH')
+        delattr(self.args, 'log_path')