Repository: dbt-labs/dbt-core

Issue:
[CT-3137] [Bug] dbt retry: `Path`-type flags passed to `retry` should override params from previous invocation
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

Previous invocation's flags (loaded from `args` in `run_results.json`) are always used, even if CLI flags are explicitly passed into the `retry` command:

https://github.com/dbt-labs/dbt-core/blob/34ab4cf9be4ab0d81ab2fa39951d747eba327b2e/core/dbt/task/retry.py#L68

https://github.com/dbt-labs/dbt-core/blob/34ab4cf9be4ab0d81ab2fa39951d747eba327b2e/core/dbt/task/retry.py#L83-L95

### Expected Behavior

CLI flags passed explicitly into the `dbt retry` command should override flags read in from the previous invocation.

I could see this getting hairy when we think about the config hierarchy of flags > env vars > user configs (in profiles.yml). I think the precedence order should be:
- CLI flags passed explicitly into this invocation (dbt retry) — _plus_ `--state` which may also be passed via env var (!)
- Flags from previous invocation (including resolution of flags > env vars > user configs)

This assumes that the env vars (save `--state`) + user configs ought to be the same across both invocations. That is an _assumption_, but it feels like a reasonable one. What feels **bad** about this is that we're breaking the 1:1 mapping between CLI flags (`--project-dir`) and env vars (`DBT_PROJECT_DIR`), _just_ for `retry`.

### Steps To Reproduce

1. Create a dbt project in a directory named `proj_location_1/` with a failing model:
```
$ dbt run --project-dir proj_location_1/
17:31:53  Running with dbt=1.7.0-b2
17:31:53  Registered adapter: postgres=1.7.0-b2
17:31:53  Found 1 model, 0 sources, 0 exposures, 0 metrics, 598 macros, 0 groups, 0 semantic models
17:31:53
17:31:53  Concurrency: 5 threads (target='dev')
17:31:53
17:31:53  1 of 1 START sql view model dbt_jcohen.cool_model .............................. [RUN]
17:31:53  1 of 1 ERROR creating sql view model dbt_jcohen.cool_model ..................... [ERROR in 0.06s]
17:31:54
17:31:54  Finished running 1 view model in 0 hours 0 minutes and 0.23 seconds (0.23s).
17:31:54
17:31:54  Completed with 1 error and 0 warnings:
17:31:54
17:31:54    Database Error in model cool_model (models/cool_model.sql)
  column "aslkjfasd" does not exist
  LINE 7:     select aslkjfasd
                     ^
  compiled Code at target/run/my_dbt_project/models/cool_model.sql
17:31:54
17:31:54  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
```

2. Move the project location: `proj_location_1` &rarr; `proj_location_2`
```
$ mv proj_location_1/ proj_location_2/
```

3. Run `dbt retry` pointing to `proj_location_2`, but `dbt retry` still uses `proj_location_1`
```
$ dbt retry --project-dir proj_location_2/
17:33:26  Running with dbt=1.7.0-b2
17:33:26  Registered adapter: postgres=1.7.0-b2
17:33:26  Warning: The state and target directories are the same: 'target'. This could lead to missing changes due to overwritten state including non-idempotent retries.
17:33:26  Encountered an error:
Path 'proj_location_1/' does not exist.
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: 13.4.1
- Python: 3.10.11
- dbt: 1.7.0-b2 (main) - but also relevant on v1.6
```


### Which database adapter are you using with dbt?

postgres

### Additional Context

_No response_
[CT-3137] [Bug] dbt retry: `Path`-type flags passed to `retry` should override params from previous invocation
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

Previous invocation's flags (loaded from `args` in `run_results.json`) are always used, even if CLI flags are explicitly passed into the `retry` command:

https://github.com/dbt-labs/dbt-core/blob/34ab4cf9be4ab0d81ab2fa39951d747eba327b2e/core/dbt/task/retry.py#L68

https://github.com/dbt-labs/dbt-core/blob/34ab4cf9be4ab0d81ab2fa39951d747eba327b2e/core/dbt/task/retry.py#L83-L95

### Expected Behavior

CLI flags passed explicitly into the `dbt retry` command should override flags read in from the previous invocation.

I could see this getting hairy when we think about the config hierarchy of flags > env vars > user configs (in profiles.yml). I think the precedence order should be:
- CLI flags passed explicitly into this invocation (dbt retry) — _plus_ `--state` which may also be passed via env var (!)
- Flags from previous invocation (including resolution of flags > env vars > user configs)

This assumes that the env vars (save `--state`) + user configs ought to be the same across both invocations. That is an _assumption_, but it feels like a reasonable one. What feels **bad** about this is that we're breaking the 1:1 mapping between CLI flags (`--project-dir`) and env vars (`DBT_PROJECT_DIR`), _just_ for `retry`.

### Steps To Reproduce

1. Create a dbt project in a directory named `proj_location_1/` with a failing model:
```
$ dbt run --project-dir proj_location_1/
17:31:53  Running with dbt=1.7.0-b2
17:31:53  Registered adapter: postgres=1.7.0-b2
17:31:53  Found 1 model, 0 sources, 0 exposures, 0 metrics, 598 macros, 0 groups, 0 semantic models
17:31:53
17:31:53  Concurrency: 5 threads (target='dev')
17:31:53
17:31:53  1 of 1 START sql view model dbt_jcohen.cool_model .............................. [RUN]
17:31:53  1 of 1 ERROR creating sql view model dbt_jcohen.cool_model ..................... [ERROR in 0.06s]
17:31:54
17:31:54  Finished running 1 view model in 0 hours 0 minutes and 0.23 seconds (0.23s).
17:31:54
17:31:54  Completed with 1 error and 0 warnings:
17:31:54
17:31:54    Database Error in model cool_model (models/cool_model.sql)
  column "aslkjfasd" does not exist
  LINE 7:     select aslkjfasd
                     ^
  compiled Code at target/run/my_dbt_project/models/cool_model.sql
17:31:54
17:31:54  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
```

2. Move the project location: `proj_location_1` &rarr; `proj_location_2`
```
$ mv proj_location_1/ proj_location_2/
```

3. Run `dbt retry` pointing to `proj_location_2`, but `dbt retry` still uses `proj_location_1`
```
$ dbt retry --project-dir proj_location_2/
17:33:26  Running with dbt=1.7.0-b2
17:33:26  Registered adapter: postgres=1.7.0-b2
17:33:26  Warning: The state and target directories are the same: 'target'. This could lead to missing changes due to overwritten state including non-idempotent retries.
17:33:26  Encountered an error:
Path 'proj_location_1/' does not exist.
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: 13.4.1
- Python: 3.10.11
- dbt: 1.7.0-b2 (main) - but also relevant on v1.6
```


### Which database adapter are you using with dbt?

postgres

### Additional Context

_No response_


Discussion:
> CLI flags passed explicitly into this invocation

I am afraid that this will be especially tricky. We have access to the resolved `Flags`, but it will be much harder to know which were passed in explicitly as CLI flags, versus set as env vars / user config in `profiles.yml`. (This is available in the `click` Context via [`get_parameter_source`](https://click.palletsprojects.com/en/8.1.x/api/#click.Context.get_parameter_source), but not in our resolved `Flags` object.)

If we just use the `retry` command's fully resolved flags to overwrite the previous invocation's fully resolved flags, we will miss any CLI flag overrides that took precedence over default values for the previous command.

---

`--project-dir` also seems particularly tricky, because it's used as the input to set other params, such as `--log-path`. In this case, I think the user just needs to set both the `--project-dir` and `--log-path`.

---

**Update:** I'm not going to pretend that I like any part of this, but it does appear to kinda sorta get the job done [here](https://github.com/dbt-labs/dbt-core/blob/d91265411061463b42d6f4d6d94e094a42a96d2b/core/dbt/task/retry.py#L68-L69):
```python
        self.previous_args = self.previous_state.results.args

        # massive hack: allow 'dbt retry --explicit --cli-flags' to override previous flags
        # I do not know if/how this could also support **kwargs passed into dbtRunner.invoke
        new_cli_args = {
            k: v
            for k, v in self.args.__dict__.items()
            if f"--{k.replace('_','-')}" in self.args.invocation_command.split(" ")
        }
        for key, value in new_cli_args.items():
            self.previous_args[key] = value
            
        self.previous_command_name = self.previous_args.get("which")
```
```
$ dbt retry --project-dir proj_location_2/
20:22:21  Running with dbt=1.7.0-b2
20:22:21  Registered adapter: postgres=1.7.0-b2
20:22:21  Warning: The state and target directories are the same: 'target'. This could lead to missing changes due to overwritten state including non-idempotent retries.
20:22:21  Found 1 model, 0 sources, 0 exposures, 0 metrics, 598 macros, 0 groups, 0 semantic models
... works ...
```
@peter-bertuglia made a great point. In the event that we're retrying an invocation which had previously enabled **deferral**, the subsequent invocation will want:
- the ability to point `--defer-state` to different file paths in the subsequent (`retry`) invocation, versus the file paths used for deferral in the previous invocation
- the ability to set `--defer-state` via environment variable (`DBT_DEFER_STATE`) — but _only_ `--defer-state`, every other flag would need to be passed explicitly

As above, plus:
```python
        # massive hack: override previous args with new CLI args
        ...
        
        # also allow overriding DEFER_STATE via env var (but only this)
        # however: would this cause issues if the command being retried *doesn't* support --defer-state?
        # right now that is only true of 'run-operation'
        if self.args.defer_state:
            self.previous_args["defer_state"] = defer_state
```
After discussing further with @peterallenwebb: These "overrides" are really just needed for "path" configs specific to the file system, because the subsequent `retry` invocation may be running in a different file system from the original invocation being retried.

So rather than the hackery I had above ("only override configs that are present in the `invocation_command`"), we should prefer the current invocation's inputs for parameters which are of type `Path`. This includes [`project_dir`](https://github.com/dbt-labs/dbt-core/blob/d91265411061463b42d6f4d6d94e094a42a96d2b/core/dbt/cli/params.py#L345) and [`defer_state`](https://github.com/dbt-labs/dbt-core/blob/d91265411061463b42d6f4d6d94e094a42a96d2b/core/dbt/cli/params.py#L477).

> CLI flags passed explicitly into this invocation

I am afraid that this will be especially tricky. We have access to the resolved `Flags`, but it will be much harder to know which were passed in explicitly as CLI flags, versus set as env vars / user config in `profiles.yml`. (This is available in the `click` Context via [`get_parameter_source`](https://click.palletsprojects.com/en/8.1.x/api/#click.Context.get_parameter_source), but not in our resolved `Flags` object.)

If we just use the `retry` command's fully resolved flags to overwrite the previous invocation's fully resolved flags, we will miss any CLI flag overrides that took precedence over default values for the previous command.

---

`--project-dir` also seems particularly tricky, because it's used as the input to set other params, such as `--log-path`. In this case, I think the user just needs to set both the `--project-dir` and `--log-path`.

---

**Update:** I'm not going to pretend that I like any part of this, but it does appear to kinda sorta get the job done [here](https://github.com/dbt-labs/dbt-core/blob/d91265411061463b42d6f4d6d94e094a42a96d2b/core/dbt/task/retry.py#L68-L69):
```python
        self.previous_args = self.previous_state.results.args

        # massive hack: allow 'dbt retry --explicit --cli-flags' to override previous flags
        # I do not know if/how this could also support **kwargs passed into dbtRunner.invoke
        new_cli_args = {
            k: v
            for k, v in self.args.__dict__.items()
            if f"--{k.replace('_','-')}" in self.args.invocation_command.split(" ")
        }
        for key, value in new_cli_args.items():
            self.previous_args[key] = value
            
        self.previous_command_name = self.previous_args.get("which")
```
```
$ dbt retry --project-dir proj_location_2/
20:22:21  Running with dbt=1.7.0-b2
20:22:21  Registered adapter: postgres=1.7.0-b2
20:22:21  Warning: The state and target directories are the same: 'target'. This could lead to missing changes due to overwritten state including non-idempotent retries.
20:22:21  Found 1 model, 0 sources, 0 exposures, 0 metrics, 598 macros, 0 groups, 0 semantic models
... works ...
```
@peter-bertuglia made a great point. In the event that we're retrying an invocation which had previously enabled **deferral**, the subsequent invocation will want:
- the ability to point `--defer-state` to different file paths in the subsequent (`retry`) invocation, versus the file paths used for deferral in the previous invocation
- the ability to set `--defer-state` via environment variable (`DBT_DEFER_STATE`) — but _only_ `--defer-state`, every other flag would need to be passed explicitly

As above, plus:
```python
        # massive hack: override previous args with new CLI args
        ...
        
        # also allow overriding DEFER_STATE via env var (but only this)
        # however: would this cause issues if the command being retried *doesn't* support --defer-state?
        # right now that is only true of 'run-operation'
        if self.args.defer_state:
            self.previous_args["defer_state"] = defer_state
```
After discussing further with @peterallenwebb: These "overrides" are really just needed for "path" configs specific to the file system, because the subsequent `retry` invocation may be running in a different file system from the original invocation being retried.

So rather than the hackery I had above ("only override configs that are present in the `invocation_command`"), we should prefer the current invocation's inputs for parameters which are of type `Path`. This includes [`project_dir`](https://github.com/dbt-labs/dbt-core/blob/d91265411061463b42d6f4d6d94e094a42a96d2b/core/dbt/cli/params.py#L345) and [`defer_state`](https://github.com/dbt-labs/dbt-core/blob/d91265411061463b42d6f4d6d94e094a42a96d2b/core/dbt/cli/params.py#L477).


Resolving patch:
diff --git a/core/dbt/task/retry.py b/core/dbt/task/retry.py
--- a/core/dbt/task/retry.py
+++ b/core/dbt/task/retry.py
@@ -19,6 +19,16 @@
 from dbt.task.test import TestTask
 
 RETRYABLE_STATUSES = {NodeStatus.Error, NodeStatus.Fail, NodeStatus.Skipped, NodeStatus.RuntimeErr}
+OVERRIDE_PARENT_FLAGS = {
+    "log_path",
+    "output_path",
+    "profiles_dir",
+    "profiles_dir_exists_false",
+    "project_dir",
+    "defer_state",
+    "deprecated_state",
+    "target_path",
+}
 
 TASK_DICT = {
     "build": BuildTask,
@@ -91,7 +101,13 @@ def run(self):
             if k in self.previous_args and v(self.previous_args[k]):
                 del self.previous_args[k]
 
-        retry_flags = Flags.from_dict(cli_command, self.previous_args)
+        previous_args = {
+            k: v for k, v in self.previous_args.items() if k not in OVERRIDE_PARENT_FLAGS
+        }
+        current_args = {k: v for k, v in self.args.__dict__.items() if k in OVERRIDE_PARENT_FLAGS}
+        combined_args = {**previous_args, **current_args}
+
+        retry_flags = Flags.from_dict(cli_command, combined_args)
         retry_config = RuntimeConfig.from_args(args=retry_flags)
 
         class TaskWrapper(self.task_class):


Test patch:
diff --git a/tests/functional/retry/test_retry.py b/tests/functional/retry/test_retry.py
--- a/tests/functional/retry/test_retry.py
+++ b/tests/functional/retry/test_retry.py
@@ -1,3 +1,5 @@
+from shutil import copytree, move
+
 import pytest
 
 from dbt.contracts.results import RunStatus, TestStatus
@@ -271,3 +273,21 @@ def test_resource_type(self, project):
         # nothing to do
         results = run_dbt(["retry"])
         assert len(results) == 0
+
+
+class TestRetryOverridePath:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "sample_model.sql": models__sample_model,
+        }
+
+    def test_retry(self, project):
+        project_root = project.project_root
+        proj_location_1 = project_root / "proj_location_1"
+        proj_location_2 = project_root / "proj_location_2"
+
+        copytree(project_root, proj_location_1)
+        run_dbt(["run", "--project-dir", "proj_location_1"], expect_pass=False)
+        move(proj_location_1, proj_location_2)
+        run_dbt(["retry", "--project-dir", "proj_location_2"], expect_pass=False)