Repository: dbt-labs/dbt-core

Issue:
[CT-3294] [Bug] `dbt retry` does not respect target-path CLI flag / env variable
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

The current implementation of `dbt retry` seems to only look for `run_results.json` when:
* The default target-path is set (e.g. no configuration)
* target-path is set within `dbt_project.yml`

While the target-path configuration in `dbt_project.yml` can solve my issue today, I recognize that it has been deprecated and will likely no longer work in future versions. The other options I have are the CLI flag / environment variable, which when tested produce two different errors that boil down to these configurations not being supported.

### Expected Behavior

When using `dbt retry`, I should be able to:
* Surface a flag like `dbt retry --target-path /path/to/artifacts`
* Set an environment variable like `DBT_TARGET_PATH=/path/to/artifacts`

### Steps To Reproduce

1. Create a new file directory in dbt project (for me `test_artifacts`) and move `run_results.json` to that folder.
3. Set a flag or env var
  3a. `dbt retry --target-path test_artifacts`
```
➜ dbt retry --target-path test_artifacts
Usage: dbt retry [OPTIONS]
Try 'dbt retry -h' for help.

Error: No such option: --target-path Did you mean --target?
```
  3b. In my .zshrc, set `export DBT_TARGET_PATH=~/path/to/dbt/test_artifacts`, source shell and run `dbt retry`
```
➜ dbt retry      
15:34:02  Running with dbt=1.6.4
15:34:02  Registered adapter: snowflake=1.6.4
15:34:06  Encountered an error:
Runtime Error
  Could not find previous run in 'target' target directory
```

4. In dbt_project.yml, set target-path: test_artifacts, run `dbt retry`
```
➜ dbt retry                   
15:39:54  Running with dbt=1.6.4
15:39:54  [WARNING]: Deprecated functionality
The `target-path` config in `dbt_project.yml` has been deprecated, and will no
longer be supported in a future version of dbt-core. If you wish to write dbt
artifacts to a custom directory, please use the --target-path CLI flag or
DBT_TARGET_PATH env var instead.
15:39:54  Registered adapter: snowflake=1.6.4
15:39:54  Unable to do partial parsing because saved manifest not found. Starting full parse.
15:40:30  Found 2577 models, 61 snapshots, 3577 tests, 90 seeds, 4 operations, 1783 sources, 0 exposures, 0 metrics, 985 macros, 0 groups, 0 semantic models
...
Successful run
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: MacOS, Sonoma 14.0
- Python: 3.10.4
- dbt: snowflake=1.6.4
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_
[CT-3294] [Bug] `dbt retry` does not respect target-path CLI flag / env variable
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

The current implementation of `dbt retry` seems to only look for `run_results.json` when:
* The default target-path is set (e.g. no configuration)
* target-path is set within `dbt_project.yml`

While the target-path configuration in `dbt_project.yml` can solve my issue today, I recognize that it has been deprecated and will likely no longer work in future versions. The other options I have are the CLI flag / environment variable, which when tested produce two different errors that boil down to these configurations not being supported.

### Expected Behavior

When using `dbt retry`, I should be able to:
* Surface a flag like `dbt retry --target-path /path/to/artifacts`
* Set an environment variable like `DBT_TARGET_PATH=/path/to/artifacts`

### Steps To Reproduce

1. Create a new file directory in dbt project (for me `test_artifacts`) and move `run_results.json` to that folder.
3. Set a flag or env var
  3a. `dbt retry --target-path test_artifacts`
```
➜ dbt retry --target-path test_artifacts
Usage: dbt retry [OPTIONS]
Try 'dbt retry -h' for help.

Error: No such option: --target-path Did you mean --target?
```
  3b. In my .zshrc, set `export DBT_TARGET_PATH=~/path/to/dbt/test_artifacts`, source shell and run `dbt retry`
```
➜ dbt retry      
15:34:02  Running with dbt=1.6.4
15:34:02  Registered adapter: snowflake=1.6.4
15:34:06  Encountered an error:
Runtime Error
  Could not find previous run in 'target' target directory
```

4. In dbt_project.yml, set target-path: test_artifacts, run `dbt retry`
```
➜ dbt retry                   
15:39:54  Running with dbt=1.6.4
15:39:54  [WARNING]: Deprecated functionality
The `target-path` config in `dbt_project.yml` has been deprecated, and will no
longer be supported in a future version of dbt-core. If you wish to write dbt
artifacts to a custom directory, please use the --target-path CLI flag or
DBT_TARGET_PATH env var instead.
15:39:54  Registered adapter: snowflake=1.6.4
15:39:54  Unable to do partial parsing because saved manifest not found. Starting full parse.
15:40:30  Found 2577 models, 61 snapshots, 3577 tests, 90 seeds, 4 operations, 1783 sources, 0 exposures, 0 metrics, 985 macros, 0 groups, 0 semantic models
...
Successful run
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: MacOS, Sonoma 14.0
- Python: 3.10.4
- dbt: snowflake=1.6.4
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_


Discussion:
Thanks for raising this @kevin-chao !

This looks like it was solved in [dbt-core 1.6.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md#dbt-core-166---october-11-2023) by #8803:
> - Override path-like args in dbt retry ([#8682](https://github.com/dbt-labs/dbt-core/issues/8682))

Could you give this a shot and see if it solves this for you?
Hey @dbeatty10, thanks for the quick triage! I gave it a try with `dbt-core==1.6.6`, but no dice. Here are some logs when using `DBT_TARGET_PATH` as an environment variable (dbt retry doesn't take the --target-path flag) that I'm hoping helps debug.

When using `dbt retry` set to `test/` directory
```bash
============================== 14:52:08.998563 | d7da2086-a55a-48f9-b343-9a8bda1ab450 ==============================
[0m14:52:08.998563 [info ] [MainThread]: Running with dbt=1.6.6
[0m14:52:08.998951 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/Users/kevin/.dbt', 'debug': 'False', 'version_check': 'True', 'log_path': '/Users/kevin/dbt/logs', 'fail_fast': 'False', 'warn_error': 'None', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'static_parser': 'True', 'invocation_command': 'dbt retry', 'introspect': 'True', 'target_path': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'send_anonymous_usage_stats': 'True'}
[0m14:52:09.186531 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105ca58b0>]}
[0m14:52:09.197814 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x107803a60>]}
[0m14:52:09.198372 [info ] [MainThread]: Registered adapter: snowflake=1.6.4
[0m14:52:09.240377 [debug] [MainThread]: checksum: 546b81fb56652c304d87abd676e84d4737d8a0c6b62160f4a6e79dcddbc842bb, vars: {}, profile: , target: , version: 1.6.6
[0m14:52:10.576116 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
[0m14:52:10.576597 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
[0m14:52:11.008990 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x125f3eee0>]}
[0m14:52:12.206905 [error] [MainThread]: Encountered an error:
Runtime Error
  Could not find previous run in 'target' target directory
[0m14:52:12.207506 [debug] [MainThread]: Command `dbt retry` failed at 14:52:12.207445 after 3.24 seconds
[0m14:52:12.207786 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1034eeb80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x152d58fd0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x152d58fa0>]}
[0m14:52:12.208054 [debug] [MainThread]: Flushing usage events
[0m14:53:55.114401 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105c2efa0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108431ee0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1084465b0>]}
```

When using `dbt run` set to `test/` directory:
```bash
============================== 14:53:55.117193 | 4ad12cdc-9bf6-4e32-be39-78f854e8e0d2 ==============================
[0m14:53:55.117193 [info ] [MainThread]: Running with dbt=1.6.6
[0m14:53:55.117858 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'profiles_dir': '/Users/kevin/.dbt', 'log_path': '/Users/kevin/dbt/logs', 'fail_fast': 'False', 'debug': 'False', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'introspect': 'True', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'target_path': '/Users/kevin/dbt/test', 'invocation_command': 'dbt run -s stg_google_ads__account_history', 'send_anonymous_usage_stats': 'True'}
[0m14:53:55.316020 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108431880>]}
[0m14:53:55.327699 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f8ebe0>]}
[0m14:53:55.328393 [info ] [MainThread]: Registered adapter: snowflake=1.6.4
[0m14:53:55.371382 [debug] [MainThread]: checksum: 546b81fb56652c304d87abd676e84d4737d8a0c6b62160f4a6e79dcddbc842bb, vars: {}, profile: , target: , version: 1.6.6
[0m14:53:55.373585 [info ] [MainThread]: Unable to do partial parsing because saved manifest not found. Starting full parse.
[0m14:53:55.373942 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f93850>]}
[0m14:54:08.840438 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'experimental_parser', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10cce7c70>]}
...
[0m14:54:34.093047 [debug] [MainThread]: Command end result
[0m14:54:34.729373 [info ] [MainThread]: 
[0m14:54:34.729889 [info ] [MainThread]: [32mCompleted successfully[0m
[0m14:54:34.730174 [info ] [MainThread]: 
[0m14:54:34.730419 [info ] [MainThread]: Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
[0m14:54:34.730822 [debug] [MainThread]: Command `dbt run` succeeded at 14:54:34.730767 after 39.65 seconds
[0m14:54:34.731093 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105c2efa0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f8ebe0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x12adc2430>]}
[0m14:54:34.731360 [debug] [MainThread]: Flushing usage events
```

Notably in the debug statement, the environment variable selectively shows up for target-path. I can also confirm that `dbt run` created files in /dbt/test but did not for `dbt retry`
Thanks for giving that a try @kevin-chao !

I tried it myself and was able to see the same thing as you.

It seems like the missing piece might be what you called out in your original issue:
- `dbt retry` doesn't support specifying the `--target-path` flag [here](https://github.com/dbt-labs/dbt-core/blob/bb21403c9e906097c4913edcf7a013d404e93d80/core/dbt/cli/main.py#L630-L646)

### Possible solution

In my local testing with an editable version of dbt-core, both `--target-path` and `DBT_TARGET_PATH` worked when I added this single line within [here](https://github.com/dbt-labs/dbt-core/blob/bb21403c9e906097c4913edcf7a013d404e93d80/core/dbt/cli/main.py#L630-L646):
```python
@p.target_path
```

#### Caveat

I get the following warning (which I think is expected due to https://github.com/dbt-labs/dbt-core/pull/8638 and https://github.com/dbt-labs/dbt-core/issues/8160#issuecomment-1708085600:

```
19:01:40  Warning: The state and target directories are the same: 'test_2'. This could lead to missing changes due to overwritten state including non-idempotent retries.
```

<details>

<summary>Toggle to see full log output</summary>

```
(core_editable) $ dbt retry --target-path test_2
19:01:40  Running with dbt=1.8.0-a1
19:01:40  Registered adapter: postgres=1.8.0-a1
19:01:40  Warning: The state and target directories are the same: 'test_2'. This could lead to missing changes due to overwritten state including non-idempotent retries.
19:01:40  Found 1 model, 0 sources, 0 exposures, 0 metrics, 401 macros, 0 groups, 0 semantic models
19:01:40  
19:01:40  Concurrency: 5 threads (target='postgres')
19:01:40  
19:01:40  1 of 1 START sql view model dbt_dbeatty.my_model ............................... [RUN]
19:01:40  1 of 1 OK created sql view model dbt_dbeatty.my_model .......................... [CREATE VIEW in 0.12s]
19:01:40  
19:01:40  Finished running 1 view model in 0 hours 0 minutes and 0.33 seconds (0.33s).
19:01:41  
19:01:41  Completed successfully
19:01:41  
19:01:41  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
```

</details>

### Acceptance criteria for implementation

- [ ] Include a functional test for `dbt retry --target-path`
- [ ] Include a functional test for `export DBT_TARGET_PATH=something` and `dbt retry`
- [ ] Make sure both tests fail
- [ ] Add `--target-path` to `dbt retry` by adding `@p.target_path` [here](https://github.com/dbt-labs/dbt-core/blob/bb21403c9e906097c4913edcf7a013d404e93d80/core/dbt/cli/main.py#L630-L646)
- [ ] Make sure both tests pass
My organization encountered this today and it's a bummer that the workaround is "use a deprecated `dbt_project.yml` configuration option". I'm looking forward to the resolution of this issue.
My team has also encountered this issue after updating dbt from 1.4.4 to 1.7.6. We would really appreciate the fix for this bug.
@graciegoheen  As of the change we added [here](https://github.com/dbt-labs/dbt-core/pull/9328/files#diff-af95c60156433e0ad9e3bf2754821f4511e6a9579f6414928fab7b0027f97b05R68), you can specify just `--state` path and it should find the `result.json` in the state dir and use that.
As @ChenyuLInx already mentioned, the snippet [here](https://github.com/dbt-labs/dbt-core/blob/8a1b9276f96def2871528572928e04c4414de425/core/dbt/task/retry.py#L68) shows why providing `--state` works.

I did a write-up in https://github.com/dbt-labs/dbt-core/issues/9575#issuecomment-1947598817 to give examples why we should enable `DBT_TARGET_PATH` and `--target-path`.

If we implement https://github.com/dbt-labs/dbt-core/issues/9575, then it would resolve https://github.com/dbt-labs/dbt-core/issues/8948 as well.
Thanks for raising this @kevin-chao !

This looks like it was solved in [dbt-core 1.6.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md#dbt-core-166---october-11-2023) by #8803:
> - Override path-like args in dbt retry ([#8682](https://github.com/dbt-labs/dbt-core/issues/8682))

Could you give this a shot and see if it solves this for you?
Hey @dbeatty10, thanks for the quick triage! I gave it a try with `dbt-core==1.6.6`, but no dice. Here are some logs when using `DBT_TARGET_PATH` as an environment variable (dbt retry doesn't take the --target-path flag) that I'm hoping helps debug.

When using `dbt retry` set to `test/` directory
```bash
============================== 14:52:08.998563 | d7da2086-a55a-48f9-b343-9a8bda1ab450 ==============================
[0m14:52:08.998563 [info ] [MainThread]: Running with dbt=1.6.6
[0m14:52:08.998951 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/Users/kevin/.dbt', 'debug': 'False', 'version_check': 'True', 'log_path': '/Users/kevin/dbt/logs', 'fail_fast': 'False', 'warn_error': 'None', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'static_parser': 'True', 'invocation_command': 'dbt retry', 'introspect': 'True', 'target_path': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'send_anonymous_usage_stats': 'True'}
[0m14:52:09.186531 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105ca58b0>]}
[0m14:52:09.197814 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x107803a60>]}
[0m14:52:09.198372 [info ] [MainThread]: Registered adapter: snowflake=1.6.4
[0m14:52:09.240377 [debug] [MainThread]: checksum: 546b81fb56652c304d87abd676e84d4737d8a0c6b62160f4a6e79dcddbc842bb, vars: {}, profile: , target: , version: 1.6.6
[0m14:52:10.576116 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
[0m14:52:10.576597 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
[0m14:52:11.008990 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x125f3eee0>]}
[0m14:52:12.206905 [error] [MainThread]: Encountered an error:
Runtime Error
  Could not find previous run in 'target' target directory
[0m14:52:12.207506 [debug] [MainThread]: Command `dbt retry` failed at 14:52:12.207445 after 3.24 seconds
[0m14:52:12.207786 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1034eeb80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x152d58fd0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x152d58fa0>]}
[0m14:52:12.208054 [debug] [MainThread]: Flushing usage events
[0m14:53:55.114401 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105c2efa0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108431ee0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1084465b0>]}
```

When using `dbt run` set to `test/` directory:
```bash
============================== 14:53:55.117193 | 4ad12cdc-9bf6-4e32-be39-78f854e8e0d2 ==============================
[0m14:53:55.117193 [info ] [MainThread]: Running with dbt=1.6.6
[0m14:53:55.117858 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'profiles_dir': '/Users/kevin/.dbt', 'log_path': '/Users/kevin/dbt/logs', 'fail_fast': 'False', 'debug': 'False', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'introspect': 'True', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'target_path': '/Users/kevin/dbt/test', 'invocation_command': 'dbt run -s stg_google_ads__account_history', 'send_anonymous_usage_stats': 'True'}
[0m14:53:55.316020 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108431880>]}
[0m14:53:55.327699 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f8ebe0>]}
[0m14:53:55.328393 [info ] [MainThread]: Registered adapter: snowflake=1.6.4
[0m14:53:55.371382 [debug] [MainThread]: checksum: 546b81fb56652c304d87abd676e84d4737d8a0c6b62160f4a6e79dcddbc842bb, vars: {}, profile: , target: , version: 1.6.6
[0m14:53:55.373585 [info ] [MainThread]: Unable to do partial parsing because saved manifest not found. Starting full parse.
[0m14:53:55.373942 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f93850>]}
[0m14:54:08.840438 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'experimental_parser', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10cce7c70>]}
...
[0m14:54:34.093047 [debug] [MainThread]: Command end result
[0m14:54:34.729373 [info ] [MainThread]: 
[0m14:54:34.729889 [info ] [MainThread]: [32mCompleted successfully[0m
[0m14:54:34.730174 [info ] [MainThread]: 
[0m14:54:34.730419 [info ] [MainThread]: Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
[0m14:54:34.730822 [debug] [MainThread]: Command `dbt run` succeeded at 14:54:34.730767 after 39.65 seconds
[0m14:54:34.731093 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105c2efa0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f8ebe0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x12adc2430>]}
[0m14:54:34.731360 [debug] [MainThread]: Flushing usage events
```

Notably in the debug statement, the environment variable selectively shows up for target-path. I can also confirm that `dbt run` created files in /dbt/test but did not for `dbt retry`
Thanks for giving that a try @kevin-chao !

I tried it myself and was able to see the same thing as you.

It seems like the missing piece might be what you called out in your original issue:
- `dbt retry` doesn't support specifying the `--target-path` flag [here](https://github.com/dbt-labs/dbt-core/blob/bb21403c9e906097c4913edcf7a013d404e93d80/core/dbt/cli/main.py#L630-L646)

### Possible solution

In my local testing with an editable version of dbt-core, both `--target-path` and `DBT_TARGET_PATH` worked when I added this single line within [here](https://github.com/dbt-labs/dbt-core/blob/bb21403c9e906097c4913edcf7a013d404e93d80/core/dbt/cli/main.py#L630-L646):
```python
@p.target_path
```

#### Caveat

I get the following warning (which I think is expected due to https://github.com/dbt-labs/dbt-core/pull/8638 and https://github.com/dbt-labs/dbt-core/issues/8160#issuecomment-1708085600:

```
19:01:40  Warning: The state and target directories are the same: 'test_2'. This could lead to missing changes due to overwritten state including non-idempotent retries.
```

<details>

<summary>Toggle to see full log output</summary>

```
(core_editable) $ dbt retry --target-path test_2
19:01:40  Running with dbt=1.8.0-a1
19:01:40  Registered adapter: postgres=1.8.0-a1
19:01:40  Warning: The state and target directories are the same: 'test_2'. This could lead to missing changes due to overwritten state including non-idempotent retries.
19:01:40  Found 1 model, 0 sources, 0 exposures, 0 metrics, 401 macros, 0 groups, 0 semantic models
19:01:40  
19:01:40  Concurrency: 5 threads (target='postgres')
19:01:40  
19:01:40  1 of 1 START sql view model dbt_dbeatty.my_model ............................... [RUN]
19:01:40  1 of 1 OK created sql view model dbt_dbeatty.my_model .......................... [CREATE VIEW in 0.12s]
19:01:40  
19:01:40  Finished running 1 view model in 0 hours 0 minutes and 0.33 seconds (0.33s).
19:01:41  
19:01:41  Completed successfully
19:01:41  
19:01:41  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
```

</details>

### Acceptance criteria for implementation

- [ ] Include a functional test for `dbt retry --target-path`
- [ ] Include a functional test for `export DBT_TARGET_PATH=something` and `dbt retry`
- [ ] Make sure both tests fail
- [ ] Add `--target-path` to `dbt retry` by adding `@p.target_path` [here](https://github.com/dbt-labs/dbt-core/blob/bb21403c9e906097c4913edcf7a013d404e93d80/core/dbt/cli/main.py#L630-L646)
- [ ] Make sure both tests pass
My organization encountered this today and it's a bummer that the workaround is "use a deprecated `dbt_project.yml` configuration option". I'm looking forward to the resolution of this issue.
My team has also encountered this issue after updating dbt from 1.4.4 to 1.7.6. We would really appreciate the fix for this bug.
@graciegoheen  As of the change we added [here](https://github.com/dbt-labs/dbt-core/pull/9328/files#diff-af95c60156433e0ad9e3bf2754821f4511e6a9579f6414928fab7b0027f97b05R68), you can specify just `--state` path and it should find the `result.json` in the state dir and use that.
As @ChenyuLInx already mentioned, the snippet [here](https://github.com/dbt-labs/dbt-core/blob/8a1b9276f96def2871528572928e04c4414de425/core/dbt/task/retry.py#L68) shows why providing `--state` works.

I did a write-up in https://github.com/dbt-labs/dbt-core/issues/9575#issuecomment-1947598817 to give examples why we should enable `DBT_TARGET_PATH` and `--target-path`.

If we implement https://github.com/dbt-labs/dbt-core/issues/9575, then it would resolve https://github.com/dbt-labs/dbt-core/issues/8948 as well.

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -595,6 +595,7 @@ def run(ctx, **kwargs):
 @p.vars
 @p.profile
 @p.target
+@p.target_path
 @p.threads
 @p.full_refresh
 @requires.postflight


Test patch:
diff --git a/tests/functional/retry/test_retry.py b/tests/functional/retry/test_retry.py
--- a/tests/functional/retry/test_retry.py
+++ b/tests/functional/retry/test_retry.py
@@ -1,3 +1,4 @@
+from pathlib import Path
 from shutil import copytree, move
 
 import pytest
@@ -327,3 +328,40 @@ def test_retry(self, project):
         # ...and so should this one, since the effect of the full-refresh parameter should persist.
         results = run_dbt(["retry"], expect_pass=False)
         assert len(results) == 1
+
+
+class TestRetryTargetPathEnvVar:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "sample_model.sql": models__sample_model,
+        }
+
+    def test_retry_target_path_env_var(self, project, monkeypatch):
+        monkeypatch.setenv("DBT_TARGET_PATH", "artifacts")
+        run_dbt(["run"], expect_pass=False)
+
+        write_file(models__second_model, "models", "sample_model.sql")
+
+        results = run_dbt(["retry"])
+        assert len(results) == 1
+
+
+class TestRetryTargetPathFlag:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "sample_model.sql": models__sample_model,
+        }
+
+    def test_retry_target_path_flag(self, project):
+        run_dbt(["run", "--target-path", "target"], expect_pass=False)
+
+        project_root = project.project_root
+        move(project_root / "target", project_root / "artifacts")
+
+        write_file(models__second_model, "models", "sample_model.sql")
+
+        results = run_dbt(["retry", "--state", "artifacts", "--target-path", "my_target_path"])
+        assert len(results) == 1
+        assert Path("my_target_path").is_dir()