Repository: dbt-labs/dbt-core

Issue:
[CT-2531] [Regression] v1.5.0 cannot run "dbt deps" in Dockerfile - "Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist."
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

If I am installing dbt v1.5.0 in a Dockerfile and then run `dbt deps` later, I get a strange failure:

```
Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist.
```

### Expected Behavior

I should be able to use `dbt deps` in my Dockerfile as I have before.

### Steps To Reproduce

With the following 4 files in a directory:

dbt_project.yml
```
# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: "foo"
version: "1.0.0"
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: "foo"

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

packages-install-path: "dbt_packages" # directory where packages are installed
clean-targets: # directories to be removed by `dbt clean`
  - "dbt_packages"
```

Dockerfile-1.4.2
```
FROM python:3.8.15-slim-bullseye

RUN apt-get update && apt-get install -y --no-install-recommends git

WORKDIR /dbt
COPY packages.yml dbt_project.yml /dbt/

RUN pip install --no-cache-dir dbt-core==1.4.2

RUN dbt deps
```

Dockerfile-1.5.0
```
FROM python:3.8.15-slim-bullseye

RUN apt-get update && apt-get install -y --no-install-recommends git

WORKDIR /dbt
COPY packages.yml dbt_project.yml /dbt/

RUN pip install --no-cache-dir dbt-core==1.5.0

RUN dbt deps
```

packages.yml
```
packages:
  - git: "https://github.com/EqualExperts/dbt-unit-testing"
    revision: v0.2.6
```

If I build my v1.4.2 image, it works:

```
docker build . --tag 1.4.2-deleteme --file Dockerfile-1.4.2
```

However, a similar command with v1.5.0 fails.

```
docker build . --tag 1.5.0-deleteme --file Dockerfile-1.5.0
```

```
Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist.
```


### Relevant log output

```shell
>docker build . --tag 1.5.0-deleteme --file Dockerfile-1.5.0 
[+] Building 2.9s (10/10) FINISHED
 => [internal] load build definition from Dockerfile-1.5.0                                                                                 0.0s 
 => => transferring dockerfile: 274B                                                                                                       0.0s 
 => [internal] load .dockerignore                                                                                                          0.0s 
 => => transferring context: 2B                                                                                                            0.0s 
 => [internal] load metadata for docker.io/library/python:3.8.15-slim-bullseye                                                             0.3s 
 => [internal] load build context                                                                                                          0.0s 
 => => transferring context: 68B                                                                                                           0.0s 
 => [1/6] FROM docker.io/library/python:3.8.15-slim-bullseye@sha256:b5abf044b0a94dbfc0463b9f7bd1e13010a096c64b98a441c7dabb5fb44b0308       0.0s 
 => CACHED [2/6] RUN apt-get update && apt-get install -y --no-install-recommends git                                                      0.0s 
 => CACHED [3/6] WORKDIR /dbt                                                                                                              0.0s 
 => CACHED [4/6] COPY packages.yml dbt_project.yml /dbt/                                                                                   0.0s 
 => CACHED [5/6] RUN pip install --no-cache-dir dbt-core==1.5.0                                                                            0.0s 
 => ERROR [6/6] RUN dbt deps                                                                                                               2.5s 
------
 > [6/6] RUN dbt deps:
#10 2.356 Usage: dbt deps [OPTIONS]
#10 2.356 Try 'dbt deps -h' for help.
#10 2.356
#10 2.356 Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist.
------
executor failed running [/bin/sh -c dbt deps]: exit code: 2
```

### Environment

_No response_

### Which database adapter are you using with dbt?

_No response_

### Additional Context

_No response_


Discussion:
Thanks for a nice a write-up @AndrewLane !

I haven't fully researched this, but starting in 1.5, the `--profiles-dir` path is being validated. When that parameter isn't supplied, the default is a subdirectory of the user's [home](https://github.com/dbt-labs/dbt-core/blob/fd7306643fd3a98e3232cbc15bf50eb2f3733603/core/dbt/cli/resolvers.py#L12-L13).

Without digging deeper, I don't know why `--profiles-dir` is an option for `dbt deps`.

### Workaround
In the meantime, here's a workaround you can use:
```shell
RUN dbt deps --profiles-dir .
```

Opting against `RUN dbt deps || true` as the recommended workaround because it would hide any other relevant failures during `dbt deps`.
After a little more digging, I found where this validation is happening:
https://github.com/dbt-labs/dbt-core/blob/fd7306643fd3a98e3232cbc15bf50eb2f3733603/core/dbt/cli/params.py#L292

There's at least four different ways we can choose to address this:
1. Leave all behavior as-is for 1.5 and add a note to the [migration guide](https://docs.getdbt.com/guides/migration/versions/upgrading-to-v1.5) so folks know how to make updates
1. Use `type=click.Path(exists=False)` to ignore path validation
1. Give `dbt deps` it's own custom version of the `--profiles-dir` parameter that ignores path validation (similar to Option 2 above)
1. Remove [`@p.profiles_dir`](https://github.com/dbt-labs/dbt-core/blob/fd7306643fd3a98e3232cbc15bf50eb2f3733603/core/dbt/cli/main.py#L427) from `dbt deps`

@jtcohen6 I'd be inclined to try out Option 2 since it would preserve the behavior of <=1.4 ubiquitously across commands. Alternatively, Option 4 would be appropriate if `dbt deps` doesn't actually need this parameter. Is there an option that you prefer?
I was going to add a new issue but I think it's just as relevant here. On a first time setup of dbt I am receiving the same error message when calling `dbt init jaffle_shop` as per the setup guide.

```
Usage: dbt init [OPTIONS] [PROJECT_NAME]
Try 'dbt init -h' for help.

Error: Invalid value for '--profiles-dir': Path '/home/girotobial/.dbt' does not exist.
```


Quick thoughts:
- `dbt deps` does not need `--profiles-dir`, it does not load a profile at all
- `dbt init` _does_ need `--profiles-dir`, but it's perfectly valid if it doesn't yet exist — the `init` command will write it!

I'm happy with Option 2: switch the parameter to `type=click.Path(exists=False)` to turn off this validation for all commands.

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -424,7 +424,7 @@ def debug(ctx, **kwargs):
 @cli.command("deps")
 @click.pass_context
 @p.profile
-@p.profiles_dir
+@p.profiles_dir_exists_false
 @p.project_dir
 @p.target
 @p.vars
@@ -446,7 +446,7 @@ def deps(ctx, **kwargs):
 # for backwards compatibility, accept 'project_name' as an optional positional argument
 @click.argument("project_name", required=False)
 @p.profile
-@p.profiles_dir
+@p.profiles_dir_exists_false
 @p.project_dir
 @p.skip_profile_setup
 @p.target
diff --git a/core/dbt/cli/params.py b/core/dbt/cli/params.py
--- a/core/dbt/cli/params.py
+++ b/core/dbt/cli/params.py
@@ -293,6 +293,8 @@
 )
 
 # `dbt debug` uses this because it implements custom behaviour for non-existent profiles.yml directories
+# `dbt deps` does not load a profile at all
+# `dbt init` will write profiles.yml if it doesn't yet exist
 profiles_dir_exists_false = click.option(
     "--profiles-dir",
     envvar="DBT_PROFILES_DIR",


Test patch:
diff --git a/tests/functional/profiles/test_profile_dir.py b/tests/functional/profiles/test_profile_dir.py
--- a/tests/functional/profiles/test_profile_dir.py
+++ b/tests/functional/profiles/test_profile_dir.py
@@ -9,6 +9,7 @@
 import dbt.flags as flags
 
 from dbt.tests.util import (
+    run_dbt,
     run_dbt_and_capture,
     write_file,
     rm_file,
@@ -89,6 +90,15 @@ def environ(env):
                 os.environ[key] = value
 
 
+class TestProfilesMayNotExist:
+    def test_debug(self, project):
+        # The database will not be able to connect; expect neither a pass or a failure (but not an exception)
+        run_dbt(["debug", "--profiles-dir", "does_not_exist"], expect_pass=None)
+
+    def test_deps(self, project):
+        run_dbt(["deps", "--profiles-dir", "does_not_exist"])
+
+
 class TestProfiles:
     def dbt_debug(self, project_dir_cli_arg=None, profiles_dir_cli_arg=None):
         # begin with no command-line args or user config (from profiles.yml)