Repository: dbt-labs/dbt-core

Issue:
[CT-259] [Bug]dbt deps and dbt clean stopped working on dbt-core-1.0.2
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Hi,

Starting today I have started receving the folling error error when trying to run `dbt deps` as well as `dbt clean`:
`
Encountered an error:
cannot pickle 'Undefined' object
`
After some investigation I could find that:
* the process was working fine till 2022-02-17 and then stopped working on 2022-02-18 so on the very same date that dbt-core-1.0.2 was released, I am using dbt-snowflake==1.0.0 so DBT is trying to pull latest dbt-core version and I can see in the logs that it was fine till it was getting dbt-core-1.0.1 and with dbt-core-1.0.2 it's failing, easy to spot because some of the pipelines in Azure DevOps just stopped working without any code change
* the issue occur in both Azure Devops pipelines (Microsoft Hosted Agents) as well as on the local VM (Linux) for newly configured environment (python virtul env)
* the only difference I can see from python packages installation is the dbt-core version and the click version
* the very same error appears for both `dbt deps` and `dbt clean`
* it's failing when running `dbt deps` as well as when trying to specify the profile manually `dbt deps --profile-dir . --target CI`
* there was a change for both `dbt deps` and `dbt clean` in the following PR #4610 so most likely it might be related, but cannot really find an issue myself
* of course temporary workaround would be to force re-installation of previous dbt-core installation as someone proposed in a bit similar other issue
* I am using at this point local DBT packages (copy of archived packages), but if the same issue is for `dbt clean` so maybe it's not the issue
* installation of dbt-snowflake==1.0.0 is working fine, installation of dbt-core==1.0.2 itself is fine too, the 
* I am using profile.yml with user and pass stored in the profile.yml file
* dbt.log is empty, no error message, no progress regarding clean or deps, if I would run any other command even `dbt debug` the log is populated correctly
* dbt debug, dbt run, dbt test seems to be working still fine

I am happy to provide more logs and isight if needed and possible.
I will continue some more testing so I will keep the issue updated.

Anyone maybe have had the same issue?

Best regards,
SD 

### Expected Behavior

_No response_

### Steps To Reproduce

_No response_

### Relevant log output

_No response_

### Environment

```markdown
- OS: Ubuntu 20.04.3
- Python: 3.8.12
- dbt: 1.0.2
```


### What database are you using dbt with?

snowflake

### Additional Context

_No response_


Discussion:
I was trying to replicate the same issue and on different PC for new DBT project I couldn't (tested on Windows, conda env, dbt-snowflake 1.00, dbt-core 1.0.2 on python 3.9.4 and 3.8.12)
Couldn't reproduce this either. The following works without issue.

```shell
python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install dbt-core==1.0.2
pip install dbt-snowflake
dbt clean && dbt deps
```

```shell
dbt deps --profiles-dir ~/.dbt
```

----

```shell
$ dbt --version
installed version: 1.0.2
   latest version: 1.0.2

Up to date!

Plugins:
  - snowflake: 1.0.0 - Up to date!
 
$ swvers
ProductName:    macOS
ProductVersion: 12.1
BuildVersion:   21C52
```

Python versions tested with.

```shell
$ python --version
Python 3.8.7

$ python --version
Python 3.9.7
```
Okay, once I was not able to reproduce and also saw the comment from jeremyyeo that he also can't reproduce I've started to remove everything from the project and try to run the deps or clean and finally got it.

So below is the code that is causing the issue. Just to be clear the intention here is to be able to parse the snowflake warehouse name as a variable and if there is none provided use the default one specifed in the profile.

This is causing the `dbt deps` and `dbt clean` failure
```
vars:
  warehouse_name: "{{ target.warehouse }}"

models:
  test:
    core:
      common:
        +snowflake_warehouse: "{{ var('warehouse_name'), target.warehouse }}"
```

That one will also not work
```
models:
  test:
    core:
      common:
        +snowflake_warehouse: "{{ target.warehouse }}"
```

That one will also not work
```
models:
  test:
    core:
      common:
        +snowflake_warehouse: "{{ target.name }}"
```

This is fine
```
models:
  test:
    core:
      common:
        +snowflake_warehouse: "my_warehouse_name"
```

As an example for query tags the reference to target seems to be fine, it's not validated/taken into consideration at this stage probably.

This is fine
```
models:
  test:
    core:
      common:
        +query_tag: "some tags, env: {{ target.name }}"
```

This is fine too
```
models:
  test:
    core:
      common:
        +query_tag: "some tags, env: {{ target.warehouse }}"
```

So it seems that starting with dbt-core==1.0.2 for dbt-snowflake==1.0.0 if `snowflake_warehouse` is about to get anything from `target` it is failing, but for other adapter specific variables like `query_tag` it is fine tough.

So maybe this is more related to dbt-snowflake package then? Hope this will help a bit. Will try to investigate a bit further soon..

This was tested on
```
Python: 3.8.10
System: Ubuntu 20.04.1
dbt-snowflake == 1.0.0
dbt-core == 1.0.2
```
Thanks for those details @sdrzymala! I was able to reproduce locally.

This is definitely the result of the change in #4610. We are no longer loading a profile for `deps` + `clean`, which means that `target` is set to `{}`, and any configuration that accesses a value from it will look like `Undefined`.

As an extension of that change, we also need to find a way to avoid loading / checking configuration for other project resources, since this information isn't strictly needed for either `deps` or `clean`, either. (I confirmed that `target` is not available to the `packages.yml` rendering context.)

As a temporary workaround, you can set a default value for when `target` is unset:
```yml
models:
  +any_config: "{{ target.name or 'default_value' }}"
```
Or just this:
```yml
models:
  +any_config: "{{ target.get('name') }}"
```
That has the effect of setting `None` (instead of `Undefined`) as the default value. When I set that default value locally, I can successfully run `dbt deps` + `dbt clean`.
Following the example above, after looking a bit further into this:
- In [`UnsetProfile`](https://github.com/dbt-labs/dbt-core/blob/a66fe7f4674eb5dcb1dcfb34652c4975da9eaa5a/core/dbt/config/runtime.py#L398-L399) + [`UnsetProfileConfig`](https://github.com/dbt-labs/dbt-core/blob/a66fe7f4674eb5dcb1dcfb34652c4975da9eaa5a/core/dbt/config/runtime.py#L432-L434), `to_target_dict` returns `{}`, so `target.name` is `Undefined`
- The issue crops up in [`Project.to_project_config()`](https://github.com/dbt-labs/dbt-core/blob/a66fe7f4674eb5dcb1dcfb34652c4975da9eaa5a/core/dbt/config/project.py#L573), specifically because `self.models` returns `{'+any_config': Undefined}`
- When the `Undefined` value is passed into `deepcopy`, we see this error: `cannot pickle 'Undefined' object`

So I think a solution here could look like either:
1. Finding a way to return `None` instead of `Undefined` from `to_target_dict`, without requiring that users add the `get()` logic above
2. Override `Project.to_project_config()` for `UnsetProfileConfig`, to avoid setting all configs that can use `target` in their rendering (`TargetContext` or subclass). Here's an extremely naive version (confirmed to resolve the issue): https://github.com/dbt-labs/dbt-core/commit/9c6142ef1b780532f85348059c765fa098bfa2f7
Good news! This continues to work just fine, since `Undefined` is falsy:
```yml
# dbt_project.yml
models:
  +any_config: "{{ 'something' if target.name == 'prod' else 'something_else' }}"
```
As does this:
```yml
models:
  +any_config: "{{ target.name }} + '_something'"
```
These feel like more common patterns, compared with setting config to a `target` value directly. It means that the failure case for this regression is much narrower; I'll do some more work to try to estimate exactly how narrow.
Thanks a lot!!! Indeed, I do confirm that both of the proposed solutions seems to be working fine.
Altough probably a small adjustment in the code would be still needed. Or at least if possible an improvement for error handling.

Thanks for the confirmation @sdrzymala! We're definitely planning a fix for this in v1.0.3.

All in all, it's pretty far from ideal that (even in older versions of dbt) this returns a very mysterious error:
```yml
# dbt_project.yml
models:
  +query_tag: "{{ target.databse }}" # typo
```
```
$ dbt clean
11:23:07  Encountered an error:
cannot pickle 'Undefined' object
```
This will raise an error with all dbt commands, including `dbt-rpc serve` (i.e. initializing a dbt Cloud develop server). It definitely makes me think that we should either:
- Turn `target` into a special object that returns `None` for missing dict entries
- Document and recomment that everyone use `target.get(<value>)`, instead of `target.<value>` or `target['value']`

Resolving patch:
diff --git a/core/dbt/config/runtime.py b/core/dbt/config/runtime.py
--- a/core/dbt/config/runtime.py
+++ b/core/dbt/config/runtime.py
@@ -14,7 +14,7 @@
 from .utils import parse_cli_vars
 from dbt import flags
 from dbt.adapters.factory import get_relation_class_by_name, get_include_paths
-from dbt.helper_types import FQNPath, PathSet
+from dbt.helper_types import FQNPath, PathSet, DictDefaultEmptyStr
 from dbt.config.profile import read_user_config
 from dbt.contracts.connection import AdapterRequiredConfig, Credentials
 from dbt.contracts.graph.manifest import ManifestMetadata
@@ -421,7 +421,7 @@ def __init__(self):
         self.threads = -1
 
     def to_target_dict(self):
-        return {}
+        return DictDefaultEmptyStr({})
 
     def __getattribute__(self, name):
         if name in {'profile_name', 'target_name', 'threads'}:
@@ -460,7 +460,7 @@ def __getattribute__(self, name):
 
     def to_target_dict(self):
         # re-override the poisoned profile behavior
-        return {}
+        return DictDefaultEmptyStr({})
 
     @classmethod
     def from_parts(
diff --git a/core/dbt/helper_types.py b/core/dbt/helper_types.py
--- a/core/dbt/helper_types.py
+++ b/core/dbt/helper_types.py
@@ -93,3 +93,10 @@ class NoValue(dbtClassMixin):
 
 FQNPath = Tuple[str, ...]
 PathSet = AbstractSet[FQNPath]
+
+
+# This class is used in to_target_dict, so that accesses to missing keys
+# will return an empty string instead of Undefined
+class DictDefaultEmptyStr(dict):
+    def __getitem__(self, key):
+        return dict.get(self, key, "")


Test patch:
diff --git a/test/integration/006_simple_dependency_test/test_simple_dependency.py b/test/integration/006_simple_dependency_test/test_simple_dependency.py
--- a/test/integration/006_simple_dependency_test/test_simple_dependency.py
+++ b/test/integration/006_simple_dependency_test/test_simple_dependency.py
@@ -257,6 +257,16 @@ def schema(self):
     def models(self):
         return "models"
 
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'models': {
+                '+any_config': "{{ target.name }}",
+                '+enabled': "{{ target.name in ['redshift', 'postgres'] | as_bool }}"
+            }
+        }
+
     def postgres_profile(self):
         # Need to set the environment variable here initially because
         # the unittest setup does a load_config.