Repository: dbt-labs/dbt-core

Issue:
[CT-2516] [Bug] Upgrade to 1.5.0 generates "TypeError: argument of type 'PosixPath' is not iterable" on model runs.
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

We just upgraded our poetry packages for dbt-core and dbt-bigquery to 1.5.0.

When running: poetry run dbt --version everything works fine. But when calling a model like:

poetry run dbt run -s model_name
poetry run dbt build -s model_name

We get a new error never seen before.

### Expected Behavior

That the models get built as prior with the normal CLI output.

### Steps To Reproduce

1. Update pyproject.toml with setting dbt-core & dbt-bigquery versions to "1.5.0"
2. run "poetry update"
3. run "poetry run dbt --version"
4. Verify that it outputs the correct versions
![image](https://user-images.githubusercontent.com/17338299/235943843-93f8b61c-fbdf-4012-9afc-88b1b8ef9e68.png)
5. run "poetry run dbt run -s name_of_model" 

### Relevant log output

```shell
14:13:06  Running with dbt=1.5.0
14:13:49  Encountered an error:
argument of type 'PosixPath' is not iterable
14:13:49  Traceback (most recent call last):
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 86, in wrapper
    result, success = func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 71, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 142, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 215, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 242, in wrapper
    manifest = ManifestLoader.get_full_manifest(
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 227, in get_full_manifest
    manifest = loader.load()
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 386, in load
    self.parse_project(
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 539, in parse_project
    hook_parser.parse_file(file_block)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/hooks.py", line 120, in parse_file
    self.parse_node(hook)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/base.py", line 394, in parse_node
    self.render_update(node, config)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/base.py", line 370, in render_update
    context = self.render_with_context(node, config)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/base.py", line 240, in render_with_context
    get_rendered(parsed_node.raw_code, context, parsed_node, capture_macros=True)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/clients/jinja.py", line 591, in get_rendered
    return render_template(template, ctx, node)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/clients/jinja.py", line 546, in render_template
    return template.render(ctx)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
TypeError: argument of type 'PosixPath' is not iterable
```


### Environment

```markdown
- OS: Ubuntu 20.04
- Python: 3.8.10
- dbt: 1.5.0
```


### Which database adapter are you using with dbt?

bigquery

### Additional Context

_No response_


Discussion:
@christopherekfeldt Thanks for opening! I'm going to label this as a potential `regression` for v1.5.

It's weird that this is happening _during_ project parsing. Based on the error message and occurrence, I'd guess that you're accessing a flag that's being passed in, and trying to loop over it (?).

Educated guesses - any chance you're doing one of the following?
- Referencing the [`invocation_args_dict`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags#invocation_args_dict) context variable somewhere in your project
- Have a package installed that's doing this, i.e. [`dbt_artifacts`](https://github.com/brooklyn-data/dbt_artifacts/blob/139ebf87a4b56197196e5827c1af75ef9c1de8fe/macros/upload_invocations.sql#L66)

Also - does your poetry package implicitly set any other flag values for dbt, via CLI flag or env var, such as `--project-dir` or `--profiles-dir`?
No I don't think so. I searched it and found nothing in our project reflecting "invocation_args_dict".

These are the packages specified in our packages.yml:

packages:
  - package: dbt-labs/dbt_utils
    version: 1.0.0

  - package: dbt-labs/codegen
    version: 0.9.0

  - package: elementary-data/elementary
    version: 0.7.5

  - package: calogica/dbt_expectations
    version: 0.8.5

On the last part my answer is no, we only have one project dir and a common profile in our dbt root folder.
The only connection to dbt apart from installing the tools dbt-core and dbt-bigquery with Poetry is the tool sqlfluff-templater-dbt = "2.0.3" which we use for pre-commit together with sqlfluff. But can't see how that is correlated with the parsing of the project.
Heads up that the elementary package uses the invocation_args_dict variable: 

https://github.com/elementary-data/dbt-data-reliability/blob/0.7.5/macros/edr/dbt_artifacts/upload_dbt_invocation.sql#L55-L90
Okay, commented out all packages besides dbt-utils and removed dbt_packages/ folder and rerun "poetry run dbt deps".

Getting the same output still when running dbt compile/run/build -s model_name
> Heads up that the elementary package uses the invocation_args_dict variable:

That felt like a promising lead! However:
- I tried installing the package into an empty project, and I was able to run everything without a problem.
- It sounds like @christopherekfeldt removed the package, and is still seeing the same error

On looking again at the stacktrace, I'm now seeing that the error seems to crop up during _hook_ parsing:
```
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 539, in parse_project
    hook_parser.parse_file(file_block)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/hooks.py", line 120, in parse_file
    self.parse_node(hook)
```

@christopherekfeldt Do you have any `on-run-start` or `on-run-end` hooks defined in `dbt_project.yml`? Do any of them make use of [the `flags` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/flags)?
These are our hooks atm:

on-run-start:
  - '{{ exceptions.raise_compiler_error("Cannot use .profiles as profiles dir locally.") if "/dbt/.profiles" in flags.PROFILES_DIR }}'
  - '{{ exceptions.raise_compiler_error("Cannot use defer in target %s, run \"export DBT_DEFER=false\"." % target.name) if flags.DEFER_MODE and target.name in ["tcm", "ver", "prd"] }}'
  - '{{ exceptions.raise_compiler_error("Cannot use defer to env %s in target %s, run \"export DBT_DEFER=false\"." % target.name) if flags.DEFER_MODE and target.name in ["prd_ci"] and env_var("DBT_DEFER_ENV") != "prd" }}'
  - '{{ exceptions.raise_compiler_error("DBT_WRAPPER not set to true. Must use bin/dbt wrapper locally!") if env_var("DBT_WRAPPER", "false") != "true" and target.name in ["lab", "dev"] }}'
  - '{{ exceptions.raise_compiler_error("Models with invalid intraday tags detected. Aborting run!") if not check_intraday_tags() }}'
  - '{{ mock_sources() if var("mock_sources") }}'

on-run-end:
  - '{% do adapter.drop_schema(api.Relation.create(database=target.database, schema=target.schema)) if target.name.endswith("_ci") %}'
So all 3 first hooks could be the bad guy here? should the logic be switched out for something else?
Looking at the first hook you shared:

```yml
- '{{ exceptions.raise_compiler_error("Cannot use .profiles as profiles dir locally.") if "/dbt/.profiles" in flags.PROFILES_DIR }}'
```

That `<str> in flags.PROFILES_DIR` will do it! Since the value of `flags.PROFILES_DIR` is actually a `pathlib.Path`, not a string.

You can fix this immediately by explicitly converting `flags.PROFILES_DIR` to a string, with the Jinja `| string` filter:
```yml
- '{{ exceptions.raise_compiler_error("Cannot use .profiles as profiles dir locally.") if "/dbt/.profiles" in flags.PROFILES_DIR | string }}'
```

We could also handle this within `dbt-core`, and restore previous behavior, by converting all path flags to strings before exposing them to the Jinja context:
https://github.com/dbt-labs/dbt-core/blob/35f8ceb7f1dc575f1fc2e9e573e526f64cf25c9f/core/dbt/flags.py#L93-L104

```python
from pathlib import Path

def get_flag_obj():
    new_flags = Namespace()
    for key, val in get_flag_dict().items():
        if isinstance(val, Path):
            val = str(val)
        setattr(new_flags, key.upper(), val)
```

That change makes sense to me more generally, anyway.

Resolving patch:
diff --git a/core/dbt/flags.py b/core/dbt/flags.py
--- a/core/dbt/flags.py
+++ b/core/dbt/flags.py
@@ -3,6 +3,7 @@
 from argparse import Namespace
 from multiprocessing import get_context
 from typing import Optional
+from pathlib import Path
 
 
 # for setting up logger for legacy logger
@@ -95,6 +96,8 @@ def get_flag_dict():
 def get_flag_obj():
     new_flags = Namespace()
     for key, val in get_flag_dict().items():
+        if isinstance(val, Path):
+            val = str(val)
         setattr(new_flags, key.upper(), val)
     # The following 3 are CLI arguments only so they're not full-fledged flags,
     # but we put in flags for users.


Test patch: