Repository: dbt-labs/dbt-core

Issue:
[CT-2391] [Bug] Following the Python API example.py example raises AttributeError
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

 I'm trying to work with the python APIs in dbt version 1.5, per the given example.py and hitting some error

`AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING'`


### Expected Behavior

Not throw AttributeError, profile getting created properly

### Steps To Reproduce

1. run these lines (project_dir is a path to a dbt project)
```python
from dbt.cli.main import dbtRunner
from dbt.config.runtime import load_profile, load_project
project_dir = "/Users/roitabach/workspace/small_fix/dbt-data-reliability/integration_tests/"
profile = load_profile(project_dir, {}, target_override="snowflake")
```


### Relevant log output

```shell
This exception is raised:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[16], line 1
----> 1 profile = load_profile(project_dir, cli_vars={"MACRO_DEBUGGING":False},target_override="snowflake")

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/runtime.py:68, in load_profile(project_root, cli_vars, profile_name_override, target_override, threads_override)
     66 raw_profile_name = raw_project.get("profile")
     67 profile_renderer = ProfileRenderer(cli_vars)
---> 68 profile_name = profile_renderer.render_value(raw_profile_name)
     69 profile = Profile.render(
     70     profile_renderer, profile_name, profile_name_override, target_override, threads_override
     71 )
     72 # Save env_vars encountered in rendering for partial parsing

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/renderer.py:185, in SecretRenderer.render_value(self, value, keypath)
    181 def render_value(self, value: Any, keypath: Optional[Keypath] = None) -> Any:
    182     # First, standard Jinja rendering, with special handling for 'secret' environment variables
    183     # "{{ env_var('DBT_SECRET_ENV_VAR') }}" -> "$$$DBT_SECRET_START$$$DBT_SECRET_ENV_{VARIABLE_NAME}$$$DBT_SECRET_END$$$"
    184     # This prevents Jinja manipulation of secrets via macros/filters that might leak partial/modified values in logs
--> 185     rendered = super().render_value(value, keypath)
    186     # Now, detect instances of the placeholder value ($$$DBT_SECRET_START...DBT_SECRET_END$$$)
    187     # and replace them with the actual secret value
    188     if SECRET_ENV_PREFIX in str(rendered):

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/config/renderer.py:42, in BaseRenderer.render_value(self, value, keypath)
     40 try:
     41     with catch_jinja():
---> 42         return get_rendered(value, self.context, native=True)
     43 except CompilationError as exc:
     44     msg = f"Could not render {value}: {exc.msg}"

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:584, in get_rendered(string, ctx, node, capture_macros, native)
    582 if not native and isinstance(string, str) and _HAS_RENDER_CHARS_PAT.search(string) is None:
    583     return string
--> 584 template = get_template(
    585     string,
    586     ctx,
    587     node,
    588     capture_macros=capture_macros,
    589     native=native,
    590 )
    591 return render_template(template, ctx, node)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:541, in get_template(string, ctx, node, capture_macros, native)
    538 env = get_environment(node, capture_macros, native=native)
    540 template_source = str(string)
--> 541 return env.from_string(template_source, globals=ctx)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/Jinja2-3.1.2-py3.10.egg/jinja2/environment.py:1105, in Environment.from_string(self, source, globals, template_class)
   1103 gs = self.make_globals(globals)
   1104 cls = template_class or self.template_class
-> 1105 return cls.from_code(self, self.compile(source), gs, None)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/Jinja2-3.1.2-py3.10.egg/jinja2/environment.py:766, in Environment.compile(self, source, name, filename, raw, defer_init)
    764     if filename is None:
    765         filename = "<template>"
--> 766     return self._compile(source, filename)
    767 except TemplateSyntaxError:
    768     self.handle_exception(source=source_hint)

File ~/workspace/env/elementary/venv/lib/python3.10/site-packages/dbt/clients/jinja.py:102, in MacroFuzzEnvironment._compile(self, source, filename)
     94 def _compile(self, source, filename):
     95     """Override jinja's compilation to stash the rendered source inside
     96     the python linecache for debugging when the appropriate environment
     97     variable is set.
   (...)
    100     WARNING: This can write a ton of data if you aren't careful.
    101     """
--> 102     macro_debugging = get_flags().MACRO_DEBUGGING
    103     if filename == "<template>" and macro_debugging:
    104         write = macro_debugging == "write"

AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING'
```

### Environment

```markdown
- OS: MacOS Monterey 12.5.1 (21G83)
- Python: 3.11.2
- dbt-core: 1.5.0-b5
```

### Which database adapter are you using with dbt?

snowflake

### Additional Context

I tried to comment out the `if` block just to see where I get- got another argparse error- 
```
AttributeError: 'Namespace' object has no attribute 'PROFILES_DIR'
```
from this row in `.../site-packages/dbt/config/profile.py`
```
--> 434 raw_profiles = read_profile(flags.PROFILES_DIR)
```


Discussion:
Thanks for reporting this @RoiTabach !

I got the same error with essentially the same script as you. Only difference is that it is aimed at the current working directory and using a differently named target:
```python
import os
from dbt.cli.main import dbtRunner
from dbt.config.runtime import load_profile, load_project

project_dir = os.getcwd()
profile = load_profile(project_dir, {}, target_override="postgres")
```

In addition to this bug, we're planning on doing the following:
- posting examples on docs.getdbt.com [docs.getdbt-com #3118](https://github.com/dbt-labs/docs.getdbt.com/pull/3118)
- removing [this](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/cli/example.py) example script entirely
We're not ready to contract `load_profile` and `load_project` for external use. Let's be sure to leave them out of any docs / examples for now.

Resolving patch:
diff --git a/core/dbt/cli/example.py b/core/dbt/cli/example.py
deleted file mode 100644
--- a/core/dbt/cli/example.py
+++ /dev/null
@@ -1,20 +0,0 @@
-from dbt.cli.main import dbtRunner
-from dbt.config.runtime import load_profile, load_project
-
-if __name__ == "__main__":
-    project_dir = "/Users/chenyuli/git/jaffle_shop"
-    cli_args = ["run", "--project-dir", project_dir]
-
-    # initialize the dbt runner
-    dbt = dbtRunner()
-    # run the command
-    res = dbt.invoke(cli_args)
-
-    # preload profile and project
-    profile = load_profile(project_dir, {}, "testing-postgres")
-    project = load_project(project_dir, False, profile, {})
-
-    # initialize the runner with pre-loaded profile and project, you can also pass in a preloaded manifest
-    dbt = dbtRunner(profile=profile, project=project)
-    # run the command, this will use the pre-loaded profile and project instead of loading
-    res = dbt.invoke(cli_args)


Test patch: