Repository: dbt-labs/dbt-core

Issue:
[CT-343] partial parse conflicts with generate_schema_name changes
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

We have a project with a custom generate_schema_name macro.
If I change the macro generate_schema_name, project compilation fails with error `a string or stream input is required`.
If I run a `dbt clean` before compiling, compilation completes without errors.


### Expected Behavior

Compilation should not depend on `dbt clean` 

### Steps To Reproduce

dbt clean
dbt deps
dbt compile --> no errors

modify generate_schema_name macro (even adding a jinja comment):

dbt compile --> error

10:01:21  Running with dbt=1.0.3
10:01:22  Change detected to override macro used during parsing. Starting full parse.
10:01:34  Encountered an error:
a string or stream input is required



### Relevant log output

_No response_

### Environment

```markdown
- OS: Windows 10
- Python: 3.7.9
- dbt-core: 1.0.3
- dbt-bigquery: 1.0.0
```


### What database are you using dbt with?

bigquery

### Additional Context

_No response_


Discussion:
I have not been able to recreate this problem (though I don't have access to a Windows machine to test). Is there anything in the logs? Can you share your generate_schema_name macro? Could you try removing all files except the partial_parse.msgpack file from the target directory? Since the msgpack file has been loaded (or the error would be earlier), I'm wondering if it's one of the other files in the target directory that's the issue.
I tried reproducing the issue on a fresh project and the error didn't arise.
It must be due to some interference with models / macro of the project.
I'll try to do some testing in order to isolate what's the root cause.
Thank you for opening this issue.  

This also happened to me.  I was running DBT inside of a docker container: 

`FROM python:3.7-slim-buster`

running poetry with this `pyproject.toml` configuration:
```
[tool.poetry.dependencies]
python = "^3.7"
dbt-core = "^1.0.0"
dbt-bigquery = "^1.0.0"
dbt-coverage = "^0.1.8"
mkdocs = "^1.3.0"
mkdocs-material = "^8.2.11"

[tool.poetry.dev-dependencies]
ipython = "^7.31.1"
ipdb = "^0.13.7"
sqlfluff = "0.5.2"
black = "^21.4b2"
pylint = "^2.12.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
```

The CI/CD pipeline broke because of this issue and I when I added the dbt clean and dbt deps commands to my workflow, it worked.

Hi, I got the same error when creating my own custom "source" macro. The macro itself just calls the builtins.source and does nothing else. It works fine when first do a "dbt clean" and a "dbt deps" like described above. But when I change something in this file  (might just be a comment) I get the output below and first need to do a clean and dpes again to get it working.


```
PS C:\GIT\project\dbt> dbt compile
14:01:01  Running with dbt=1.1.0
14:01:01  Change detected to override macro used during parsing. Starting full parse.
14:01:02  No external sources selected
14:01:03  Encountered an error:
a string or stream input is required
```


I have the dbt_utils, spark_utils and dbt_external_tables packages installed. I use the dbt-databricks adapter and I get the error while running in Visual Studio Code on a WIN11 machine.

Hope this helps with finding the issue.
I hit this running DBT in a docker container whenever I changed the `\macros\get_custom_schema.sql` macro. Resolved the issue by running `dbt clean` after each change.
Getting the same error in a fresh project that uses macros for generating schema and aliases names.  `dbt clean` solves the problem
I have this issue now, updated the generate_schema_name macro, with a minor change. then it now fails with the above mentioned error msgs. But I am not able to run a dbt deps (for Cloud) as suggested here as the UI instantly fails and gives me this message: 
![image](https://user-images.githubusercontent.com/79144766/184876332-ce433758-cae7-4854-b209-2ffa37727bda.png)

I've got a repro of this one (ensure partial parsing hasn't been accidentally disabled).

```yaml
# dbt_project.yml

name: "my_dbt_project"
version: "1.0.0"
config-version: 2
profile: "snowflake"

models:
  my_dbt_project:
    +materialized: table
    +database: development
```

```sql
-- models/foo.sql
select 1 as user_id
```

```jinja
-- macros/generate_database_name.sql
{% macro generate_database_name(custom_database_name, node) -%}
    {%- set default_database = target.database -%}
    {%- if target.name == 'dev' -%}
        {{ default_database }}
    {%- elif target.name == 'prod' -%}
        {{ custom_database_name | trim }}
    {%- else -%}
        {{ default_database }}
    {%- endif -%}
{%- endmacro %}
```

Key to this is you want to add a file `models/schema.yml` that is completely empty (has no content in it - not even yaml comments or anything):

```yaml

```

1. Clean out old `partial_parse.msgpack` files and then compile:

```
$ dbt clean && dbt compile

01:33:53  Running with dbt=1.2.1
01:33:53  Checking target/*
01:33:53  Cleaned target/*
01:33:53  Finished cleaning all paths.
01:33:58  Running with dbt=1.2.1
01:33:58  Partial parse save file not found. Starting full parse.
01:33:59  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 268 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
01:33:59  
01:34:02  Concurrency: 1 threads (target='dev')
01:34:02  
01:34:02  Done.
```

2. Modify the `generate_database_name` macro slightly:

```jinja
-- macros/generate_database_name.sql
{% macro generate_database_name(custom_database_name, node) -%}
    {%- set default_database = target.database -%}
    {%- if target.name == 'dev' -%}
        {{ default_database }}
    {%- elif target.name == 'ci' -%}
        {{ custom_database_name | trim }}
    {%- else -%}
        {{ default_database }}
    {%- endif -%}
{%- endmacro %}
```

3. Recompile but do not clean out the `partial_parse.msgpack` file like you'd do in step (1):

```
$ dbt compile

01:38:10  Running with dbt=1.2.1
01:38:10  Change detected to override macro used during parsing. Starting full parse.
01:38:11  Encountered an error:
a string or stream input is required
01:38:11  Traceback (most recent call last):
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/main.py", line 129, in main
    results, succeeded = handle_and_check(args)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/main.py", line 191, in handle_and_check
    task, res = run_from_args(parsed)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/main.py", line 238, in run_from_args
    results = task.run()
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/task/runnable.py", line 451, in run
    self._runtime_initialize()
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/task/runnable.py", line 159, in _runtime_initialize
    super()._runtime_initialize()
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/task/runnable.py", line 92, in _runtime_initialize
    self.load_manifest()
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/task/runnable.py", line 81, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 219, in get_full_manifest
    manifest = loader.load()
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 366, in load
    self.parse_project(
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 469, in parse_project
    parser.parse_file(block, dct=dct)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/parser/schemas.py", line 488, in parse_file
    dct = yaml_from_file(block.file)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/parser/schemas.py", line 118, in yaml_from_file
    return load_yaml_text(source_file.contents, source_file.path)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/clients/yaml_helper.py", line 56, in load_yaml_text
    return safe_load(contents)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/dbt/clients/yaml_helper.py", line 51, in safe_load
    return yaml.load(contents, Loader=SafeLoader)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/yaml/__init__.py", line 79, in load
    loader = Loader(stream)
  File "/Users/jeremy/src/dbt-sandcastles/venv_dbt_latest/lib/python3.8/site-packages/yaml/cyaml.py", line 26, in __init__
    CParser.__init__(self, stream)
  File "yaml/_yaml.pyx", line 288, in yaml._yaml.CParser.__init__
TypeError: a string or stream input is required
```

Good ol yaml trying to parse an empty on the second go around - but the first time is okay though (full parse) because we do not try and read `partial_parse.msgpack` on the first go round.

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

Resolving patch:
diff --git a/core/dbt/context/base.py b/core/dbt/context/base.py
--- a/core/dbt/context/base.py
+++ b/core/dbt/context/base.py
@@ -634,9 +634,8 @@ def flags(self) -> Any:
             {% endif %}
 
         This supports all flags defined in flags submodule (core/dbt/flags.py)
-        TODO: Replace with object that provides read-only access to flag values
         """
-        return flags
+        return flags.get_flag_obj()
 
     @contextmember
     @staticmethod
diff --git a/core/dbt/flags.py b/core/dbt/flags.py
--- a/core/dbt/flags.py
+++ b/core/dbt/flags.py
@@ -1,7 +1,9 @@
-import os
+# Do not import the os package because we expose this package in jinja
+from os import name as os_name, path as os_path, getcwd as os_getcwd, getenv as os_getenv
 import multiprocessing
+from argparse import Namespace
 
-if os.name != "nt":
+if os_name != "nt":
     # https://bugs.python.org/issue41567
     import multiprocessing.popen_spawn_posix  # type: ignore
 from pathlib import Path
@@ -10,14 +12,14 @@
 # PROFILES_DIR must be set before the other flags
 # It also gets set in main.py and in set_from_args because the rpc server
 # doesn't go through exactly the same main arg processing.
-GLOBAL_PROFILES_DIR = os.path.join(os.path.expanduser("~"), ".dbt")
-LOCAL_PROFILES_DIR = os.getcwd()
+GLOBAL_PROFILES_DIR = os_path.join(os_path.expanduser("~"), ".dbt")
+LOCAL_PROFILES_DIR = os_getcwd()
 # Use the current working directory if there is a profiles.yml file present there
-if os.path.exists(Path(LOCAL_PROFILES_DIR) / Path("profiles.yml")):
+if os_path.exists(Path(LOCAL_PROFILES_DIR) / Path("profiles.yml")):
     DEFAULT_PROFILES_DIR = LOCAL_PROFILES_DIR
 else:
     DEFAULT_PROFILES_DIR = GLOBAL_PROFILES_DIR
-PROFILES_DIR = os.path.expanduser(os.getenv("DBT_PROFILES_DIR", DEFAULT_PROFILES_DIR))
+PROFILES_DIR = os_path.expanduser(os_getenv("DBT_PROFILES_DIR", DEFAULT_PROFILES_DIR))
 
 STRICT_MODE = False  # Only here for backwards compatibility
 FULL_REFRESH = False  # subcommand
@@ -88,7 +90,7 @@ def env_set_truthy(key: str) -> Optional[str]:
     """Return the value if it was set to a "truthy" string value or None
     otherwise.
     """
-    value = os.getenv(key)
+    value = os_getenv(key)
     if not value or value.lower() in ("0", "false", "f"):
         return None
     return value
@@ -101,7 +103,7 @@ def env_set_bool(env_value):
 
 
 def env_set_path(key: str) -> Optional[Path]:
-    value = os.getenv(key)
+    value = os_getenv(key)
     if value is None:
         return value
     else:
@@ -181,7 +183,7 @@ def get_flag_value(flag, args, user_config):
     if flag == "PRINTER_WIDTH":  # must be ints
         flag_value = int(flag_value)
     if flag == "PROFILES_DIR":
-        flag_value = os.path.abspath(flag_value)
+        flag_value = os_path.abspath(flag_value)
 
     return flag_value
 
@@ -205,7 +207,7 @@ def _load_flag_value(flag, args, user_config):
 def _get_flag_value_from_env(flag):
     # Environment variables use pattern 'DBT_{flag name}'
     env_flag = _get_env_flag(flag)
-    env_value = os.getenv(env_flag)
+    env_value = os_getenv(env_flag)
     if env_value is None or env_value == "":
         return None
 
@@ -241,4 +243,21 @@ def get_flag_dict():
         "log_cache_events": LOG_CACHE_EVENTS,
         "quiet": QUIET,
         "no_print": NO_PRINT,
+        "cache_selected_only": CACHE_SELECTED_ONLY,
+        "target_path": TARGET_PATH,
+        "log_path": LOG_PATH,
     }
+
+
+# This is used by core/dbt/context/base.py to return a flag object
+# in Jinja.
+def get_flag_obj():
+    new_flags = Namespace()
+    for k, v in get_flag_dict().items():
+        setattr(new_flags, k.upper(), v)
+    # The following 3 are CLI arguments only so they're not full-fledged flags,
+    # but we put in flags for users.
+    setattr(new_flags, "FULL_REFRESH", FULL_REFRESH)
+    setattr(new_flags, "STORE_FAILURES", STORE_FAILURES)
+    setattr(new_flags, "WHICH", WHICH)
+    return new_flags
diff --git a/core/dbt/parser/schemas.py b/core/dbt/parser/schemas.py
--- a/core/dbt/parser/schemas.py
+++ b/core/dbt/parser/schemas.py
@@ -99,11 +99,11 @@
 
 def yaml_from_file(source_file: SchemaSourceFile) -> Dict[str, Any]:
     """If loading the yaml fails, raise an exception."""
-    path = source_file.path.relative_path
     try:
-        return load_yaml_text(source_file.contents, source_file.path)
+        # source_file.contents can sometimes be None
+        return load_yaml_text(source_file.contents or "", source_file.path)
     except ValidationException as e:
-        raise YamlLoadFailure(source_file.project_name, path, e)
+        raise YamlLoadFailure(source_file.project_name, source_file.path.relative_path, e)
 
 
 class ParserRef:


Test patch:
diff --git a/test/integration/068_partial_parsing_tests/test_partial_parsing.py b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
--- a/test/integration/068_partial_parsing_tests/test_partial_parsing.py
+++ b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
@@ -477,6 +477,8 @@ def test_postgres_skip_macros(self):
         # initial run so we have a msgpack file
         self.setup_directories()
         self.copy_file('test-files/model_one.sql', 'models/model_one.sql')
+        # use empty_schema file for bug #4850
+        self.copy_file('test-files/empty_schema.yml', 'models/eschema.yml')
         results = self.run_dbt()
 
         # add a new ref override macro
diff --git a/tests/functional/context_methods/test_builtin_functions.py b/tests/functional/context_methods/test_builtin_functions.py
--- a/tests/functional/context_methods/test_builtin_functions.py
+++ b/tests/functional/context_methods/test_builtin_functions.py
@@ -112,8 +112,9 @@ def test_builtin_invocation_args_dict_function(self, project):
         expected = "invocation_result: {'debug': True, 'log_format': 'json', 'write_json': True, 'use_colors': True, 'printer_width': 80, 'version_check': True, 'partial_parse': True, 'static_parser': True, 'profiles_dir': "
         assert expected in str(result)
 
-        expected = "'send_anonymous_usage_stats': False, 'quiet': False, 'no_print': False, 'macro': 'validate_invocation', 'args': '{my_variable: test_variable}', 'which': 'run-operation', 'rpc_method': 'run-operation', 'indirect_selection': 'eager'}"
-        assert expected in str(result)
+        expected = ("'send_anonymous_usage_stats': False", "'quiet': False", "'no_print': False", "'cache_selected_only': False", "'macro': 'validate_invocation'", "'args': '{my_variable: test_variable}'", "'which': 'run-operation'", "'rpc_method': 'run-operation'", "'indirect_selection': 'eager'")
+        for element in expected:
+            assert element in str(result)
 
     def test_builtin_dbt_metadata_envs_function(self, project, monkeypatch):
         envs = {