Repository: dbt-labs/dbt-core

Issue:
[CT-3383][Bug] Metadata freshness checks that include a filter cause dbt to hang
### Is this a new bug in dbt-snowflake?

- [X] I believe this is a new bug in dbt-snowflake
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

I found this when I was testing the new information_schema/metadata driven freshness feature released in v1.7.

I removed the loaded_at field from my freshness test configs and ran them, not realizing I still had a `filter` on my freshness check definition. The output in the terminal showed the freshness checks being executed, but they never completed. Viewing my warehouse query history, I confirmed that dbt didn't actually issue any freshness queries at all. dbt output just sat there for several minutes before I manually cancelled it.

Removing the `filter` from the freshness check definition resolved the issue.

### Expected Behavior

Metadata driven freshness checks should either support `filter` if theres a use case for that behavior, or `filter` should be ignored and there should be a warning message indicating that `filter` was ignored.

Either way the current behavior where `filter` causes metadata driven freshness checks to just hang does seem to be a bug.

### Steps To Reproduce

This should cause the freshness check to "run" but no query against the warehouse will be issued:
```
version: 2
sources:
  - name: src
    database: db
    schema: test

    tables:
      - name: test_table
        freshness:
          warn_after:
            count: 4
            period: hour
          error_after:
            count: 8
            period: hour
          filter: 1=1
```

This should resolve the error (filter config is removed--freshness check query is actually issued and runs to completion)
```
version: 2
sources:
  - name: src
    database: db
    schema: test

    tables:
      - name: test_table
        freshness:
          warn_after:
            count: 4
            period: hour
          error_after:
            count: 8
            period: hour
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: macOS-13.5.2-x86_64-i386-64bit
- Python: 3.11.6
- dbt-core: 1.7.1
- dbt-snowflake: 1.7.0
```


### Additional Context

_No response_


Discussion:
Thanks for reporting this @sp-tkerlavage 🙏

I was able to reproduce this in dbt-snowflake and something similar in dbt-postgres as well, so I'm going to transfer this issue to dbt-core.

<details>
<summary>

### Reprex (for Snowflake)
</summary>

`seeds/my_seed.csv`

```csv
id
1
```

Build the seed (to use it as a source):
```shell
dbt seed
```

`models/_sources.yml`
```yaml
version: 2

sources:
  - name: src
    database: "{{ target.database }}"
    schema: "{{ target.schema }}"
    tables:
      - name: my_seed
        freshness:
          warn_after:
            count: 4
            period: hour
          error_after:
            count: 8
            period: hour
          filter: 1=1
```

Let this run for 30 seconds (or whatever) before using ctrl-c to cancel it:
```shell
dbt source freshness
```

Stack trace:
```
05:07:05  1 of 1 START freshness of src.my_seed .......................................... [RUN]
^C05:10:58  CANCEL query source.my_project.src.my_seed ..................................... [CANCEL]
05:10:59  
05:10:59  Exited because of keyboard interrupt
05:10:59  
05:10:59  Done. PASS=0 WARN=0 ERROR=0 SKIP=0 TOTAL=0
05:10:59  Encountered an error:

05:10:59  Traceback (most recent call last):
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/cli/requires.py", line 90, in wrapper
    result, success = func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/cli/requires.py", line 75, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/cli/requires.py", line 197, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/cli/requires.py", line 244, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/cli/requires.py", line 284, in wrapper
    return func(*args, **kwargs)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/cli/main.py", line 846, in freshness
    results = task.run()
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/task/runnable.py", line 474, in run
    result = self.execute_with_hooks(selected_uids)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/task/runnable.py", line 435, in execute_with_hooks
    res = self.execute_nodes()
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/task/runnable.py", line 359, in execute_nodes
    self.run_queue(pool)
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/task/runnable.py", line 291, in run_queue
    self.job_queue.join()
  File "/Users/dbeatty/projects/environments/snowflake_1.7/lib/python3.10/site-packages/dbt/graph/queue.py", line 198, in join
    self.inner.join()
  File "/Users/dbeatty/.pyenv/versions/3.10.10/lib/python3.10/queue.py", line 90, in join
    self.all_tasks_done.wait()
  File "/Users/dbeatty/.pyenv/versions/3.10.10/lib/python3.10/threading.py", line 320, in wait
    waiter.acquire()
KeyboardInterrupt
```
</details>

<details>
<summary>

### Reprex (for Postgres)
</summary>

Everything the same as the reprex for Snowflake above, but doesn't even need the `filter: 1=1` part:

```yaml
version: 2

sources:
  - name: src
    database: "{{ target.database }}"
    schema: "{{ target.schema }}"
    tables:
      - name: my_seed
        freshness:
          warn_after:
            count: 4
            period: hour
          error_after:
            count: 8
            period: hour
```

It should give similar (or same) stacktrace as in Snowflake.
</details>

Notes from estimation:
- if there's an unhandled exception in _one_ of the threads, it could cause this to hang
There are a few things happening here that cause the hang.

First, we're logging Note events that have positional arguments that [we explicitly do not allow.](https://github.com/dbt-labs/dbt-core/blob/1c9cec17878ca0fec537a4cc13812f91ee97b29b/core/dbt/events/base_types.py#L64-L66).  This assert should really raise a `DbtInternalError` instead of being an assert.  Is there a reason we're doing this check other than to discourage putting vars into the message instead of passing them as data?  If there's not, we should exclude the `Note` event from this check to allow the check to pass through.  

Once that's fixed, another error will surface with the `loaded_at_field`.  Right now for a warehouse that does not support metadata checks, if the loaded_at field is blank, we have another uncaught exception [here](https://github.com/dbt-labs/dbt-core/blob/1c9cec17878ca0fec537a4cc13812f91ee97b29b/core/dbt/task/freshness.py#L153) because the `freshness` kwarg is empty.

This behavior needs to be tweaked where if you try to configure freshness for a node and you don’t supply the loaded_at_field but also the warehouse doesn’t support this feature, we should show the freshness as skipped and log the warning message.

This involves adding `NodeStatus.Skipped` to the `FreshnessStatus` and plumbing it through.  We will also need to handle returning the SourceFreshnessResult more appropriately [here](https://github.com/dbt-labs/dbt-core/blob/1c9cec17878ca0fec537a4cc13812f91ee97b29b/core/dbt/task/freshness.py#L134-L137).

Resolving patch:
diff --git a/core/dbt/task/base.py b/core/dbt/task/base.py
--- a/core/dbt/task/base.py
+++ b/core/dbt/task/base.py
@@ -207,6 +207,9 @@ def __init__(self, config, adapter, node, node_index, num_nodes) -> None:
     def compile(self, manifest: Manifest) -> Any:
         pass
 
+    def _node_build_path(self) -> Optional[str]:
+        return self.node.build_path if hasattr(self.node, "build_path") else None
+
     def get_result_status(self, result) -> Dict[str, str]:
         if result.status == NodeStatus.Error:
             return {"node_status": "error", "node_error": str(result.message)}
@@ -339,7 +342,7 @@ def _handle_catchable_exception(self, e, ctx):
     def _handle_internal_exception(self, e, ctx):
         fire_event(
             InternalErrorOnRun(
-                build_path=self.node.build_path, exc=str(e), node_info=get_node_info()
+                build_path=self._node_build_path(), exc=str(e), node_info=get_node_info()
             )
         )
         return str(e)
@@ -347,7 +350,7 @@ def _handle_internal_exception(self, e, ctx):
     def _handle_generic_exception(self, e, ctx):
         fire_event(
             GenericExceptionOnRun(
-                build_path=self.node.build_path,
+                build_path=self._node_build_path(),
                 unique_id=self.node.unique_id,
                 exc=str(e),
                 node_info=get_node_info(),
diff --git a/core/dbt/task/freshness.py b/core/dbt/task/freshness.py
--- a/core/dbt/task/freshness.py
+++ b/core/dbt/task/freshness.py
@@ -120,9 +120,9 @@ def execute(self, compiled_node, manifest):
                 if compiled_node.freshness.filter is not None:
                     fire_event(
                         Note(
-                            f"A filter cannot be applied to a metadata freshness check on source '{compiled_node.name}'.",
-                            EventLevel.WARN,
-                        )
+                            msg=f"A filter cannot be applied to a metadata freshness check on source '{compiled_node.name}'."
+                        ),
+                        EventLevel.WARN,
                     )
 
                 adapter_response, freshness = self.adapter.calculate_freshness_from_metadata(
@@ -132,9 +132,8 @@ def execute(self, compiled_node, manifest):
 
                 status = compiled_node.freshness.status(freshness["age"])
             else:
-                status = FreshnessStatus.Warn
-                fire_event(
-                    Note(f"Skipping freshness for source {compiled_node.name}."),
+                raise DbtRuntimeError(
+                    f"Could not compute freshness for source {compiled_node.name}: no 'loaded_at_field' provided and {self.adapter.type()} adapter does not support metadata-based freshness checks."
                 )
 
         # adapter_response was not returned in previous versions, so this will be None


Test patch:
diff --git a/tests/functional/sources/test_source_freshness.py b/tests/functional/sources/test_source_freshness.py
--- a/tests/functional/sources/test_source_freshness.py
+++ b/tests/functional/sources/test_source_freshness.py
@@ -5,6 +5,8 @@
 import yaml
 
 import dbt.version
+from dbt.artifacts.schemas.freshness import FreshnessResult
+from dbt.artifacts.schemas.results import FreshnessStatus
 from dbt.cli.main import dbtRunner
 from tests.functional.sources.common_source_setup import BaseSourcesTest
 from tests.functional.sources.fixtures import (
@@ -385,7 +387,7 @@ class TestMetadataFreshnessFails:
     def models(self):
         return {"schema.yml": freshness_via_metadata_schema_yml}
 
-    def test_metadata_freshness_fails(self, project):
+    def test_metadata_freshness_unsupported_parse_warning(self, project):
         """Since the default test adapter (postgres) does not support metadata
         based source freshness checks, trying to use that mechanism should
         result in a parse-time warning."""
@@ -401,6 +403,16 @@ def warning_probe(e):
 
         assert got_warning
 
+    def test_metadata_freshness_unsupported_error_when_run(self, project):
+
+        runner = dbtRunner()
+        result = runner.invoke(["source", "freshness"])
+        assert isinstance(result.result, FreshnessResult)
+        assert len(result.result.results) == 1
+        freshness_result = result.result.results[0]
+        assert freshness_result.status == FreshnessStatus.RuntimeErr
+        assert "Could not compute freshness for source test_table" in freshness_result.message
+
 
 class TestHooksInSourceFreshness(SuccessfulSourceFreshnessTest):
     @pytest.fixture(scope="class")
diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/conftest.py
@@ -0,0 +1,29 @@
+import pytest
+
+from dbt.artifacts.resources import Quoting, SourceConfig
+from dbt.artifacts.resources.types import NodeType
+from dbt.contracts.graph.nodes import SourceDefinition
+
+
+@pytest.fixture
+def basic_parsed_source_definition_object():
+    return SourceDefinition(
+        columns={},
+        database="some_db",
+        description="",
+        fqn=["test", "source", "my_source", "my_source_table"],
+        identifier="my_source_table",
+        loader="stitch",
+        name="my_source_table",
+        original_file_path="/root/models/sources.yml",
+        package_name="test",
+        path="/root/models/sources.yml",
+        quoting=Quoting(),
+        resource_type=NodeType.Source,
+        schema="some_schema",
+        source_description="my source description",
+        source_name="my_source",
+        unique_id="test.source.my_source.my_source_table",
+        tags=[],
+        config=SourceConfig(),
+    )
diff --git a/tests/unit/task/test_base.py b/tests/unit/task/test_base.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/task/test_base.py
@@ -0,0 +1,24 @@
+from dbt.task.base import BaseRunner
+from dbt.contracts.graph.nodes import SourceDefinition
+
+
+class MockRunner(BaseRunner):
+    def compile(self):
+        pass
+
+
+class TestBaseRunner:
+    def test_handle_generic_exception_handles_nodes_without_build_path(
+        self, basic_parsed_source_definition_object: SourceDefinition
+    ):
+        # Source definition nodes don't have `build_path` attributes. Thus, this
+        # test will fail if _handle_generic_exception doesn't account for this
+        runner = MockRunner(
+            config=None,
+            adapter=None,
+            node=basic_parsed_source_definition_object,
+            node_index=None,
+            num_nodes=None,
+        )
+        assert not hasattr(basic_parsed_source_definition_object, "build_path")
+        runner._handle_generic_exception(Exception("bad thing happened"), ctx=None)
diff --git a/tests/unit/test_contracts_graph_parsed.py b/tests/unit/test_contracts_graph_parsed.py
--- a/tests/unit/test_contracts_graph_parsed.py
+++ b/tests/unit/test_contracts_graph_parsed.py
@@ -1954,30 +1954,6 @@ def basic_parsed_source_definition_dict():
     }
 
 
-@pytest.fixture
-def basic_parsed_source_definition_object():
-    return SourceDefinition(
-        columns={},
-        database="some_db",
-        description="",
-        fqn=["test", "source", "my_source", "my_source_table"],
-        identifier="my_source_table",
-        loader="stitch",
-        name="my_source_table",
-        original_file_path="/root/models/sources.yml",
-        package_name="test",
-        path="/root/models/sources.yml",
-        quoting=Quoting(),
-        resource_type=NodeType.Source,
-        schema="some_schema",
-        source_description="my source description",
-        source_name="my_source",
-        unique_id="test.source.my_source.my_source_table",
-        tags=[],
-        config=SourceConfig(),
-    )
-
-
 @pytest.fixture
 def complex_parsed_source_definition_dict():
     return {