Repository: dbt-labs/dbt-core

Issue:
[Bug] Unhandled exception for failing `on-run-*` hooks in `source freshness`
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

_follow up to https://github.com/dbt-labs/dbt-core/pull/9366_

Failing hooks in `source freshness` lead to an unhandled exception

### Expected Behavior

The exception is handled, as it is for other commands

### Steps To Reproduce

1. Create a simple project that defines a source with `freshness` + `loaded_at_field`
2. In the project file:
```yml
# dbt_project.yml
flags:
  source_freshness_run_project_hooks: true

on-run-start:
  "select fake_column from table_does_not_exist"
```
3. `dbt source freshness`

### Relevant log output

```shell
$ dbt source freshness
12:11:06  Running with dbt=1.8.0-a1
12:11:06  Registered adapter: postgres=1.8.0-a1
12:11:06  Unable to do partial parsing because a project config has changed
12:11:07  Found 1 operation, 1 source, 521 macros
12:11:07
12:11:08
12:11:08  Running 1 on-run-start hook
12:11:08  1 of 1 START hook: my_dbt_project.on-run-start.0 ............................... [RUN]
12:11:08  Database error while running on-run-start
12:11:08  Concurrency: 5 threads (target='dev')
12:11:08
12:11:08  1 of 1 START freshness of my_src.my_tbl ........................................ [RUN]
12:11:08  1 of 1 WARN freshness of my_src.my_tbl ......................................... [WARN in 0.04s]
12:11:08
12:11:08  Finished running 1 source in 0 hours 0 minutes and 0.32 seconds (0.32s).
12:11:08  Encountered an error:
'BaseResult' object has no attribute 'node'
12:11:08  Traceback (most recent call last):
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 97, in wrapper
    result, success = func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 175, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 204, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 251, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 280, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/main.py", line 789, in freshness
    results = task.run()
  File "/Users/jerco/dev/product/dbt-core/core/dbt/task/runnable.py", line 524, in run
    result.write(self.result_path())
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 67, in write
    FreshnessExecutionResultArtifact.from_result(self).write(path)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 110, in from_result
    processed = [process_freshness_result(r) for r in base.results]
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 110, in <listcomp>
    processed = [process_freshness_result(r) for r in base.results]
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 119, in process_freshness_result
    unique_id = result.node.unique_id
AttributeError: 'BaseResult' object has no attribute 'node'
```


### Environment

```markdown
- OS: 13.4.1
- Python: 3.10.11
- dbt: 1.8.0-a1
```


### Which database adapter are you using with dbt?

_No response_

### Additional Context

_No response_
[Bug] Unhandled exception for failing `on-run-*` hooks in `source freshness`
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

_follow up to https://github.com/dbt-labs/dbt-core/pull/9366_

Failing hooks in `source freshness` lead to an unhandled exception

### Expected Behavior

The exception is handled, as it is for other commands

### Steps To Reproduce

1. Create a simple project that defines a source with `freshness` + `loaded_at_field`
2. In the project file:
```yml
# dbt_project.yml
flags:
  source_freshness_run_project_hooks: true

on-run-start:
  "select fake_column from table_does_not_exist"
```
3. `dbt source freshness`

### Relevant log output

```shell
$ dbt source freshness
12:11:06  Running with dbt=1.8.0-a1
12:11:06  Registered adapter: postgres=1.8.0-a1
12:11:06  Unable to do partial parsing because a project config has changed
12:11:07  Found 1 operation, 1 source, 521 macros
12:11:07
12:11:08
12:11:08  Running 1 on-run-start hook
12:11:08  1 of 1 START hook: my_dbt_project.on-run-start.0 ............................... [RUN]
12:11:08  Database error while running on-run-start
12:11:08  Concurrency: 5 threads (target='dev')
12:11:08
12:11:08  1 of 1 START freshness of my_src.my_tbl ........................................ [RUN]
12:11:08  1 of 1 WARN freshness of my_src.my_tbl ......................................... [WARN in 0.04s]
12:11:08
12:11:08  Finished running 1 source in 0 hours 0 minutes and 0.32 seconds (0.32s).
12:11:08  Encountered an error:
'BaseResult' object has no attribute 'node'
12:11:08  Traceback (most recent call last):
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 97, in wrapper
    result, success = func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 175, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 204, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 251, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/requires.py", line 280, in wrapper
    return func(*args, **kwargs)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/cli/main.py", line 789, in freshness
    results = task.run()
  File "/Users/jerco/dev/product/dbt-core/core/dbt/task/runnable.py", line 524, in run
    result.write(self.result_path())
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 67, in write
    FreshnessExecutionResultArtifact.from_result(self).write(path)
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 110, in from_result
    processed = [process_freshness_result(r) for r in base.results]
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 110, in <listcomp>
    processed = [process_freshness_result(r) for r in base.results]
  File "/Users/jerco/dev/product/dbt-core/core/dbt/artifacts/schemas/freshness/v3/freshness.py", line 119, in process_freshness_result
    unique_id = result.node.unique_id
AttributeError: 'BaseResult' object has no attribute 'node'
```


### Environment

```markdown
- OS: 13.4.1
- Python: 3.10.11
- dbt: 1.8.0-a1
```


### Which database adapter are you using with dbt?

_No response_

### Additional Context

_No response_


Discussion:
This is an issue while serializing the `on-run-*` result into the freshness result artifact (`sources.json`) [here](https://github.com/dbt-labs/dbt-core/blob/db65e627cab89fb5ef6f1f8793aa2a261106a3ca/core/dbt/artifacts/schemas/freshness/v3/freshness.py#L108-L125). Naïve fix:
```python
    @classmethod
    def from_result(cls, base: FreshnessResult):
        # on-run-* hooks are of type BaseResult, not SourceFreshnessResult
        # they don't make sense in sources.json -- let's exclude them here
        processed = [
            process_freshness_result(r)
            for r in base.results
            if isinstance(r, SourceFreshnessResult)
        ]
        return cls(
            metadata=base.metadata,
            results=processed,
            elapsed_time=base.elapsed_time,
        )
```
@jtcohen6 @graciegoheen 

I tried the proposed fix locally and observed the following behaviors:
1. ✅ The error is surfaced in the CLI
2. ❓ The hook result isn't written to `run_results.json` or `sources.json`

**Is behavior 2 to be expected?**

Relevant logs

```bash
❯ dbt source freshness
22:24:43  Running with dbt=1.8.0-b1
22:24:43  Registered adapter: duckdb=1.7.2
22:24:43  Found 5 models, 3 seeds, 1 operation, 20 data tests, 1 source, 515 macros
22:24:43  
22:24:43  
22:24:43  Running 1 on-run-start hook
22:24:43  1 of 1 START hook: jaffle_shop.on-run-start.0 .................................. [RUN]
22:24:43  Database error while running on-run-start
22:24:43  Concurrency: 24 threads (target='dev')
22:24:43  
22:24:43  1 of 1 START freshness of external_source.flights .............................. [RUN]
22:24:43  1 of 1 WARN freshness of external_source.flights ............................... [WARN in 0.11s]
22:24:43  
22:24:43  Finished running 1 source in 0 hours 0 minutes and 0.18 seconds (0.18s).
22:24:43  
22:24:43    on-run-start failed, error:
 Catalog Error: Table with name table_does_not_exist does not exist!
Did you mean "system.information_schema.tables"?
LINE 2: select fake_column from table_does_not_exist
                                ^
22:24:43  Done.
```

```bash
❯ jq . target/sources.json
{
  "metadata": {
    "dbt_schema_version": "https://schemas.getdbt.com/dbt/sources/v3.json",
    "dbt_version": "1.8.0b1",
    "generated_at": "2024-03-17T06:05:18.514529Z",
    "invocation_id": "0f3a372c-80c0-4847-ae15-8e2f77c3a1ca",
    "env": {}
  },
  "results": [
    {
      "unique_id": "source.jaffle_shop.external_source.flights",
      "max_loaded_at": "2013-12-31T23:00:00+00:00",
      "snapshotted_at": "2024-03-17T06:05:18.448000+00:00",
      "max_loaded_at_time_ago_in_s": 322124718.448,
      "status": "warn",
      "criteria": {
        "warn_after": {
          "count": 1,
          "period": "minute"
        },
        "error_after": {
          "count": null,
          "period": null
        },
        "filter": null
      },
      "adapter_response": {
        "_message": "OK"
      },
      "timing": [
        {
          "name": "compile",
          "started_at": "2024-03-17T06:05:18.399754Z",
          "completed_at": "2024-03-17T06:05:18.399759Z"
        },
        {
          "name": "execute",
          "started_at": "2024-03-17T06:05:18.400089Z",
          "completed_at": "2024-03-17T06:05:18.508173Z"
        }
      ],
      "thread_id": "Thread-1 (worker)",
      "execution_time": 0.1091008186340332
    }
  ],
  "elapsed_time": 0.1851181983947754
}
```

```bash
❯ jq . target/run_results.json
{
  "metadata": {
    "dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v6.json",
    "dbt_version": "1.8.0b1",
    "generated_at": "2024-03-17T06:03:22.241956Z",
    "invocation_id": "bfc13b1d-bf76-4849-965b-3843b65b1c20",
    "env": {}
  },
  "results": [
    {
      "status": "success",
      "timing": [
        {
          "name": "compile",
          "started_at": "2024-03-17T06:03:22.191457Z",
          "completed_at": "2024-03-17T06:03:22.193270Z"
        },
        {
          "name": "execute",
          "started_at": "2024-03-17T06:03:22.193623Z",
          "completed_at": "2024-03-17T06:03:22.234649Z"
        }
      ],
      "thread_id": "Thread-1 (worker)",
      "execution_time": 0.04505109786987305,
      "adapter_response": {
        "_message": "OK"
      },
      "message": "OK",
      "failures": null,
      "unique_id": "model.jaffle_shop.stg_customers",
      "compiled": true,
      "compiled_code": "with source as (\n    select * from \"jaffle_shop\".\"main\".\"raw_customers\"\n\n),\n\nrenamed as (\n\n    select\n        id as customer_id,\n        first_name,\n        last_name\n\n    from source\n\n)\n\nselect * from renamed",
      "relation_name": "\"jaffle_shop\".\"main\".\"stg_customers\""
    }
  ],
  "elapsed_time": 0.11771893501281738,
  "args": {
    "exclude": [],
    "print": true,
    "printer_width": 80,
    "send_anonymous_usage_stats": true,
    "quiet": false,
    "static_parser": true,
    "vars": {},
    "empty": false,
    "version_check": true,
    "which": "run",
    "log_format": "default",
    "use_colors_file": true,
    "introspect": true,
    "source_freshness_run_project_hooks": true,
    "favor_state": false,
    "log_level_file": "debug",
    "log_format_file": "debug",
    "select": [
      "stg_customers"
    ],
    "enable_legacy_logger": false,
    "log_level": "info",
    "log_file_max_bytes": 10485760,
    "project_dir": "/Users/karanke/workspace/jaffle_shop_duckdb",
    "strict_mode": false,
    "use_colors": true,
    "defer": false,
    "cache_selected_only": false,
    "write_json": true,
    "show_resource_report": false,
    "partial_parse_file_diff": true,
    "invocation_command": "dbt run --select stg_customers",
    "partial_parse": true,
    "warn_error_options": {
      "include": [],
      "exclude": []
    },
    "log_path": "/Users/karanke/workspace/jaffle_shop_duckdb/logs",
    "populate_cache": true,
    "profiles_dir": "/Users/karanke/workspace/jaffle_shop_duckdb",
    "macro_debugging": false,
    "indirect_selection": "eager"
  }
}
```
I am surprised that the failing hook isn't written to the run_results - how are similar errors typically handled?
It looks like `on-run-start` errors for `dbt run` aren't written to `run_results.json`, so I'll maintain this behavior for `dbt source freshness` for now.

Do we want to keep this behavior, or should we create a follow-up ticket to fix this?
This is an issue while serializing the `on-run-*` result into the freshness result artifact (`sources.json`) [here](https://github.com/dbt-labs/dbt-core/blob/db65e627cab89fb5ef6f1f8793aa2a261106a3ca/core/dbt/artifacts/schemas/freshness/v3/freshness.py#L108-L125). Naïve fix:
```python
    @classmethod
    def from_result(cls, base: FreshnessResult):
        # on-run-* hooks are of type BaseResult, not SourceFreshnessResult
        # they don't make sense in sources.json -- let's exclude them here
        processed = [
            process_freshness_result(r)
            for r in base.results
            if isinstance(r, SourceFreshnessResult)
        ]
        return cls(
            metadata=base.metadata,
            results=processed,
            elapsed_time=base.elapsed_time,
        )
```
@jtcohen6 @graciegoheen 

I tried the proposed fix locally and observed the following behaviors:
1. ✅ The error is surfaced in the CLI
2. ❓ The hook result isn't written to `run_results.json` or `sources.json`

**Is behavior 2 to be expected?**

Relevant logs

```bash
❯ dbt source freshness
22:24:43  Running with dbt=1.8.0-b1
22:24:43  Registered adapter: duckdb=1.7.2
22:24:43  Found 5 models, 3 seeds, 1 operation, 20 data tests, 1 source, 515 macros
22:24:43  
22:24:43  
22:24:43  Running 1 on-run-start hook
22:24:43  1 of 1 START hook: jaffle_shop.on-run-start.0 .................................. [RUN]
22:24:43  Database error while running on-run-start
22:24:43  Concurrency: 24 threads (target='dev')
22:24:43  
22:24:43  1 of 1 START freshness of external_source.flights .............................. [RUN]
22:24:43  1 of 1 WARN freshness of external_source.flights ............................... [WARN in 0.11s]
22:24:43  
22:24:43  Finished running 1 source in 0 hours 0 minutes and 0.18 seconds (0.18s).
22:24:43  
22:24:43    on-run-start failed, error:
 Catalog Error: Table with name table_does_not_exist does not exist!
Did you mean "system.information_schema.tables"?
LINE 2: select fake_column from table_does_not_exist
                                ^
22:24:43  Done.
```

```bash
❯ jq . target/sources.json
{
  "metadata": {
    "dbt_schema_version": "https://schemas.getdbt.com/dbt/sources/v3.json",
    "dbt_version": "1.8.0b1",
    "generated_at": "2024-03-17T06:05:18.514529Z",
    "invocation_id": "0f3a372c-80c0-4847-ae15-8e2f77c3a1ca",
    "env": {}
  },
  "results": [
    {
      "unique_id": "source.jaffle_shop.external_source.flights",
      "max_loaded_at": "2013-12-31T23:00:00+00:00",
      "snapshotted_at": "2024-03-17T06:05:18.448000+00:00",
      "max_loaded_at_time_ago_in_s": 322124718.448,
      "status": "warn",
      "criteria": {
        "warn_after": {
          "count": 1,
          "period": "minute"
        },
        "error_after": {
          "count": null,
          "period": null
        },
        "filter": null
      },
      "adapter_response": {
        "_message": "OK"
      },
      "timing": [
        {
          "name": "compile",
          "started_at": "2024-03-17T06:05:18.399754Z",
          "completed_at": "2024-03-17T06:05:18.399759Z"
        },
        {
          "name": "execute",
          "started_at": "2024-03-17T06:05:18.400089Z",
          "completed_at": "2024-03-17T06:05:18.508173Z"
        }
      ],
      "thread_id": "Thread-1 (worker)",
      "execution_time": 0.1091008186340332
    }
  ],
  "elapsed_time": 0.1851181983947754
}
```

```bash
❯ jq . target/run_results.json
{
  "metadata": {
    "dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v6.json",
    "dbt_version": "1.8.0b1",
    "generated_at": "2024-03-17T06:03:22.241956Z",
    "invocation_id": "bfc13b1d-bf76-4849-965b-3843b65b1c20",
    "env": {}
  },
  "results": [
    {
      "status": "success",
      "timing": [
        {
          "name": "compile",
          "started_at": "2024-03-17T06:03:22.191457Z",
          "completed_at": "2024-03-17T06:03:22.193270Z"
        },
        {
          "name": "execute",
          "started_at": "2024-03-17T06:03:22.193623Z",
          "completed_at": "2024-03-17T06:03:22.234649Z"
        }
      ],
      "thread_id": "Thread-1 (worker)",
      "execution_time": 0.04505109786987305,
      "adapter_response": {
        "_message": "OK"
      },
      "message": "OK",
      "failures": null,
      "unique_id": "model.jaffle_shop.stg_customers",
      "compiled": true,
      "compiled_code": "with source as (\n    select * from \"jaffle_shop\".\"main\".\"raw_customers\"\n\n),\n\nrenamed as (\n\n    select\n        id as customer_id,\n        first_name,\n        last_name\n\n    from source\n\n)\n\nselect * from renamed",
      "relation_name": "\"jaffle_shop\".\"main\".\"stg_customers\""
    }
  ],
  "elapsed_time": 0.11771893501281738,
  "args": {
    "exclude": [],
    "print": true,
    "printer_width": 80,
    "send_anonymous_usage_stats": true,
    "quiet": false,
    "static_parser": true,
    "vars": {},
    "empty": false,
    "version_check": true,
    "which": "run",
    "log_format": "default",
    "use_colors_file": true,
    "introspect": true,
    "source_freshness_run_project_hooks": true,
    "favor_state": false,
    "log_level_file": "debug",
    "log_format_file": "debug",
    "select": [
      "stg_customers"
    ],
    "enable_legacy_logger": false,
    "log_level": "info",
    "log_file_max_bytes": 10485760,
    "project_dir": "/Users/karanke/workspace/jaffle_shop_duckdb",
    "strict_mode": false,
    "use_colors": true,
    "defer": false,
    "cache_selected_only": false,
    "write_json": true,
    "show_resource_report": false,
    "partial_parse_file_diff": true,
    "invocation_command": "dbt run --select stg_customers",
    "partial_parse": true,
    "warn_error_options": {
      "include": [],
      "exclude": []
    },
    "log_path": "/Users/karanke/workspace/jaffle_shop_duckdb/logs",
    "populate_cache": true,
    "profiles_dir": "/Users/karanke/workspace/jaffle_shop_duckdb",
    "macro_debugging": false,
    "indirect_selection": "eager"
  }
}
```
I am surprised that the failing hook isn't written to the run_results - how are similar errors typically handled?
It looks like `on-run-start` errors for `dbt run` aren't written to `run_results.json`, so I'll maintain this behavior for `dbt source freshness` for now.

Do we want to keep this behavior, or should we create a follow-up ticket to fix this?

Resolving patch:
diff --git a/core/dbt/artifacts/schemas/freshness/v3/freshness.py b/core/dbt/artifacts/schemas/freshness/v3/freshness.py
--- a/core/dbt/artifacts/schemas/freshness/v3/freshness.py
+++ b/core/dbt/artifacts/schemas/freshness/v3/freshness.py
@@ -107,7 +107,11 @@ class FreshnessExecutionResultArtifact(
 
     @classmethod
     def from_result(cls, base: FreshnessResult):
-        processed = [process_freshness_result(r) for r in base.results]
+        processed = [
+            process_freshness_result(r)
+            for r in base.results
+            if isinstance(r, SourceFreshnessResult)
+        ]
         return cls(
             metadata=base.metadata,
             results=processed,


Test patch:
diff --git a/tests/functional/sources/test_source_freshness.py b/tests/functional/sources/test_source_freshness.py
--- a/tests/functional/sources/test_source_freshness.py
+++ b/tests/functional/sources/test_source_freshness.py
@@ -430,6 +430,44 @@ def test_hooks_do_run_for_source_freshness(
         assert "on-run-end" in log_output
 
 
+class TestHooksInSourceFreshnessError:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "schema.yml": error_models_schema_yml,
+            "model.sql": error_models_model_sql,
+        }
+
+    @pytest.fixture(scope="class")
+    def project_config_update(self):
+        return {
+            "config-version": 2,
+            "on-run-start": ["select fake_column from table_does_not_exist"],
+            "flags": {
+                "source_freshness_run_project_hooks": True,
+            },
+        }
+
+    def test_hooks_do_not_run_for_source_freshness(
+        self,
+        project,
+    ):
+        run_result_error = None
+
+        def run_result_error_probe(e):
+            nonlocal run_result_error
+            if (
+                e.info.name == "RunResultError"
+                and e.info.level == "error"
+                and "on-run-start" in e.info.msg
+            ):
+                run_result_error = e.info.msg
+
+        runner = dbtRunner(callbacks=[run_result_error_probe])
+        runner.invoke(["source", "freshness"])
+        assert 'relation "table_does_not_exist" does not exist' in run_result_error
+
+
 class TestHooksInSourceFreshnessDisabled(SuccessfulSourceFreshnessTest):
     @pytest.fixture(scope="class")
     def project_config_update(self):