Repository: dbt-labs/dbt-core

Issue:
[CT-985] [Feature] on-run-end hook and results variable support for source freshness command 
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

- Currently `dbt source freshness` does not support running an [on-run-end hook](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end). 
- Other dbt commands like `dbt run`, `dbt test`, `dbt snapshot` and more already support running an on-run-end hook and [getting results jinja variable](https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context) as their context. 
- Using an on-run-end hook and the results variable in the `dbt source freshness` command, we can upload the source freshness results to the data warehouse. This could be used for displaying source freshness results on a dashboard, monitoring, and alerting.

### Describe alternatives you've considered

- The `dbt source freshness` command already writes the freshness results to a file called sources.json. 
- A possible alternative is to process this json file to get the source freshness results. 
- This solution requires implementing an external solution that will process this json and automatically upload the results to the data warehouse after every execution of the `dbt source freshness` command. 
- This alternative is harder to implement and requires implementing code outside of the dbt project. 
- Additionally, this solution requires orchestrating this new tool and giving it access to the same project so it can extract the source freshness results from the sources.json file.
- Support for on-run-end and results variable will simplify this dramatically, enabling it to be implemented in the dbt project itself.

### Who will this benefit?

- dbt users who rely on on-run-end hook and the results variable to monitor their dbt commands, executions and results.
- dbt users who want to build dashboards to present their source freshness results.
- dbt users who rely on dbt packages like [Elementary](https://github.com/elementary-data/dbt-data-reliability) to upload their dbt artifacts and monitor their dbt projects. 

### Are you interested in contributing this feature?

Yes

### Anything else?

_No response_


Discussion:
@lostmygithubaccount is something I can contribute to?
@jtalmi apologies I missed that comment! I've been messing around with how I check my GitHub notifications and I think it fell through the cracks. We would welcome contribution on this issue and can likely provide some guidance if needed. 

@oravi thank you for opening. `dbt source freshness` is a bit different than the other commands that support on-run-end hooks, but we don't see any reason not to allow this. That said, it's likely not something we can prioritize in the near future and so we would be looking for help with the community (it sounds like @jtalmi might be up for it!). We'll leave this issue open on the backlog and mark it as help wanted. 
The `on-run-*` hook + context behavior is defined in the `safe_run_hooks`, `before_run`, and `after_run` methods of the `RunTask`:

https://github.com/dbt-labs/dbt-core/blob/a3b018fd3b0c789c976e0e189788471edef8acb1/core/dbt/task/run.py#L318-L393

https://github.com/dbt-labs/dbt-core/blob/a3b018fd3b0c789c976e0e189788471edef8acb1/core/dbt/task/run.py#L418-L439

The reason `FreshnessTask` doesn't run hooks today is that [it inherits from the `GraphRunnableTask`](https://github.com/dbt-labs/dbt-core/blob/a3b018fd3b0c789c976e0e189788471edef8acb1/core/dbt/task/freshness.py#L183) (which doesn't know about `on-run-*` hooks), rather than the `RunTask`. To turn on hooks for `source freshness`, that hook code might need to move back onto `GraphRunnableTask` to accommodate. If that's the approach taken, `CompileTask` would need to disable those hook methods in order to preserve existing behavior (= not running `on-run-*` hooks).
Thanks @jtcohen6 ! could you please share a bit more context about GraphRunnableTask and RunTask? What's the unique purpose of each one of them? Specifically curious to learn why `FreshnessTask` does not inherit from `RunTask`. 

Also in case we will go with the suggested approach - as you mentioned I saw that `CompileTask` inherits from `GraphRunnableTask` and `RunTask` inherits from `CompileTask`. When you say that we need to disable those hooks to avoid running them during compile - you mean we just need to override them inside `CompileTask` and do nothing inside their implementation (just return). Then in `RunTask` should we specifically call the implementation of `GraphRunnableTask`?

Thanks for all your help here!
Hi @jtcohen6 @lostmygithubaccount,
I'm thinking of picking this task up soon.
I would appreciate a confirmation that this feature could be merged once implemented and reviewed.
Thanks.
hi @elongl, Jeremy or one of the engineers should be able to answer for sure -- I think so, though there are some ongoing internal API refactors for v1.5 that could affect this
@oravi @elongl Sorry for the long delay getting back to you!

Our current task logic is implemented with a lot of class inheritance, rather than composition, which makes this logic much trickier to follow than it should be. That is something we'd be hoping to improve as part of our API refactors over the next several months, but we haven't touched this part of the codebase just yet.

I think the desirable outcomes here are:
- Move logic on running hooks up into `GraphRunnableTask`
- Use that same logic in `FreshnessTask`, which should run hooks (this issue)
- Disable that logic in `CompileTask`, which shouldn't run hooks

If one of you wanted to open a PR for that, I'd be happy to see it merged!

> `CompileTask`, which shouldn't run hooks

This is status quo & my opinion — does anyone have a different opinion?
Hi @jtcohen6 👋 
I'd be happy to contribute this feature, tho i have a question about the solution you suggested:
As i look at the code now, it seems that more classes inherit from `GraphRunnableTask` (like `CloneTask` and `ListTask`), so your suggested solution of moving the logic to it and disabling it in the inheriting classes sounds a bit messy, so i thought of a few solutions:
 * Make `FreshnessTask` inherit from `RunTask` as well, tho there might be a reason i do not fully understand that explains why this cannot be the case.
 * Add a new class to the hierarchy, something like `HooksTask` that exists lower in the hierarchy tree, and both `RunTask` and `FreshnessTask` inherit.

Resolving patch:
diff --git a/core/dbt/cli/flags.py b/core/dbt/cli/flags.py
--- a/core/dbt/cli/flags.py
+++ b/core/dbt/cli/flags.py
@@ -236,8 +236,8 @@ def _assign_params(
         # Add entire invocation command to flags
         object.__setattr__(self, "INVOCATION_COMMAND", "dbt " + " ".join(sys.argv[1:]))
 
-        # Overwrite default assignments with user config if available.
         if project_flags:
+            # Overwrite default assignments with project flags if available.
             param_assigned_from_default_copy = params_assigned_from_default.copy()
             for param_assigned_from_default in params_assigned_from_default:
                 project_flags_param_value = getattr(
@@ -252,6 +252,13 @@ def _assign_params(
                     param_assigned_from_default_copy.remove(param_assigned_from_default)
             params_assigned_from_default = param_assigned_from_default_copy
 
+            # Add project-level flags that are not available as CLI options / env vars
+            for (
+                project_level_flag_name,
+                project_level_flag_value,
+            ) in project_flags.project_only_flags.items():
+                object.__setattr__(self, project_level_flag_name.upper(), project_level_flag_value)
+
         # Set hard coded flags.
         object.__setattr__(self, "WHICH", invoked_subcommand_name or ctx.info_name)
 
diff --git a/core/dbt/contracts/project.py b/core/dbt/contracts/project.py
--- a/core/dbt/contracts/project.py
+++ b/core/dbt/contracts/project.py
@@ -307,6 +307,7 @@ class ProjectFlags(ExtensibleDbtClassMixin, Replaceable):
     populate_cache: Optional[bool] = None
     printer_width: Optional[int] = None
     send_anonymous_usage_stats: bool = DEFAULT_SEND_ANONYMOUS_USAGE_STATS
+    source_freshness_run_project_hooks: bool = False
     static_parser: Optional[bool] = None
     use_colors: Optional[bool] = None
     use_colors_file: Optional[bool] = None
@@ -316,6 +317,10 @@ class ProjectFlags(ExtensibleDbtClassMixin, Replaceable):
     warn_error_options: Optional[Dict[str, Union[str, List[str]]]] = None
     write_json: Optional[bool] = None
 
+    @property
+    def project_only_flags(self) -> Dict[str, Any]:
+        return {"source_freshness_run_project_hooks": self.source_freshness_run_project_hooks}
+
 
 @dataclass
 class ProfileConfig(dbtClassMixin, Replaceable):
diff --git a/core/dbt/task/freshness.py b/core/dbt/task/freshness.py
--- a/core/dbt/task/freshness.py
+++ b/core/dbt/task/freshness.py
@@ -1,13 +1,13 @@
 import os
 import threading
 import time
-from typing import Optional
+from typing import Optional, List
 
 from .base import BaseRunner
 from .printer import (
     print_run_result_error,
 )
-from .runnable import GraphRunnableTask
+from .run import RunTask
 
 from dbt.artifacts.freshness import (
     FreshnessResult,
@@ -23,11 +23,12 @@
     LogStartLine,
     LogFreshnessResult,
 )
-from dbt.node_types import NodeType
+from dbt.contracts.results import RunStatus
+from dbt.node_types import NodeType, RunHookType
 
 from dbt.adapters.capability import Capability
 from dbt.adapters.contracts.connection import AdapterResponse
-from dbt.contracts.graph.nodes import SourceDefinition
+from dbt.contracts.graph.nodes import SourceDefinition, HookNode
 from dbt_common.events.base_types import EventLevel
 from dbt.graph import ResourceTypeSelector
 
@@ -170,7 +171,7 @@ def node_is_match(self, node):
         return node.has_freshness
 
 
-class FreshnessTask(GraphRunnableTask):
+class FreshnessTask(RunTask):
     def result_path(self):
         if self.args.output:
             return os.path.realpath(self.args.output)
@@ -200,7 +201,17 @@ def get_result(self, results, elapsed_time, generated_at):
 
     def task_end_messages(self, results):
         for result in results:
-            if result.status in (FreshnessStatus.Error, FreshnessStatus.RuntimeErr):
+            if result.status in (
+                FreshnessStatus.Error,
+                FreshnessStatus.RuntimeErr,
+                RunStatus.Error,
+            ):
                 print_run_result_error(result)
 
         fire_event(FreshnessCheckComplete())
+
+    def get_hooks_by_type(self, hook_type: RunHookType) -> List[HookNode]:
+        if self.args.source_freshness_run_project_hooks:
+            return super().get_hooks_by_type(hook_type)
+        else:
+            return []


Test patch:
diff --git a/tests/functional/sources/common_source_setup.py b/tests/functional/sources/common_source_setup.py
--- a/tests/functional/sources/common_source_setup.py
+++ b/tests/functional/sources/common_source_setup.py
@@ -2,7 +2,7 @@
 import pytest
 import yaml
 
-from dbt.tests.util import run_dbt
+from dbt.tests.util import run_dbt, run_dbt_and_capture
 from tests.functional.sources.fixtures import (
     models_schema_yml,
     models_view_model_sql,
@@ -57,10 +57,17 @@ def project_config_update(self):
             },
         }
 
-    def run_dbt_with_vars(self, project, cmd, *args, **kwargs):
+    def _extend_cmd_with_vars(self, project, cmd):
         vars_dict = {
             "test_run_schema": project.test_schema,
             "test_loaded_at": project.adapter.quote("updated_at"),
         }
         cmd.extend(["--vars", yaml.safe_dump(vars_dict)])
+
+    def run_dbt_with_vars(self, project, cmd, *args, **kwargs):
+        self._extend_cmd_with_vars(project, cmd)
         return run_dbt(cmd, *args, **kwargs)
+
+    def run_dbt_and_capture_with_vars(self, project, cmd, *args, **kwargs):
+        self._extend_cmd_with_vars(project, cmd)
+        return run_dbt_and_capture(cmd, *args, **kwargs)
diff --git a/tests/functional/sources/test_source_freshness.py b/tests/functional/sources/test_source_freshness.py
--- a/tests/functional/sources/test_source_freshness.py
+++ b/tests/functional/sources/test_source_freshness.py
@@ -400,3 +400,85 @@ def warning_probe(e):
         runner.invoke(["parse"])
 
         assert got_warning
+
+
+class TestHooksInSourceFreshness(SuccessfulSourceFreshnessTest):
+    @pytest.fixture(scope="class")
+    def project_config_update(self):
+        return {
+            "config-version": 2,
+            "on-run-start": ["{{ log('on-run-start hooks called') }}"],
+            "on-run-end": ["{{ log('on-run-end hooks called') }}"],
+            "flags": {
+                "source_freshness_run_project_hooks": True,
+            },
+        }
+
+    def test_hooks_do_run_for_source_freshness(
+        self,
+        project,
+    ):
+        _, log_output = self.run_dbt_and_capture_with_vars(
+            project,
+            [
+                "source",
+                "freshness",
+            ],
+            expect_pass=False,
+        )
+        assert "on-run-start" in log_output
+        assert "on-run-end" in log_output
+
+
+class TestHooksInSourceFreshnessDisabled(SuccessfulSourceFreshnessTest):
+    @pytest.fixture(scope="class")
+    def project_config_update(self):
+        return {
+            "config-version": 2,
+            "on-run-start": ["{{ log('on-run-start hooks called') }}"],
+            "on-run-end": ["{{ log('on-run-end hooks called') }}"],
+            "flags": {
+                "source_freshness_run_project_hooks": False,
+            },
+        }
+
+    def test_hooks_do_run_for_source_freshness(
+        self,
+        project,
+    ):
+        _, log_output = self.run_dbt_and_capture_with_vars(
+            project,
+            [
+                "source",
+                "freshness",
+            ],
+            expect_pass=False,
+        )
+        assert "on-run-start" not in log_output
+        assert "on-run-end" not in log_output
+
+
+class TestHooksInSourceFreshnessDefault(SuccessfulSourceFreshnessTest):
+    @pytest.fixture(scope="class")
+    def project_config_update(self):
+        return {
+            "config-version": 2,
+            "on-run-start": ["{{ log('on-run-start hooks called') }}"],
+            "on-run-end": ["{{ log('on-run-end hooks called') }}"],
+        }
+
+    def test_hooks_do_run_for_source_freshness(
+        self,
+        project,
+    ):
+        _, log_output = self.run_dbt_and_capture_with_vars(
+            project,
+            [
+                "source",
+                "freshness",
+            ],
+            expect_pass=False,
+        )
+        # default behaviour - no hooks run in source freshness
+        assert "on-run-start" not in log_output
+        assert "on-run-end" not in log_output
diff --git a/tests/unit/test_cli_flags.py b/tests/unit/test_cli_flags.py
--- a/tests/unit/test_cli_flags.py
+++ b/tests/unit/test_cli_flags.py
@@ -366,6 +366,14 @@ def test_global_flag_at_child_context(self):
 
         assert flags_a.USE_COLORS == flags_b.USE_COLORS
 
+    def test_set_project_only_flags(self, project_flags, run_context):
+        flags = Flags(run_context, project_flags)
+
+        for project_only_flag, project_only_flag_value in project_flags.project_only_flags.items():
+            assert getattr(flags, project_only_flag) == project_only_flag_value
+            # sanity check: ensure project_only_flag is not part of the click context
+            assert project_only_flag not in run_context.params
+
     def _create_flags_from_dict(self, cmd, d):
         write_file("", "profiles.yml")
         result = Flags.from_dict(cmd, d)
diff --git a/tests/unit/test_graph.py b/tests/unit/test_graph.py
--- a/tests/unit/test_graph.py
+++ b/tests/unit/test_graph.py
@@ -16,6 +16,7 @@
 from dbt import tracking
 from dbt.contracts.files import SourceFile, FileHash, FilePath
 from dbt.contracts.graph.manifest import MacroManifest, ManifestStateCheck
+from dbt.contracts.project import ProjectFlags
 from dbt.graph import NodeSelector, parse_difference
 from dbt.events.logging import setup_event_logger
 from dbt.mp_context import get_mp_context
@@ -130,7 +131,7 @@ def get_config(self, extra_cfg=None):
         cfg.update(extra_cfg)
 
         config = config_from_parts_or_dicts(project=cfg, profile=self.profile)
-        dbt.flags.set_from_args(Namespace(), config)
+        dbt.flags.set_from_args(Namespace(), ProjectFlags())
         setup_event_logger(dbt.flags.get_flags())
         object.__setattr__(dbt.flags.get_flags(), "PARTIAL_PARSE", False)
         return config