Repository: dbt-labs/dbt-core

Issue:
Selection method based on source freshness: new `max_loaded_at`, new data
See also: https://github.com/dbt-labs/dbt/issues/2465#issuecomment-763776415,  https://github.com/dbt-labs/dbt/issues/2743#issuecomment-841230958, https://github.com/dbt-labs/dbt/issues/3804#issuecomment-921115079, https://github.com/dbt-labs/dbt/issues/3862#issuecomment-916828824, [Benn's tweet](https://twitter.com/bennstancil/status/1433959272891785216)

Use the `sources.json` artifact to determine which sources have new data since the last dbt build.

Imagine:
```
$ dbt source freshness
$ dbt build -s sources:fresh+ --state my/previous/run/artifacts
```

This would require handling _multiple_ `sources.json`:
- the version from the last run in the `--state` directory
- the one from just now in the `target/` directory

Then, dbt would compare the high watermarks recorded in each, and select all source tables with a greater/changed `max_loaded_at`.

### Questions

- Should we reconsider the inclusion of the `freshness` task within `dbt build`? I think it would be _tricky_ to check source freshness _and_ perform dynamic selection _and_ build those now-selected resources, all in one invocation. It would require blurring lines that are pretty well delineated today (selection &rarr; execution &rarr; artifacts).
- Should we reconsider the consolidation of `sources.json` into `run_results.json`? It seems like the separation would actually be a plus, for the sake of this feature, but I'm sure we could manage either way
- Would it be better to perform this comparison _outside_ dbt-core (e.g. dbt Cloud metadata API), and then simply pass the set of sources that have new data? If we know that `source:stripe` and `source:salesforce.account` have new data, it's easy enough to then define a job with `dbt build -s source:stripe+ source:salesforce.account+`.

### Big idea

Coupled with `state:modified` and `result:<status>` (https://github.com/dbt-labs/dbt/pull/4017), you could dynamically select the part of your DAG that's relevant by comparing the current state of your sources/project and the artifacts from your last run.

```yml
selectors:
  - name: one_selector_to_rule_them_all
    default: true
    description: Select sub-DAG based on new source data, changed logic, &/or failure in last run
    definition:
      union:
      
        # check freshness + test all sources
        - 'source:*'
      
        # new source data + table/incremental
        - intersection:
            - union:
                - 'config.materialized:table'
                - 'config.materialized:incremental'
            - method: source
              value: fresh  # imagine
              children: true
      
        # changed logic (need this for views + MVs too)
        - method: state
          value: modified
          children: true

        # failed last time
        - union:
            - 'result:error'
            - 'result:skipped'
```
```
$ export DBT_ARTIFACT_STATE_PATH=my/previous/run/artifacts
$ dbt source freshness
$ dbt build
```

Schedule that to run every 5 minutes, set up solid slack notifications on failures, and put on a good album.

While this is very cool, it does take its cue from the source data loaders—whenever they've ingested new rows, we're going to build downstream tables. This is the inverse of defining freshness / latency expectations for specific "final" models, and using that as the primary input to deployment/scheduling. That still seems like a cool and important thing to do... perhaps via `exposures`??


Discussion:
Wow, smart freshness runs. Now this answers a question I get all the time: "How do I get dbt jobs to run based on data freshness and not generically process everything?"
@anaisvaillant and I want to work on this together 🧃 

@jtcohen6, let us know if this is dependent on other functionality before we start! I noticed you explicitly mentioned having the one selector to rule them all. Is that something you'd want baked in for testing or that can wait?
Is this a good idea? A bad one? A huge step in the wrong direction? I'm open to discussion, and also experimentation. I included the big hair-brained idea as a motivation for this line of thinking; it can definitely wait.

@sungchun12 @anaisvaillant In that spirit, I'd be thrilled if you two want to play around with this. I imagine you could reuse some similar logic from #4016, though you'll need to find ways to handle multiple `sources.json`.

@barryaron @drewbanin I'd be especially interested in your thoughts here :)
Hi Jérémy !
Thank you for your kind words concerning #4016 !
IMHO you are right, this is all about watermarks.

Here is a really good paper on the matter : 

[Data-Vault-Implementation-and-Automation-Consistency-and-Referential-Integrity](http://roelantvos.com/blog/wp-content/uploads/2018/08/Data-Vault-Implementation-and-Automation-Consistency-and-Referential-Integrity.pdf)

by Roelant Vos, a very very skilled data vault consultant. He wrote a paper about [dbt](https://roelantvos.com/blog/generating-dbt-etl-for-snowflake/)

Did you already hear about him ?

His notion of "eventual consistency" is really interesting, and I wonder how it could apply to dbt, with its current threading feature, not to mention my dream of multi-processed partitioned DAG.

In data vault's world, The timestamp the data was fiirst loaded in the DWH is a first class citizen.
In any case, this seems to have something to do with the `max_loaded_at` feature !

(@jtcohen6)

This is a great idea, and I'll expand on your original description with how I see it live and breathe in practice. 

**Problem 1**
- **Current State:** How do I create an events-based transformation pipeline, so I'm not generically running all 100 dbt models when I want to run only 10-15 dbt models every 10 minutes from my streamed in EL ingestion processes? Today, I have to go all or nothing or I have to look at history and manually scope out dbt commands to run the models in scope at specific time intervals.
- **Future State:** I have a simple conditional workflow below.
If `dbt source freshness` shows fresh data, then run `dbt build -s sources:fresh+ --state my/previous/run/artifacts`. If stale data only, run nothing. I have full confidence that all downstream data is based on the freshest data possible. No need to fiddle around with manually invoking certain models or over-engineering CLI commands to get things right. 

Regarding [Benn's Tweet](https://twitter.com/bennstancil/status/1433959272891785216), latency guarantees based on models to maintain(think: exposures) is a great working philosophy. Building this feature seems like it combats this vision at a surface level. However, this feature goes a step further in vision: latency is defined by ALL models in scope(with freshness requirements), not only those we manually scope downstream. If we work backwards from exposures, it will require at least one more step in the above workflow. I don't see value in an extra step to make some data fresh knowing ALL data can be fresh just as easily with less mental overhead. 

"I want all my data to be as fresh as possible. I shouldn't have all these mental models and over-engineered project setups to feel confident of that." **- Future data person**

I'm excited for a future where events-based data experiences are the norm and not the exception.

For your open technical implementation questions, those are for future Sung and Anais to figure out in the planning process :) 
Completely agree with @sungchun12 here.  This would be awesome, @jtcohen6.  
Still doing work on some of the other open questions (should this live within run_results.json).
Currently we are not able to do these things with dbt and therefore created all kinds of complex and expensive macros to check for looking at source freshness states. A solution like @sungchun12 did, would be great! 

Resolving patch:
diff --git a/core/dbt/contracts/state.py b/core/dbt/contracts/state.py
--- a/core/dbt/contracts/state.py
+++ b/core/dbt/contracts/state.py
@@ -1,20 +1,23 @@
 from pathlib import Path
 from .graph.manifest import WritableManifest
 from .results import RunResultsArtifact
+from .results import FreshnessExecutionResultArtifact
 from typing import Optional
 from dbt.exceptions import IncompatibleSchemaException
 
 
 class PreviousState:
-    def __init__(self, path: Path):
+    def __init__(self, path: Path, current_path: Path):
         self.path: Path = path
+        self.current_path: Path = current_path
         self.manifest: Optional[WritableManifest] = None
         self.results: Optional[RunResultsArtifact] = None
+        self.sources: Optional[FreshnessExecutionResultArtifact] = None
+        self.sources_current: Optional[FreshnessExecutionResultArtifact] = None
 
         manifest_path = self.path / "manifest.json"
         if manifest_path.exists() and manifest_path.is_file():
             try:
-                # we want to bail with an error if schema versions don't match
                 self.manifest = WritableManifest.read_and_check_versions(str(manifest_path))
             except IncompatibleSchemaException as exc:
                 exc.add_filename(str(manifest_path))
@@ -23,8 +26,27 @@ def __init__(self, path: Path):
         results_path = self.path / "run_results.json"
         if results_path.exists() and results_path.is_file():
             try:
-                # we want to bail with an error if schema versions don't match
                 self.results = RunResultsArtifact.read_and_check_versions(str(results_path))
             except IncompatibleSchemaException as exc:
                 exc.add_filename(str(results_path))
                 raise
+
+        sources_path = self.path / "sources.json"
+        if sources_path.exists() and sources_path.is_file():
+            try:
+                self.sources = FreshnessExecutionResultArtifact.read_and_check_versions(
+                    str(sources_path)
+                )
+            except IncompatibleSchemaException as exc:
+                exc.add_filename(str(sources_path))
+                raise
+
+        sources_current_path = self.current_path / "sources.json"
+        if sources_current_path.exists() and sources_current_path.is_file():
+            try:
+                self.sources_current = FreshnessExecutionResultArtifact.read_and_check_versions(
+                    str(sources_current_path)
+                )
+            except IncompatibleSchemaException as exc:
+                exc.add_filename(str(sources_current_path))
+                raise
diff --git a/core/dbt/graph/selector_methods.py b/core/dbt/graph/selector_methods.py
--- a/core/dbt/graph/selector_methods.py
+++ b/core/dbt/graph/selector_methods.py
@@ -48,6 +48,7 @@ class MethodName(StrEnum):
     Exposure = "exposure"
     Metric = "metric"
     Result = "result"
+    SourceStatus = "source_status"
 
 
 def is_selected_node(fqn: List[str], node_selector: str):
@@ -526,6 +527,62 @@ def search(self, included_nodes: Set[UniqueId], selector: str) -> Iterator[Uniqu
                 yield node
 
 
+class SourceStatusSelectorMethod(SelectorMethod):
+    def search(self, included_nodes: Set[UniqueId], selector: str) -> Iterator[UniqueId]:
+
+        if self.previous_state is None or self.previous_state.sources is None:
+            raise InternalException(
+                "No previous state comparison freshness results in sources.json"
+            )
+        elif self.previous_state.sources_current is None:
+            raise InternalException(
+                "No current state comparison freshness results in sources.json"
+            )
+
+        current_state_sources = {
+            result.unique_id: getattr(result, "max_loaded_at", None)
+            for result in self.previous_state.sources_current.results
+            if hasattr(result, "max_loaded_at")
+        }
+
+        current_state_sources_runtime_error = {
+            result.unique_id
+            for result in self.previous_state.sources_current.results
+            if not hasattr(result, "max_loaded_at")
+        }
+
+        previous_state_sources = {
+            result.unique_id: getattr(result, "max_loaded_at", None)
+            for result in self.previous_state.sources.results
+            if hasattr(result, "max_loaded_at")
+        }
+
+        previous_state_sources_runtime_error = {
+            result.unique_id
+            for result in self.previous_state.sources_current.results
+            if not hasattr(result, "max_loaded_at")
+        }
+
+        matches = set()
+        if selector == "fresher":
+            for unique_id in current_state_sources:
+                if unique_id not in previous_state_sources:
+                    matches.add(unique_id)
+                elif current_state_sources[unique_id] > previous_state_sources[unique_id]:
+                    matches.add(unique_id)
+
+            for unique_id in matches:
+                if (
+                    unique_id in previous_state_sources_runtime_error
+                    or unique_id in current_state_sources_runtime_error
+                ):
+                    matches.remove(unique_id)
+
+        for node, real_node in self.all_nodes(included_nodes):
+            if node in matches:
+                yield node
+
+
 class MethodManager:
     SELECTOR_METHODS: Dict[MethodName, Type[SelectorMethod]] = {
         MethodName.FQN: QualifiedNameSelectorMethod,
@@ -541,6 +598,7 @@ class MethodManager:
         MethodName.Exposure: ExposureSelectorMethod,
         MethodName.Metric: MetricSelectorMethod,
         MethodName.Result: ResultSelectorMethod,
+        MethodName.SourceStatus: SourceStatusSelectorMethod,
     }
 
     def __init__(
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -1,6 +1,7 @@
 import os
 import time
 import json
+from pathlib import Path
 from abc import abstractmethod
 from concurrent.futures import as_completed
 from datetime import datetime
@@ -116,7 +117,9 @@ def __init__(self, args, config):
 
     def set_previous_state(self):
         if self.args.state is not None:
-            self.previous_state = PreviousState(self.args.state)
+            self.previous_state = PreviousState(
+                path=self.args.state, current_path=Path(self.config.target_path)
+            )
 
     def index_offset(self, value: int) -> int:
         return value


Test patch:
diff --git a/test/unit/test_graph_selector_methods.py b/test/unit/test_graph_selector_methods.py
--- a/test/unit/test_graph_selector_methods.py
+++ b/test/unit/test_graph_selector_methods.py
@@ -787,7 +787,7 @@ def test_select_metric(manifest):
 @pytest.fixture
 def previous_state(manifest):
     writable = copy.deepcopy(manifest).writable_manifest()
-    state = PreviousState(Path('/path/does/not/exist'))
+    state = PreviousState(path=Path('/path/does/not/exist'),current_path=Path('/path/does/not/exist'))
     state.manifest = writable
     return state
 
diff --git a/tests/functional/sources/fixtures.py b/tests/functional/sources/fixtures.py
--- a/tests/functional/sources/fixtures.py
+++ b/tests/functional/sources/fixtures.py
@@ -68,6 +68,10 @@
           - relationships:
              to: source('test_source', 'test_table')
              field: favorite_color
+      - name: id
+        tests:
+          - unique
+          - not_null
 
 sources:
   - name: test_source
@@ -158,6 +162,10 @@
 models__nonsource_descendant_sql = """select * from {{ schema }}.source
 """
 
+models__newly_added_model_sql = """select 2 as id"""
+
+models__newly_added_error_model_sql = """select error from fake_table"""
+
 malformed_models__schema_yml = """version: 2
 sources:
   - name: test_source
diff --git a/tests/functional/sources/test_simple_source.py b/tests/functional/sources/test_simple_source.py
--- a/tests/functional/sources/test_simple_source.py
+++ b/tests/functional/sources/test_simple_source.py
@@ -73,7 +73,7 @@ def test_basic_source_def(self, project):
         )
         check_relations_equal(project.adapter, ["expected_multi_source", "multi_source_model"])
         results = self.run_dbt_with_vars(project, ["test"])
-        assert len(results) == 6
+        assert len(results) == 8
 
 
 class TestSourceSelector(SuccessfulSourcesTest):
@@ -96,16 +96,16 @@ def test_source_selector(self, project):
         results = self.run_dbt_with_vars(
             project, ["test", "--models", "source:test_source.test_table+"]
         )
-        assert len(results) == 4
+        assert len(results) == 6
 
         results = self.run_dbt_with_vars(
             project, ["test", "--models", "tag:my_test_source_table_tag+"]
         )
-        assert len(results) == 4
+        assert len(results) == 6
 
         results = self.run_dbt_with_vars(project, ["test", "--models", "tag:my_test_source_tag+"])
         # test_table + other_test_table
-        assert len(results) == 6
+        assert len(results) == 8
 
         results = self.run_dbt_with_vars(project, ["test", "--models", "tag:id_column"])
         # all 4 id column tests
diff --git a/tests/functional/sources/test_source_fresher_state.py b/tests/functional/sources/test_source_fresher_state.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/sources/test_source_fresher_state.py
@@ -0,0 +1,727 @@
+import os
+import json
+import shutil
+import pytest
+from datetime import datetime, timedelta
+
+from dbt.exceptions import InternalException
+
+
+import dbt.version
+from tests.functional.sources.common_source_setup import BaseSourcesTest
+
+from tests.functional.sources.fixtures import (
+    error_models__schema_yml,
+    models__newly_added_model_sql,
+    models__newly_added_error_model_sql,
+)
+
+
+# TODO: We may create utility classes to handle reusable fixtures.
+def copy_to_previous_state():
+    shutil.copyfile("target/manifest.json", "previous_state/manifest.json")
+    shutil.copyfile("target/run_results.json", "previous_state/run_results.json")
+
+
+# put these here for now to get tests working
+class AnyStringWith:
+    def __init__(self, contains=None):
+        self.contains = contains
+
+    def __eq__(self, other):
+        if not isinstance(other, str):
+            return False
+
+        if self.contains is None:
+            return True
+
+        return self.contains in other
+
+    def __repr__(self):
+        return "AnyStringWith<{!r}>".format(self.contains)
+
+
+class AnyFloat:
+    """Any float. Use this in assertEqual() calls to assert that it is a float."""
+
+    def __eq__(self, other):
+        return isinstance(other, float)
+
+
+class SuccessfulSourceFreshnessTest(BaseSourcesTest):
+    @pytest.fixture(scope="class", autouse=True)
+    def setUp(self, project):
+        self.run_dbt_with_vars(project, ["seed"])
+        pytest._id = 101
+        pytest.freshness_start_time = datetime.utcnow()
+        # this is the db initial value
+        pytest.last_inserted_time = "2016-09-19T14:45:51+00:00"
+
+        os.environ["DBT_ENV_CUSTOM_ENV_key"] = "value"
+
+        yield
+
+        del os.environ["DBT_ENV_CUSTOM_ENV_key"]
+
+    def _set_updated_at_to(self, project, delta):
+        insert_time = datetime.utcnow() + delta
+        timestr = insert_time.strftime("%Y-%m-%d %H:%M:%S")
+        # favorite_color,id,first_name,email,ip_address,updated_at
+        insert_id = pytest._id
+        pytest._id += 1
+        quoted_columns = ",".join(
+            project.adapter.quote(c)
+            for c in ("favorite_color", "id", "first_name", "email", "ip_address", "updated_at")
+        )
+        kwargs = {
+            "schema": project.test_schema,
+            "time": timestr,
+            "id": insert_id,
+            "source": project.adapter.quote("source"),
+            "quoted_columns": quoted_columns,
+        }
+        raw_sql = """INSERT INTO {schema}.{source}
+            ({quoted_columns})
+        VALUES (
+            'blue',{id},'Jake','abc@example.com','192.168.1.1','{time}'
+        )""".format(
+            **kwargs
+        )
+        project.run_sql(raw_sql)
+        pytest.last_inserted_time = insert_time.strftime("%Y-%m-%dT%H:%M:%S+00:00")
+
+    def assertBetween(self, timestr, start, end=None):
+        datefmt = "%Y-%m-%dT%H:%M:%S.%fZ"
+        if end is None:
+            end = datetime.utcnow()
+
+        parsed = datetime.strptime(timestr, datefmt)
+
+        assert start <= parsed
+        assert end >= parsed
+
+    def _assert_freshness_results(self, path, state):
+        assert os.path.exists(path)
+        with open(path) as fp:
+            data = json.load(fp)
+
+        assert set(data) == {"metadata", "results", "elapsed_time"}
+        assert "generated_at" in data["metadata"]
+        assert isinstance(data["elapsed_time"], float)
+        self.assertBetween(data["metadata"]["generated_at"], pytest.freshness_start_time)
+        assert (
+            data["metadata"]["dbt_schema_version"]
+            == "https://schemas.getdbt.com/dbt/sources/v3.json"
+        )
+        assert data["metadata"]["dbt_version"] == dbt.version.__version__
+        assert data["metadata"]["invocation_id"] == dbt.tracking.active_user.invocation_id
+        key = "key"
+        if os.name == "nt":
+            key = key.upper()
+        assert data["metadata"]["env"] == {key: "value"}
+
+        last_inserted_time = pytest.last_inserted_time
+
+        assert len(data["results"]) == 1
+
+        assert data["results"] == [
+            {
+                "unique_id": "source.test.test_source.test_table",
+                "max_loaded_at": last_inserted_time,
+                "snapshotted_at": AnyStringWith(),
+                "max_loaded_at_time_ago_in_s": AnyFloat(),
+                "status": state,
+                "criteria": {
+                    "filter": None,
+                    "warn_after": {"count": 10, "period": "hour"},
+                    "error_after": {"count": 18, "period": "hour"},
+                },
+                "adapter_response": {},
+                "thread_id": AnyStringWith("Thread-"),
+                "execution_time": AnyFloat(),
+                "timing": [
+                    {
+                        "name": "compile",
+                        "started_at": AnyStringWith(),
+                        "completed_at": AnyStringWith(),
+                    },
+                    {
+                        "name": "execute",
+                        "started_at": AnyStringWith(),
+                        "completed_at": AnyStringWith(),
+                    },
+                ],
+            }
+        ]
+
+
+class TestSourceFresherNothingToDo(SuccessfulSourceFreshnessTest):
+    def test_source_fresher_nothing_to_do(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        self._set_updated_at_to(project, timedelta(hours=-2))
+        previous_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "previous_state/sources.json"]
+        )
+        self._assert_freshness_results("previous_state/sources.json", "pass")
+        copy_to_previous_state()
+
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+        self._assert_freshness_results("target/sources.json", "pass")
+
+        assert previous_state_results[0].max_loaded_at == current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher+", "--defer", "--state", "./previous_state"],
+        )
+        assert source_fresher_results.results == []
+
+
+class TestSourceFresherRun(SuccessfulSourceFreshnessTest):
+    def test_source_fresher_run_error(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=False,
+        )
+        self._assert_freshness_results("previous_state/sources.json", "error")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-20))
+        current_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "target/sources.json"],
+            expect_pass=False,
+        )
+        self._assert_freshness_results("target/sources.json", "error")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["run", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        assert source_fresher_results.results == []
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["run", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {"descendant_model"}
+
+    def test_source_fresher_run_warn(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        self._set_updated_at_to(project, timedelta(hours=-17))
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=True,
+        )
+        self._assert_freshness_results("previous_state/sources.json", "warn")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-11))
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+        self._assert_freshness_results("target/sources.json", "warn")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["run", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        assert source_fresher_results.results == []
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["run", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {"descendant_model"}
+
+    def test_source_fresher_run_pass(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        self._set_updated_at_to(project, timedelta(hours=-2))
+        previous_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "previous_state/sources.json"]
+        )
+        self._assert_freshness_results("previous_state/sources.json", "pass")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-1))
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+        self._assert_freshness_results("target/sources.json", "pass")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["run", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        assert source_fresher_results.results == []
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["run", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {"descendant_model"}
+
+
+class TestSourceFresherBuildStateModified(SuccessfulSourceFreshnessTest):
+    def test_source_fresher_build_state_modified_pass(self, project, project_root):
+        self.run_dbt_with_vars(project, ["run"])
+
+        self._set_updated_at_to(project, timedelta(hours=-2))
+        previous_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "previous_state/sources.json"]
+        )
+        self._assert_freshness_results("previous_state/sources.json", "pass")
+
+        self._set_updated_at_to(project, timedelta(hours=-1))
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+        self._assert_freshness_results("target/sources.json", "pass")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        models_path = project_root.join("models/")
+        assert os.path.exists(models_path)
+        with open(f"{models_path}/newly_added_model.sql", "w") as fp:
+            fp.write(models__newly_added_model_sql)
+
+        copy_to_previous_state()
+        state_modified_results = self.run_dbt_with_vars(
+            project,
+            [
+                "build",
+                "--select",
+                "source_status:fresher+",
+                "state:modified+",
+                "--defer",
+                "--state",
+                "previous_state",
+            ],
+        )
+        nodes = set([elem.node.name for elem in state_modified_results])
+        assert nodes == {
+            "newly_added_model",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+            "source_not_null_test_source_test_table_id",
+            "descendant_model",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+        }
+
+
+class TestSourceFresherRuntimeError(SuccessfulSourceFreshnessTest):
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "schema.yml": error_models__schema_yml,
+        }
+
+    def test_runtime_error_states(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=False,
+        )
+        assert len(previous_state_results) == 1
+        assert previous_state_results[0].status == "runtime error"
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-1))
+        current_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "target/sources.json"],
+            expect_pass=False,
+        )
+        assert len(current_state_results) == 1
+        assert current_state_results[0].status == "runtime error"
+
+        assert not hasattr(previous_state_results[0], "max_loaded_at")
+        assert not hasattr(current_state_results[0], "max_loaded_at")
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        assert source_fresher_results.results == []
+
+
+class TestSourceFresherTest(SuccessfulSourceFreshnessTest):
+    def test_source_fresher_run_error(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=False,
+        )
+        self._assert_freshness_results("previous_state/sources.json", "error")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-20))
+        current_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "target/sources.json"],
+            expect_pass=False,
+        )
+        self._assert_freshness_results("target/sources.json", "error")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+        }
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+        }
+
+    def test_source_fresher_test_warn(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        self._set_updated_at_to(project, timedelta(hours=-17))
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=True,
+        )
+        self._assert_freshness_results("previous_state/sources.json", "warn")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-11))
+        current_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "target/sources.json"],
+            expect_pass=True,
+        )
+        self._assert_freshness_results("target/sources.json", "warn")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+        }
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+        }
+
+    def test_source_fresher_test_pass(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        self._set_updated_at_to(project, timedelta(hours=-2))
+        previous_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "previous_state/sources.json"]
+        )
+        self._assert_freshness_results("previous_state/sources.json", "pass")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-1))
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+        self._assert_freshness_results("target/sources.json", "pass")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+        }
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["test", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+        }
+
+
+class TestSourceFresherBuild(SuccessfulSourceFreshnessTest):
+    def test_source_fresher_build_error(self, project):
+        self.run_dbt_with_vars(project, ["build"])
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=False,
+        )
+        self._assert_freshness_results("previous_state/sources.json", "error")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-20))
+        current_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "target/sources.json"],
+            expect_pass=False,
+        )
+        self._assert_freshness_results("target/sources.json", "error")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["build", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+        }
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["build", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {
+            "descendant_model",
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+        }
+
+    def test_source_fresher_build_warn(self, project):
+        self.run_dbt_with_vars(project, ["build"])
+        self._set_updated_at_to(project, timedelta(hours=-17))
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=True,
+        )
+        self._assert_freshness_results("previous_state/sources.json", "warn")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-11))
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+        self._assert_freshness_results("target/sources.json", "warn")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["build", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+        }
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["build", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {
+            "descendant_model",
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+        }
+
+    def test_source_fresher_build_pass(self, project):
+        self.run_dbt_with_vars(project, ["build"])
+        self._set_updated_at_to(project, timedelta(hours=-2))
+        previous_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "previous_state/sources.json"]
+        )
+        self._assert_freshness_results("previous_state/sources.json", "pass")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-1))
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+        self._assert_freshness_results("target/sources.json", "pass")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        source_fresher_results = self.run_dbt_with_vars(
+            project,
+            ["build", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_results])
+        assert nodes == {
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+        }
+
+        source_fresher_plus_results = self.run_dbt_with_vars(
+            project,
+            ["build", "-s", "source_status:fresher+", "--defer", "--state", "previous_state"],
+        )
+        nodes = set([elem.node.name for elem in source_fresher_plus_results])
+        assert nodes == {
+            "descendant_model",
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+            "source_not_null_test_source_test_table_id",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+        }
+
+
+class TestSourceFresherNoPreviousState(SuccessfulSourceFreshnessTest):
+    def test_intentional_failure_no_previous_state(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        # TODO add the current and previous but with previous as null
+        with pytest.raises(InternalException) as excinfo:
+            self.run_dbt_with_vars(
+                project,
+                ["run", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+            )
+        assert "No previous state comparison freshness results in sources.json" in str(
+            excinfo.value
+        )
+
+
+class TestSourceFresherNoCurrentState(SuccessfulSourceFreshnessTest):
+    def test_intentional_failure_no_previous_state(self, project):
+        self.run_dbt_with_vars(project, ["run"])
+        previous_state_results = self.run_dbt_with_vars(
+            project,
+            ["source", "freshness", "-o", "previous_state/sources.json"],
+            expect_pass=False,
+        )
+        self._assert_freshness_results("previous_state/sources.json", "error")
+        copy_to_previous_state()
+        assert previous_state_results[0].max_loaded_at is not None
+
+        with pytest.raises(InternalException) as excinfo:
+            self.run_dbt_with_vars(
+                project,
+                ["run", "-s", "source_status:fresher", "--defer", "--state", "previous_state"],
+            )
+        assert "No current state comparison freshness results in sources.json" in str(
+            excinfo.value
+        )
+
+
+class TestSourceFresherBuildResultSelectors(SuccessfulSourceFreshnessTest):
+    def test_source_fresher_build_state_modified_pass(self, project, project_root):
+        models_path = project_root.join("models/")
+        assert os.path.exists(models_path)
+        with open(f"{models_path}/newly_added_error_model.sql", "w") as fp:
+            fp.write(models__newly_added_error_model_sql)
+
+        self.run_dbt_with_vars(project, ["run"], expect_pass=False)
+
+        self._set_updated_at_to(project, timedelta(hours=-2))
+        previous_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "previous_state/sources.json"]
+        )
+
+        self._assert_freshness_results("previous_state/sources.json", "pass")
+        copy_to_previous_state()
+
+        self._set_updated_at_to(project, timedelta(hours=-1))
+        current_state_results = self.run_dbt_with_vars(
+            project, ["source", "freshness", "-o", "target/sources.json"]
+        )
+
+        self._assert_freshness_results("target/sources.json", "pass")
+
+        assert previous_state_results[0].max_loaded_at < current_state_results[0].max_loaded_at
+
+        state_modified_results = self.run_dbt_with_vars(
+            project,
+            [
+                "build",
+                "--select",
+                "source_status:fresher+",
+                "result:error+",
+                "--defer",
+                "--state",
+                "previous_state",
+            ],
+            expect_pass=False,
+        )
+        nodes = set([elem.node.name for elem in state_modified_results])
+        assert nodes == {
+            "newly_added_error_model",
+            "source_unique_test_source_test_table_id",
+            "unique_descendant_model_id",
+            "not_null_descendant_model_id",
+            "source_not_null_test_source_test_table_id",
+            "descendant_model",
+            "source_relationships_test_source_test_table_favorite_color__favorite_color__ref_descendant_model_",
+            "relationships_descendant_model_favorite_color__favorite_color__source_test_source_test_table_",
+        }