Repository: dbt-labs/dbt-core

Issue:
[CT-3515] [Feature] Source freshness: allow `loaded_at_field` set to `null` at table level to overwrite default value set at the source level
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

While defining source freshness we use 2 properties: `freshness` and `loaded_at_field`.
Value resolution is different (from hierarchy perspective) for these 2 properties:

When `freshness` is set to None at table level (using `freshness: ` or `freshness: null`) it overwrites `freshness` set at source level, but `loaded_at_field` set to None at table level doesn't overwrite value set at source level.

### Expected Behavior

`loaded_at_field` set to None at table level overwrites value set at source level

### Steps To Reproduce

```yaml
version: 2
sources:
  - name: source_name
    database: db_name
    schema: schema_name
    freshness:
      warn_after:
        count: 1
        period: day
      error_after:
        count: 4
        period: day
    loaded_at_field: loaded_at
    tables:
      - name: loaded_at_is_overwritten
        loaded_at_field: xxx
      - name: loaded_at_is_NOT_overwritten
        loaded_at_field: null
      - name: loaded_at_is_NOT_overwritten2
        loaded_at_field: 
      - name: loaded_at_is_NOT_overwritten3
        loaded_at_field: ~
      - name: freshness_disabled_OK
        freshness: null
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: Windows/Linux
- Python: 3.11
- dbt-core: 1.7.4
- dbt-snowflake: 1.7.1
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_


Discussion:
Here is DBT project (using sqlite) which shows the error:
[yaml_null_error.zip](https://github.com/dbt-labs/dbt-core/files/13794818/yaml_null_error.zip)

Steps to reproduce:
1. unpack
2. `pip install -r requirements.txt`
3. `dbt seed`
4. `dbt source freshness`

Sources yaml (present in the project):
```yaml
version: 2
sources:
  - name: sources_db
    database: test
    schema: main
    freshness:
      warn_after:
        count: 1
        period: day
      error_after:
        count: 4
        period: day
    loaded_at_field: absent
    tables:
      - name: freshness_fails
        identifier: example
        loaded_at_field: null
        description: |
          loaded_at_field is set to null and thus DBT should use warehouse metadata (at least try)
          Notice error message:
          16:07:56    Runtime Error in source freshness_fails (models\sources.yml)
            Database Error
              no such column: absent
      - name: freshness_skipped
        identifier: example
        freshness: null
```

Thanks for reaching out @kokorin !

The crux is that it's tricky to within the dbt implementation details to determine the difference between an _explicit_ null / none and an _implicit_ one. Our current implementation treats them both as the same thing, and I don't think we're inclined to change it at this point.

Either way, it doesn't look like dbt is behaving erroneously here, so I'm going to re-categorize this as a feature request.

Have you tried either of the following instead?

### Option 1

Two sources. One for freshness from metadata and the other based off column names.

```yaml
  # source tables with source freshness from metadata
  - name: sources_with_metadata_based_freshness
    database: db_name
    schema: schema_name
    freshness:
      warn_after:
        count: 1
        period: day
      error_after:
        count: 4
        period: day
    tables:
      - name: table_1
      - name: table_2

  # source tables with source freshness based off column names
  - name: sources_with_column_based_freshness
    database: db_name
    schema: schema_name
    freshness:
      warn_after:
        count: 1
        period: day
      error_after:
        count: 4
        period: day
    loaded_at_field: loaded_at  # default at the source level
    tables:
      - name: table_3
      - name: table_4
      - name: table_5
        loaded_at_field: xxx  # override of source-level default
```

### Option 2

One source.  `loaded_at_field` is defined under each table name instead of having a default at the source level.

```yaml
  - name: sources_with_metadata_freshness
    database: db_name
    schema: schema_name
    freshness:
      warn_after:
        count: 1
        period: day
      error_after:
        count: 4
        period: day

    # no source-level default for loaded_at_field

    tables:

      # source tables with source freshness from metadata
      - name: table_1
      - name: table_2

      # source tables with standardized loaded_at_field
      - name: table_3
        loaded_at_field: loaded_at
      - name: table_4
        loaded_at_field: loaded_at

      # other source tables with custom loaded_at_field
      - name: table_5
        loaded_at_field: xxx
```
@dbeatty10 thank you for the reply and for suggestions. I thought about the same options.

> Either way, it doesn't look like dbt is behaving erroneously here, so I'm going to re-categorize this as a feature request.

Sorry, but I can't agree. I still consider this as bug because behavior is different for `freshness` and `loaded_at_field` properties.
Freshness set to explicit null at table level overwrites freshness at source level, while `loaded_at_field` does not.
The problem is that those 2 properties are often used together and should behave similar, otherwise it's not clear for end users.

I can work on pull request if you consider it possible.
> ... behavior is different for freshness and loaded_at_field properties.
> Freshness set to explicit null at table level overwrites `freshness` at source level, while `loaded_at_field` does not.

Thanks for pointing this out @kokorin 👍 Agreed that it's less clear for end users if `freshness` and `loaded_at_field` don't behave similarly to each other.

I'm going to label this as "refinement" for us to 1) reproduce that they are behaving differently and 2) determine how they should behave going forward.
@dbeatty10 please notice that I have attached zip archive to this ticket with minimal project reproducing the issue 
> I can work on pull request if you consider it possible.

That would be awesome if you work on a pull request for this @kokorin 🏆

@graciegoheen and I discussed briefly offline, and we're planning to keep this labeled as a feature request and not backport it to dbt-core 1.7.

### Acceptance criteria

- To align with `freshness`, when `loaded_at_field` is set with an explicit `null` at the table level, it should overwrite `loaded_at_field` which is set at the source level
- When `loaded_at_field` is null, then it uses metadata-based source freshness checks

Resolving patch:
diff --git a/core/dbt/contracts/graph/unparsed.py b/core/dbt/contracts/graph/unparsed.py
--- a/core/dbt/contracts/graph/unparsed.py
+++ b/core/dbt/contracts/graph/unparsed.py
@@ -269,6 +269,7 @@ class UnparsedMacroUpdate(HasConfig, HasColumnProps, HasYamlMetadata):
 class UnparsedSourceTableDefinition(HasColumnTests, HasColumnAndTestProps):
     config: Dict[str, Any] = field(default_factory=dict)
     loaded_at_field: Optional[str] = None
+    loaded_at_field_present: Optional[bool] = None
     identifier: Optional[str] = None
     quoting: Quoting = field(default_factory=Quoting)
     freshness: Optional[FreshnessThreshold] = field(default_factory=FreshnessThreshold)
@@ -293,10 +294,22 @@ class UnparsedSourceDefinition(dbtClassMixin):
     quoting: Quoting = field(default_factory=Quoting)
     freshness: Optional[FreshnessThreshold] = field(default_factory=FreshnessThreshold)
     loaded_at_field: Optional[str] = None
+    loaded_at_field_present: Optional[bool] = None
     tables: List[UnparsedSourceTableDefinition] = field(default_factory=list)
     tags: List[str] = field(default_factory=list)
     config: Dict[str, Any] = field(default_factory=dict)
 
+    @classmethod
+    def validate(cls, data):
+        super(UnparsedSourceDefinition, cls).validate(data)
+
+        if data.get("loaded_at_field", None) == "":
+            raise ValidationError("loaded_at_field cannot be an empty string.")
+        if "tables" in data:
+            for table in data["tables"]:
+                if table.get("loaded_at_field", None) == "":
+                    raise ValidationError("loaded_at_field cannot be an empty string.")
+
     @property
     def yaml_key(self) -> "str":
         return "sources"
@@ -316,6 +329,7 @@ class SourceTablePatch(dbtClassMixin):
     data_type: Optional[str] = None
     docs: Optional[Docs] = None
     loaded_at_field: Optional[str] = None
+    loaded_at_field_present: Optional[bool] = None
     identifier: Optional[str] = None
     quoting: Quoting = field(default_factory=Quoting)
     freshness: Optional[FreshnessThreshold] = field(default_factory=FreshnessThreshold)
@@ -358,6 +372,7 @@ class SourcePatch(dbtClassMixin):
     quoting: Optional[Quoting] = None
     freshness: Optional[Optional[FreshnessThreshold]] = field(default_factory=FreshnessThreshold)
     loaded_at_field: Optional[str] = None
+    loaded_at_field_present: Optional[bool] = None
     tables: Optional[List[SourceTablePatch]] = None
     tags: Optional[List[str]] = None
 
diff --git a/core/dbt/parser/schemas.py b/core/dbt/parser/schemas.py
--- a/core/dbt/parser/schemas.py
+++ b/core/dbt/parser/schemas.py
@@ -111,11 +111,24 @@
 # ===============================================================================
 
 
-def yaml_from_file(source_file: SchemaSourceFile) -> Dict[str, Any]:
+def yaml_from_file(source_file: SchemaSourceFile) -> Optional[Dict[str, Any]]:
     """If loading the yaml fails, raise an exception."""
     try:
         # source_file.contents can sometimes be None
-        return load_yaml_text(source_file.contents or "", source_file.path)
+        contents = load_yaml_text(source_file.contents or "", source_file.path)
+
+        if contents is None:
+            return contents
+
+        # When loaded_loaded_at_field is defined as None or null, it shows up in
+        # the dict but when it is not defined, it does not show up in the dict
+        # We need to capture this to be able to override source level settings later.
+        for source in contents.get("sources", []):
+            for table in source.get("tables", []):
+                if "loaded_at_field" in table:
+                    table["loaded_at_field_present"] = True
+
+        return contents
     except DbtValidationError as e:
         raise YamlLoadError(
             project_name=source_file.project_name, path=source_file.path.relative_path, exc=e
diff --git a/core/dbt/parser/sources.py b/core/dbt/parser/sources.py
--- a/core/dbt/parser/sources.py
+++ b/core/dbt/parser/sources.py
@@ -133,7 +133,14 @@ def parse_source(self, target: UnpatchedSourceDefinition) -> SourceDefinition:
         unique_id = target.unique_id
         description = table.description or ""
         source_description = source.description or ""
-        loaded_at_field = table.loaded_at_field or source.loaded_at_field
+
+        # We need to be able to tell the difference between explicitly setting the loaded_at_field to None/null
+        # and when it's simply not set.  This allows a user to override the source level loaded_at_field so that
+        # specific table can default to metadata-based freshness.
+        if table.loaded_at_field_present or table.loaded_at_field is not None:
+            loaded_at_field = table.loaded_at_field
+        else:
+            loaded_at_field = source.loaded_at_field  # may be None, that's okay
 
         freshness = merge_freshness(source.freshness, table.freshness)
         quoting = source.quoting.merged(table.quoting)


Test patch:
diff --git a/tests/functional/sources/test_source_loaded_at_field.py b/tests/functional/sources/test_source_loaded_at_field.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/sources/test_source_loaded_at_field.py
@@ -0,0 +1,136 @@
+import pytest
+from dbt.tests.util import run_dbt, get_manifest, write_file
+from dbt.exceptions import YamlParseDictError
+
+
+loaded_at_field_null_schema_yml = """
+sources:
+  - name: test_source
+    freshness:
+      warn_after:
+        count: 1
+        period: day
+      error_after:
+        count: 4
+        period: day
+    loaded_at_field: updated_at
+    tables:
+      - name: table1
+        loaded_at_field: null
+"""
+
+loaded_at_field_blank_schema_yml = """
+sources:
+  - name: test_source
+    freshness:
+      warn_after:
+        count: 1
+        period: day
+      error_after:
+        count: 4
+        period: day
+    loaded_at_field: updated_at
+    tables:
+      - name: table1
+        loaded_at_field: null
+"""
+
+loaded_at_field_missing_schema_yml = """
+sources:
+  - name: test_source
+    freshness:
+      warn_after:
+        count: 1
+        period: day
+      error_after:
+        count: 4
+        period: day
+    loaded_at_field: updated_at
+    tables:
+      - name: table1
+"""
+
+loaded_at_field_defined_schema_yml = """
+sources:
+  - name: test_source
+    freshness:
+      warn_after:
+        count: 1
+        period: day
+      error_after:
+        count: 4
+        period: day
+    loaded_at_field: updated_at
+    tables:
+      - name: table1
+        loaded_at_field: updated_at_another_place
+"""
+
+loaded_at_field_empty_string_schema_yml = """
+sources:
+  - name: test_source
+    freshness:
+      warn_after:
+        count: 1
+        period: day
+      error_after:
+        count: 4
+        period: day
+    loaded_at_field: updated_at
+    tables:
+      - name: table1
+        loaded_at_field: ""
+"""
+
+
+class TestParsingLoadedAtField:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {"schema.yml": loaded_at_field_null_schema_yml}
+
+    def test_loaded_at_field(self, project):
+        # test setting loaded_at_field to null explicitly at table level
+        run_dbt(["parse"])
+        manifest = get_manifest(project.project_root)
+
+        assert "source.test.test_source.table1" in manifest.sources
+        assert manifest.sources.get("source.test.test_source.table1").loaded_at_field is None
+
+        # test setting loaded_at_field at source level, do not set at table level
+        # end up with source level loaded_at_field
+        write_file(
+            loaded_at_field_missing_schema_yml, project.project_root, "models", "schema.yml"
+        )
+        run_dbt(["parse"])
+        manifest = get_manifest(project.project_root)
+        assert "source.test.test_source.table1" in manifest.sources
+        assert (
+            manifest.sources.get("source.test.test_source.table1").loaded_at_field == "updated_at"
+        )
+
+        # test setting loaded_at_field to nothing, should override Source value for None
+        write_file(loaded_at_field_blank_schema_yml, project.project_root, "models", "schema.yml")
+        run_dbt(["parse"])
+        manifest = get_manifest(project.project_root)
+
+        assert "source.test.test_source.table1" in manifest.sources
+        assert manifest.sources.get("source.test.test_source.table1").loaded_at_field is None
+
+        # test setting loaded_at_field at table level to a value - it should override source level
+        write_file(
+            loaded_at_field_defined_schema_yml, project.project_root, "models", "schema.yml"
+        )
+        run_dbt(["parse"])
+        manifest = get_manifest(project.project_root)
+        assert "source.test.test_source.table1" in manifest.sources
+        assert (
+            manifest.sources.get("source.test.test_source.table1").loaded_at_field
+            == "updated_at_another_place"
+        )
+
+        # test setting loaded_at_field at table level to an empty string - should error
+        write_file(
+            loaded_at_field_empty_string_schema_yml, project.project_root, "models", "schema.yml"
+        )
+        with pytest.raises(YamlParseDictError):
+            run_dbt(["parse"])