Repository: TobikoData/sqlmesh
Pull request #2630 (2024-05-17 19:03:12)

Fix: Include virtual properties into the metadata hash
Without this changes to the `virtual_properties` attribute are not picked up by the `plan` command.


Patch:
diff --git a/docs/reference/model_configuration.md b/docs/reference/model_configuration.md
index cb59be11..01a270e7 100644
--- a/docs/reference/model_configuration.md
+++ b/docs/reference/model_configuration.md
@@ -8,30 +8,30 @@ Learn more about specifying SQLMesh model properties in the [model concepts over
 
 Configuration options for SQLMesh model properties. Supported by all model kinds other than [`SEED` models](#seed-models).
 
-| Option                | Description                                                                                                                                                                                                                                                                                 | Type              | Required |
-|-----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------:|:--------:|
-| `name`                | The model name. Must include at least a qualifying schema (`<schema>.<model>`) and may include a catalog (`<catalog>.<schema>.<model>`). If any project model name includes a catalog, all model names must include a catalog or a default catalog [must be set](#default-schema--catalog). | str               | Y        |
-| `kind`                | The model kind ([Additional Details](#model-kind-properties)) (Default: `VIEW`)                                                                                                                                                                                                             | str \| dict       | N        |
-| `audits`              | SQLMesh [audits](../concepts/audits.md) that should run against the model's output                                                                                                                                                                                                          | array[str]        | N        |
-| `dialect`             | The SQL dialect in which the model's query is written. All SQL dialects [supported by the SQLGlot library](https://github.com/tobymao/sqlglot/blob/main/sqlglot/dialects/dialect.py) are allowed.                                                                                           | str               | N        |
-| `owner`               | The owner of a model; may be used for notification purposes                                                                                                                                                                                                                                 | str               | N        |
-| `stamp`               | Arbitrary string used to indicate a model's version without changing the model name                                                                                                                                                                                                         | str               | N        |
-| `tags`                | Arbitrary strings used to organize or classify a model                                                                                                                                                                                                                                      | array[str]        | N        |
-| `cron`                | The cron expression specifying how often the model should be refreshed. (Default: `@daily`)                                                                                                                                                                                                 | str               | N        |
-| `interval_unit`       | The temporal granularity of the model's data intervals. Supported values: `year`, `month`, `day`, `hour`, `half_hour`, `quarter_hour`, `five_minute`. (Default: inferred from `cron`)                                                                                                       | str               | N        |
-| `start`               | The date/time that determines the earliest date interval that should be processed by a model. Can be a datetime string, epoch time in milliseconds, or a relative datetime such as `1 year ago`.                                                                                            | str \| int        | N        |
-| `end`                 | The date/time that determines the latest date interval that should be processed by a model. Can be a datetime string, epoch time in milliseconds, or a relative datetime such as `1 year ago`.                                                                                              | str \| int        | N        |
-| `grains`              | The column(s) whose combination uniquely identifies each row in the model                                                                                                                                                                                                                   | str \| array[str] | N        |
-| `references`          | The model column(s) used to join to other models' grains                                                                                                                                                                                                                                    | str \| array[str] | N        |
-| `depends_on`          | Models on which this model depends. (Default: dependencies inferred from model code)                                                                                                                                                                                                        | array[str]        | N        |
-| `storage_format`      | The storage format that should be used to store physical tables; only applicable to engines such as Spark                                                                                                                                                                                   | str               | N        |
-| `partitioned_by`      | The column(s) used to partition the model's physical table; only applicable to engines that support partitioning                                                                                                                                                                            | str               | N        |
-| `clustered_by`        | The column(s) used to cluster the model's physical table; only applicable to engines that support clustering                                                                                                                                                                                | str               | N        |
-| `columns`             | The column names and data types returned by the model. Disables [automatic inference of column names and types](../concepts/models/overview.md#conventions) from the SQL query.                                                                                                             | array[str]        | N        |
-| `physical_properties` | A key-value mapping of arbitrary properties specific to the target engine that are applied to the model table / view in the physical layer. Specified as key-value pairs (`key = value`)                                                                                                    | dict              | N        |
-| `virtual_properties`  | A key-value mapping of arbitrary properties specific to the target engine that are applied to the model view in the virtual layer. Specified as key-value pairs (`key = value`)                                                                                                             | dict              | N        |
-| `allow_partials`      | Whether this model can process partial (incomplete) data intervals                                                                                                                                                                                                                          | bool              | N        |
-| `description`         | Description of the model. Automatically registered in the SQL engine's table COMMENT field or equivalent (if supported by the engine).                                                                                                                                                      | str               | N        |
+| Option                | Description                                                                                                                                                                                                             |       Type        | Required |
+| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------: | :------: |
+| `name`                | The model name. Must include at least a qualifying schema (`<schema>.<model>`) and may include a catalog (`<catalog>.<schema>.<model>`).                                                                                |        str        |    Y     |
+| `kind`                | The model kind ([Additional Details](#model-kind-properties)). (Default: `VIEW`)                                                                                                                                        |    str \| dict    |    N     |
+| `audits`              | SQLMesh [audits](../concepts/audits.md) that should run against the model's output                                                                                                                                      |    array[str]     |    N     |
+| `dialect`             | The SQL dialect in which the model's query is written. All SQL dialects [supported by the SQLGlot library](https://github.com/tobymao/sqlglot/blob/main/sqlglot/dialects/dialect.py) are allowed.                       |        str        |    N     |
+| `owner`               | The owner of a model; may be used for notification purposes                                                                                                                                                             |        str        |    N     |
+| `stamp`               | Arbitrary string used to indicate a model's version without changing the model name                                                                                                                                     |        str        |    N     |
+| `tags`                | Arbitrary strings used to organize or classify a model                                                                                                                                                                  |    array[str]     |    N     |
+| `cron`                | The cron expression specifying how often the model should be refreshed. (Default: `@daily`)                                                                                                                             |        str        |    N     |
+| `interval_unit`       | The temporal granularity of the model's data intervals. Supported values: `year`, `month`, `day`, `hour`, `half_hour`, `quarter_hour`, `five_minute`. (Default: inferred from `cron`)                                   |        str        |    N     |
+| `start`               | The date/time that determines the earliest date interval that should be processed by a model. Can be a datetime string, epoch time in milliseconds, or a relative datetime such as `1 year ago`. (Default: `yesterday`) |    str \| int     |    N     |
+| `end`                 | The date/time that determines the latest date interval that should be processed by a model. Can be a datetime string, epoch time in milliseconds, or a relative datetime such as `1 year ago`.                          |    str \| int     |    N     |
+| `grains`              | The column(s) whose combination uniquely identifies each row in the model                                                                                                                                               | str \| array[str] |    N     |
+| `references`          | The model column(s) used to join to other models' grains                                                                                                                                                                | str \| array[str] |    N     |
+| `depends_on`          | Models on which this model depends. (Default: dependencies inferred from model code)                                                                                                                                    |    array[str]     |    N     |
+| `storage_format`      | The storage format that should be used to store physical tables; only applicable to engines such as Spark                                                                                                               |        str        |    N     |
+| `partitioned_by`      | The column(s) used to partition the model's physical table; only applicable to engines that support partitioning                                                                                                        |        str        |    N     |
+| `clustered_by`        | The column(s) used to cluster the model's physical table; only applicable to engines that support clustering                                                                                                            |        str        |    N     |
+| `columns`             | The column names and data types returned by the model. Disables [automatic inference of column names and types](../concepts/models/overview.md#conventions) from the SQL query.                                         |    array[str]     |    N     |
+| `physical_properties` | A key-value mapping of arbitrary properties specific to the target engine that are applied to the model table / view in the physical layer. Specified as key-value pairs (`key = value`).                               |       dict        |    N     |
+| `virtual_properties`  | A key-value mapping of arbitrary properties specific to the target engine that are applied to the model view in the virtual layer. Specified as key-value pairs (`key = value`).                                        |       dict        |    N     |
+| `allow_partials`      | Whether this model can process partial (incomplete) data intervals                                                                                                                                                      |       bool        |    N     |
+| `description`         | Description of the model. Automatically registered in the SQL engine's table COMMENT field or equivalent (if supported by the engine).                                                                                  |        str        |    N     |
 
 ### Model defaults
 
@@ -74,10 +74,10 @@ Python model kind `name` enum value: [ModelKindName.FULL](https://sqlmesh.readth
 Configuration options for all incremental models (in addition to [general model properties](#general-model-properties)).
 
 | Option              | Description                                                                                                                                                                                                                                                                                                                      | Type | Required |
-|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----:|:--------:|
-| `batch_size`        | The maximum number of intervals that can be evaluated in a single backfill task. If this is `None`, all intervals will be processed as part of a single task. If this is set, a model's backfill will be chunked such that each individual task only contains jobs with the maximum of `batch_size` intervals. (Default: `None`) | int  | N        |
-| `batch_concurrency` | The maximum number of batches that can run concurrently for this model (Default: the number of concurrent tasks set in the connection settings).                                                                                                                                                                                 | int  | N        |
-| `lookback`          | The number of time unit intervals prior to the current interval that should be processed. (Default: `0`)                                                                                                                                                                                                                         | int  | N        |
+| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--: | :------: |
+| `batch_size`        | The maximum number of intervals that can be evaluated in a single backfill task. If this is `None`, all intervals will be processed as part of a single task. If this is set, a model's backfill will be chunked such that each individual task only contains jobs with the maximum of `batch_size` intervals. (Default: `None`) | int  |    N     |
+| `batch_concurrency` | The maximum number of batches that can run concurrently for this model. (Default: the number of concurrent tasks set in the connection settings)                                                                                                                                                                                 | int  |    N     |
+| `lookback`          | The number of time unit intervals prior to the current interval that should be processed. (Default: `0`)                                                                                                                                                                                                                         | int  |    N     |
 
 #### Incremental by time range
 
@@ -111,7 +111,7 @@ Configuration options for [`SCD_TYPE_2` models](../concepts/models/model_kinds.m
 
 | Option                    | Description                                                                                                                                                                                 |   Type    | Required |
 | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------: | :------: |
-| `unique_key`              | The model column(s) containing each row's unique key.                                                                                                                                       | list[str] |    Y     |
+| `unique_key`              | The model column(s) containing each row's unique key                                                                                                                                        | list[str] |    Y     |
 | `valid_from_name`         | The model column containing each row's valid from date. (Default: `valid_from`)                                                                                                             |    str    |    N     |
 | `valid_to_name`           | The model column containing each row's valid to date. (Default: `valid_to`)                                                                                                                 |    str    |    N     |
 | `invalidate_hard_deletes` | If set to true, when a record is missing from the source table it will be marked as invalid - see [here](../concepts/models/model_kinds.md#deletes) for more information. (Default: `True`) |   bool    |    N     |
@@ -144,16 +144,16 @@ Configuration options for [`SEED` models](../concepts/models/model_kinds.md#seed
 
 Top-level options inside the MODEL DDL:
 
-| Option        | Description                                                                                                                                                                                                                                                                                 |    Type    | Required |
-| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------: | :------: |
-| `name`        | The model name. Must include at least a qualifying schema (`<schema>.<model>`) and may include a catalog (`<catalog>.<schema>.<model>`). If any project model name includes a catalog, all model names must include a catalog or a default catalog [must be set](#default-schema--catalog). |    str     |    Y     |
-| `kind`        | The model kind. Must be `SEED`.                                                                                                                                                                                                                                                             |    str     |    Y     |
-| `columns`     | The column names and data types in the CSV file. Disables automatic inference of column names and types by the pandas CSV reader. NOTE: order of columns overrides the order specified in the CSV header row (if present).                                                                  | array[str] |    N     |
-| `audits`      | SQLMesh [audits](../concepts/audits.md) that should run against the model's output                                                                                                                                                                                                          | array[str] |    N     |
-| `owner`       | The owner of a model; may be used for notification purposes                                                                                                                                                                                                                                 |    str     |    N     |
-| `stamp`       | Arbitrary string used to indicate a model's version without changing the model name                                                                                                                                                                                                         |    str     |    N     |
-| `tags`        | Arbitrary strings used to organize or classify a model                                                                                                                                                                                                                                      | array[str] |    N     |
-| `description` | Description of the model. Automatically registered in the SQL engine's table COMMENT field or equivalent (if supported by the engine).                                                                                                                                                      |    str     |    N     |
+| Option        | Description                                                                                                                                                                                                                |    Type    | Required |
+| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------: | :------: |
+| `name`        | The model name. Must include at least a qualifying schema (`<schema>.<model>`) and may include a catalog (`<catalog>.<schema>.<model>`).                                                                                   |    str     |    Y     |
+| `kind`        | The model kind. Must be `SEED`.                                                                                                                                                                                            |    str     |    Y     |
+| `columns`     | The column names and data types in the CSV file. Disables automatic inference of column names and types by the pandas CSV reader. NOTE: order of columns overrides the order specified in the CSV header row (if present). | array[str] |    N     |
+| `audits`      | SQLMesh [audits](../concepts/audits.md) that should run against the model's output                                                                                                                                         | array[str] |    N     |
+| `owner`       | The owner of a model; may be used for notification purposes                                                                                                                                                                |    str     |    N     |
+| `stamp`       | Arbitrary string used to indicate a model's version without changing the model name                                                                                                                                        |    str     |    N     |
+| `tags`        | Arbitrary strings used to organize or classify a model                                                                                                                                                                     | array[str] |    N     |
+| `description` | Description of the model. Automatically registered in the SQL engine's table COMMENT field or equivalent (if supported by the engine).                                                                                     |    str     |    N     |
 
 Options specified within the top-level `kind` property:
 
diff --git a/sqlmesh/core/model/definition.py b/sqlmesh/core/model/definition.py
index b7dc314b..0e165748 100644
--- a/sqlmesh/core/model/definition.py
+++ b/sqlmesh/core/model/definition.py
@@ -778,6 +778,10 @@ class _Model(ModelMeta, frozen=True):
             else:
                 raise SQLMeshError(f"Unexpected audit name '{audit_name}'.")
 
+        for key, value in (self.virtual_properties or {}).items():
+            metadata.append(key)
+            metadata.append(gen(value))
+
         metadata.extend(gen(s) for s in self.signals)
         metadata.extend(self._additional_metadata)
 


Test patch:
diff --git a/tests/core/test_snapshot.py b/tests/core/test_snapshot.py
index ddcc3d8d..4c3a1801 100644
--- a/tests/core/test_snapshot.py
+++ b/tests/core/test_snapshot.py
@@ -30,7 +30,7 @@ from sqlmesh.core.model import (
     create_seed_model,
     load_sql_based_model,
 )
-from sqlmesh.core.model.kind import TimeColumn
+from sqlmesh.core.model.kind import TimeColumn, ModelKindName
 from sqlmesh.core.snapshot import (
     DeployabilityIndex,
     QualifiedViewName,
@@ -54,7 +54,7 @@ from sqlmesh.utils.jinja import JinjaMacroRegistry, MacroInfo
 def parent_model():
     return SqlModel(
         name="parent.tbl",
-        kind=IncrementalByTimeRangeKind(time_column="ds"),
+        kind=dict(time_column="ds", name=ModelKindName.INCREMENTAL_BY_TIME_RANGE),
         dialect="spark",
         query=parse_one("SELECT 1, ds"),
     )
@@ -64,7 +64,7 @@ def parent_model():
 def model():
     return SqlModel(
         name="name",
-        kind=IncrementalByTimeRangeKind(time_column="ds", batch_size=30),
+        kind=dict(time_column="ds", batch_size=30, name=ModelKindName.INCREMENTAL_BY_TIME_RANGE),
         owner="owner",
         dialect="spark",
         cron="1 0 * * *",
@@ -105,10 +105,11 @@ def test_json(snapshot: Snapshot):
             "cron": "1 0 * * *",
             "kind": {
                 "name": "INCREMENTAL_BY_TIME_RANGE",
-                "time_column": {"column": '"ds"'},
+                "time_column": {"column": "`ds`"},
                 "batch_size": 30,
                 "forward_only": False,
                 "disable_restatement": False,
+                "dialect": "spark",
             },
             "mapping_schema": {},
             "start": "2020-01-01",
@@ -630,7 +631,7 @@ def test_fingerprint(model: Model, parent_model: Model):
     fingerprint = fingerprint_from_node(model, nodes={})
 
     original_fingerprint = SnapshotFingerprint(
-        data_hash="3163676913",
+        data_hash="3582214120",
         metadata_hash="892368116",
     )
 
@@ -788,6 +789,22 @@ def test_fingerprint_standalone_audits(parent_model: Model):
     assert new_fingerprint.metadata_hash != fingerprint.metadata_hash
 
 
+def test_fingerprint_virtual_properties(model: Model, parent_model: Model):
+    original_model = deepcopy(model)
+    fingerprint = fingerprint_from_node(model, nodes={})
+
+    updated_model = SqlModel(
+        **original_model.dict(),
+        virtual_properties=parse_one("(labels = [('test-virtual-label', 'label-virtual-value')],)"),
+    )
+    assert "labels" in updated_model.virtual_properties
+    updated_fingerprint = fingerprint_from_node(updated_model, nodes={})
+
+    assert updated_fingerprint != fingerprint
+    assert updated_fingerprint.metadata_hash != fingerprint.metadata_hash
+    assert updated_fingerprint.data_hash == fingerprint.data_hash
+
+
 def test_stamp(model: Model):
     original_fingerprint = fingerprint_from_node(model, nodes={})