Repository: dbt-labs/dbt-core

Issue:
Move --full-refresh into a model config
## Feature

### Feature description
Presently, `--full-refresh` and `--non-destructive` can only be set on the CLI and apply to all models. Instead, these flags should be pushed down as model configs. The `--non-destructive` and `--full-refresh` flags can still exist, but they should set individual model configurations under the hood.

Since these configs will be specified with model-level granularity, complex behavior can be coded using variables, macros, etc.

### Who will this benefit?
Users with sophisticated dbt deployments who need to trigger complex dbt invocations

Via @elexisvenator in Slack:
> I want to run dbt as frequently as possible, which means ideally I only want to run models with data that has changed, or where a model has changed.
I already have the means to know which source tables have changed, and which model files have been modified since last run.

See also: https://github.com/fishtown-analytics/dbt/issues/320


Discussion:
Presently, flags are set at a global level in dbt. In `main.py`, dbt plucks configuration values out of the [parsed CLI arguments](https://github.com/fishtown-analytics/dbt/blob/a31745683b42524a5e5dfab28ac96dd117ea81e3/dbt/main.py#L253-L264) and sticks them into the global fields defined in [flags.py](https://github.com/fishtown-analytics/dbt/blob/dev/guion-bluford/dbt/flags.py). 

In the jinja layer, materializations use these flags to determine exactly what SQL should be run. There's an example of that over [here](https://github.com/fishtown-analytics/dbt/blob/a31745683b42524a5e5dfab28ac96dd117ea81e3/dbt/include/global_project/macros/materializations/table/table.sql#L5). 

A better approach is to [add new configuration options](https://github.com/fishtown-analytics/dbt/blob/dev/guion-bluford/dbt/utils.py#L39) (like non_destructive or full_refresh) for dbt resources, then use the global flags to set these options on the resources [during parsing](https://github.com/fishtown-analytics/dbt/blob/a31745683b42524a5e5dfab28ac96dd117ea81e3/dbt/parser/base.py#L100). Finally, the materialization code can be updated to reference an attribute of a model, like `model.full_refresh`, rather than pulling this value from the flags directly.

As a result of this change, the behavior of the `--full-refresh` flag will not change, but users can optionally supply a `full_refresh` configuration option to configure specific models based on the value of a variable, for instance.

This might be out of scope for _this_ issue, but I think we should deprecate (and eventually remove) the `--non-destructive` flag/concept entirely from dbt. That flag existed as a workaround for Redshift's bound views, which the incidence of late-bound views wholly obviates!

Resolving patch:
diff --git a/core/dbt/contracts/graph/model_config.py b/core/dbt/contracts/graph/model_config.py
--- a/core/dbt/contracts/graph/model_config.py
+++ b/core/dbt/contracts/graph/model_config.py
@@ -321,6 +321,7 @@ class NodeConfig(BaseConfig):
         # TODO: hide this one?
         metadata=MergeBehavior.Append.meta(),
     )
+    full_refresh: Optional[bool] = None
 
     @classmethod
     def from_dict(cls, data, validate=True):


Test patch:
diff --git a/test/integration/005_simple_seed_test/models-downstream-seed/model.sql b/test/integration/005_simple_seed_test/models-downstream-seed/model.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/005_simple_seed_test/models-downstream-seed/model.sql
@@ -0,0 +1 @@
+select * from {{ ref('seed_actual') }}
diff --git a/test/integration/005_simple_seed_test/test_simple_seed.py b/test/integration/005_simple_seed_test/test_simple_seed.py
--- a/test/integration/005_simple_seed_test/test_simple_seed.py
+++ b/test/integration/005_simple_seed_test/test_simple_seed.py
@@ -7,7 +7,6 @@ class TestSimpleSeed(DBTIntegrationTest):
 
     def setUp(self):
         DBTIntegrationTest.setUp(self)
-
         self.run_sql_file("seed.sql")
 
     @property
@@ -16,7 +15,7 @@ def schema(self):
 
     @property
     def models(self):
-        return "models"
+        return "models-downstream-seed"
 
     @property
     def project_config(self):
@@ -28,27 +27,61 @@ def project_config(self):
             }
         }
 
+    def use_full_refresh_project(self, full_refresh: bool):
+        overrides = {
+            'seeds': {
+                'quote_columns': False,
+                'full_refresh': full_refresh,
+            }
+        }
+        self.use_default_project(overrides)
+
+    def _seed_and_run(self):
+        assert len(self.run_dbt(['seed'])) == 1
+        self.assertTablesEqual('seed_actual', 'seed_expected')
+
+        assert len(self.run_dbt(['run'])) == 1
+        self.assertTablesEqual('model', 'seed_expected')
+
+    def _after_seed_model_state(self, cmd, exists: bool):
+        assert len(self.run_dbt(cmd)) == 1
+        self.assertTablesEqual('seed_actual', 'seed_expected')
+        if exists:
+            self.assertTableDoesExist('model')
+        else:
+            self.assertTableDoesNotExist('model')
+
     @use_profile('postgres')
     def test_postgres_simple_seed(self):
-        results = self.run_dbt(["seed"])
-        self.assertEqual(len(results),  1)
-        self.assertTablesEqual("seed_actual","seed_expected")
+        self._seed_and_run()
 
         # this should truncate the seed_actual table, then re-insert.
-        results = self.run_dbt(["seed"])
-        self.assertEqual(len(results),  1)
-        self.assertTablesEqual("seed_actual","seed_expected")
+        self._after_seed_model_state(['seed'], exists=True)
 
     @use_profile('postgres')
-    def test_postgres_simple_seed_with_drop(self):
-        results = self.run_dbt(["seed"])
-        self.assertEqual(len(results),  1)
-        self.assertTablesEqual("seed_actual","seed_expected")
+    def test_postgres_simple_seed_full_refresh_flag(self):
+        self._seed_and_run()
 
-        # this should drop the seed table, then re-create
-        results = self.run_dbt(["seed", "--full-refresh"])
-        self.assertEqual(len(results),  1)
-        self.assertTablesEqual("seed_actual","seed_expected")
+        # this should drop the seed_actual table, then re-create it, so the
+        # model won't exist.
+        self._after_seed_model_state(['seed', '--full-refresh'], exists=False)
+
+    @use_profile('postgres')
+    def test_postgres_simple_seed_full_refresh_config(self):
+        self._seed_and_run()
+
+        # set the full_refresh config to False
+        self.use_full_refresh_project(False)
+
+        self._after_seed_model_state(['seed'], exists=True)
+        # make sure we ignore the full-refresh flag (the config is higher
+        # priority than the flag)
+        self._after_seed_model_state(['seed', '--full-refresh'], exists=True)
+
+        # this should drop the seed_actual table, then re-create it, so the
+        # model won't exist.
+        self.use_full_refresh_project(True)
+        self._after_seed_model_state(['seed'], exists=False)
 
 
 class TestSimpleSeedCustomSchema(DBTIntegrationTest):
@@ -89,7 +122,6 @@ def test_postgres_simple_seed_with_schema(self):
         self.assertEqual(len(results),  1)
         self.assertTablesEqual("seed_actual","seed_expected", table_a_schema=schema_name)
 
-
     @use_profile('postgres')
     def test_postgres_simple_seed_with_drop_and_schema(self):
         schema_name = "{}_{}".format(self.unique_schema(), 'custom_schema')
diff --git a/test/integration/012_deprecation_tests/models/already_exists.sql b/test/integration/012_deprecation_tests/models/already_exists.sql
--- a/test/integration/012_deprecation_tests/models/already_exists.sql
+++ b/test/integration/012_deprecation_tests/models/already_exists.sql
@@ -1,5 +1,5 @@
 select 1 as id
 
-{% if adapter.already_exists(this.schema, this.identifier) and not flags.FULL_REFRESH %}
+{% if adapter.already_exists(this.schema, this.identifier) and not should_full_refresh() %}
 	where id > (select max(id) from {{this}})
 {% endif %}
diff --git a/test/integration/017_runtime_materialization_tests/test_runtime_materialization.py b/test/integration/017_runtime_materialization_tests/test_runtime_materialization.py
--- a/test/integration/017_runtime_materialization_tests/test_runtime_materialization.py
+++ b/test/integration/017_runtime_materialization_tests/test_runtime_materialization.py
@@ -25,10 +25,13 @@ def schema(self):
     def models(self):
         return "models"
 
+    def run_dbt_full_refresh(self):
+        return self.run_dbt(['run', '--full-refresh'])
+
     @use_profile('postgres')
     def test_postgres_full_refresh(self):
         # initial full-refresh should have no effect
-        results = self.run_dbt(['run', '--full-refresh'])
+        results = self.run_dbt_full_refresh()
         self.assertEqual(len(results), 3)
 
         self.assertTablesEqual("seed", "view")
@@ -37,13 +40,13 @@ def test_postgres_full_refresh(self):
 
         # adds one record to the incremental model. full-refresh should truncate then re-run
         self.run_sql_file("invalidate_incremental.sql")
-        results = self.run_dbt(['run', '--full-refresh'])
+        results = self.run_dbt_full_refresh()
         self.assertEqual(len(results), 3)
         self.assertTablesEqual("seed", "incremental")
 
         self.run_sql_file("update.sql")
 
-        results = self.run_dbt(['run', '--full-refresh'])
+        results = self.run_dbt_full_refresh()
         self.assertEqual(len(results), 3)
 
         self.assertTablesEqual("seed", "view")
@@ -59,3 +62,14 @@ def test_postgres_delete__dbt_tmp_relation(self):
 
         self.assertTableDoesNotExist('view__dbt_tmp')
         self.assertTablesEqual("seed", "view")
+
+
+class TestRuntimeMaterializationWithConfig(TestRuntimeMaterialization):
+    @property
+    def project_config(self):
+        result = super().project_config
+        result.update({'models': {'full_refresh': True}})
+        return result
+
+    def run_dbt_full_refresh(self):
+        return self.run_dbt(['run'])
diff --git a/test/integration/029_docs_generate_tests/test_docs_generate.py b/test/integration/029_docs_generate_tests/test_docs_generate.py
--- a/test/integration/029_docs_generate_tests/test_docs_generate.py
+++ b/test/integration/029_docs_generate_tests/test_docs_generate.py
@@ -898,6 +898,7 @@ def expected_seeded_manifest(self, model_database=None):
             'quoting': {},
             'tags': [],
             'persist_docs': {},
+            'full_refresh': None,
         }
 
         return {
@@ -983,6 +984,7 @@ def expected_seeded_manifest(self, model_database=None):
                         'quoting': {},
                         'tags': [],
                         'quote_columns': True,
+                        'full_refresh': None,
                     },
                     'patch_path': seed_schema_yml_path,
                     'path': 'seed.csv',
@@ -1064,6 +1066,7 @@ def expected_seeded_manifest(self, model_database=None):
                         'vars': {},
                         'tags': [],
                         'severity': 'ERROR',
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -1117,6 +1120,7 @@ def expected_seeded_manifest(self, model_database=None):
                         'vars': {},
                         'tags': [],
                         'severity': 'ERROR',
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -1169,6 +1173,7 @@ def expected_seeded_manifest(self, model_database=None):
                         'vars': {},
                         'tags': [],
                         'severity': 'ERROR',
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -1286,6 +1291,7 @@ def expected_postgres_references_manifest(self, model_database=None):
                         'quoting': {},
                         'vars': {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'sources': [['my_source', 'my_table']],
                     'depends_on': {
@@ -1347,6 +1353,7 @@ def expected_postgres_references_manifest(self, model_database=None):
                         'quoting': {},
                         'vars': {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -1410,6 +1417,7 @@ def expected_postgres_references_manifest(self, model_database=None):
                         'quoting': {},
                         'vars': {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'database': self.default_database,
                     'depends_on': {
@@ -1494,6 +1502,7 @@ def expected_postgres_references_manifest(self, model_database=None):
                         'vars': {},
                         'tags': [],
                         'quote_columns': True,
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {'macros': [], 'nodes': []},
@@ -1725,6 +1734,7 @@ def expected_bigquery_complex_manifest(self):
                         'quoting': {},
                         'vars': {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {'macros': [], 'nodes': ['seed.test.seed']},
@@ -1804,6 +1814,7 @@ def expected_bigquery_complex_manifest(self):
                         'quoting': {},
                         'tags': [],
                         'vars': {},
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {'macros': [], 'nodes': ['seed.test.seed']},
@@ -1880,6 +1891,7 @@ def expected_bigquery_complex_manifest(self):
                         'quoting': {},
                         'vars':  {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -1959,6 +1971,7 @@ def expected_bigquery_complex_manifest(self):
                         'quoting': {},
                         'vars': {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -2020,6 +2033,7 @@ def expected_bigquery_complex_manifest(self):
                         'quoting': {},
                         'tags': [],
                         'quote_columns': True,
+                        'full_refresh': None,
                     },
                     'schema': my_schema_name,
                     'database': self.default_database,
@@ -2160,6 +2174,7 @@ def expected_redshift_incremental_view_manifest(self):
                         'quoting': {},
                         'tags': [],
                         'vars': {},
+                        'full_refresh': None,
                     },
                     'schema': my_schema_name,
                     'database': self.default_database,
@@ -2241,6 +2256,7 @@ def expected_redshift_incremental_view_manifest(self):
                         'tags': [],
                         'vars': {},
                         'quote_columns': True,
+                        'full_refresh': None,
                     },
                     'schema': my_schema_name,
                     'database': self.default_database,
@@ -2364,6 +2380,7 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
             'column_types': {},
             'quoting': {},
             'tags': [],
+            'full_refresh': None,
         }
         schema = self.unique_schema()
 
@@ -2523,6 +2540,7 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
                         'vars': {},
                         'tags': [],
                         'quote_columns': True,
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {'macros': [], 'nodes': []},
@@ -2575,6 +2593,7 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
                         'vars': {},
                         'tags': [],
                         'severity': 'ERROR',
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -2638,6 +2657,7 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
                         'vars': {},
                         'tags': [],
                         'severity': 'ERROR',
+                        'full_refresh': None,
                     },
                     'database': self.default_database,
                     'depends_on': {
@@ -2700,6 +2720,7 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
                         'vars': {},
                         'tags': [],
                         'severity': 'ERROR',
+                        'full_refresh': None,
                     },
                     'database': self.default_database,
                     'depends_on': {
@@ -2804,6 +2825,7 @@ def expected_postgres_references_run_results(self):
                         'column_types': {},
                         'quoting': {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -2884,6 +2906,7 @@ def expected_postgres_references_run_results(self):
                         'column_types': {},
                         'quoting': {},
                         'tags': [],
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {
@@ -2981,6 +3004,7 @@ def expected_postgres_references_run_results(self):
                         'vars': {},
                         'tags': [],
                         'quote_columns': True,
+                        'full_refresh': None,
                     },
                     'sources': [],
                     'depends_on': {'macros': [], 'nodes': []},
diff --git a/test/integration/047_dbt_ls_test/test_ls.py b/test/integration/047_dbt_ls_test/test_ls.py
--- a/test/integration/047_dbt_ls_test/test_ls.py
+++ b/test/integration/047_dbt_ls_test/test_ls.py
@@ -83,7 +83,8 @@ def expect_snapshot_output(self):
                     'target_schema': self.unique_schema(),
                     'unique_key': 'id',
                     'strategy': 'timestamp',
-                    'updated_at': 'updated_at'
+                    'updated_at': 'updated_at',
+                    'full_refresh': None,
                 },
                 'alias': 'my_snapshot',
                 'resource_type': 'snapshot',
@@ -111,6 +112,7 @@ def expect_analyses_output(self):
                     'vars': {},
                     'column_types': {},
                     'persist_docs': {},
+                    'full_refresh': None,
                 },
                 'alias': 'a',
                 'resource_type': 'analysis',
@@ -139,6 +141,7 @@ def expect_model_output(self):
                         'vars': {},
                         'column_types': {},
                         'persist_docs': {},
+                        'full_refresh': None,
                     },
                     'alias': 'ephemeral',
                     'resource_type': 'model',
@@ -158,6 +161,7 @@ def expect_model_output(self):
                         'vars': {},
                         'column_types': {},
                         'persist_docs': {},
+                        'full_refresh': None,
                     },
                     'alias': 'inner',
                     'resource_type': 'model',
@@ -177,6 +181,7 @@ def expect_model_output(self):
                         'vars': {},
                         'column_types': {},
                         'persist_docs': {},
+                        'full_refresh': None,
                     },
                     'alias': 'outer',
                     'resource_type': 'model',
@@ -207,6 +212,7 @@ def expect_model_ephemeral_output(self):
                         'vars': {},
                         'column_types': {},
                         'persist_docs': {},
+                        'full_refresh': None,
                     },
                     'alias': 'outer',
                     'resource_type': 'model',
@@ -256,6 +262,7 @@ def expect_seed_output(self):
                     'column_types': {},
                     'persist_docs': {},
                     'quote_columns': False,
+                    'full_refresh': None,
                 },
                 'alias': 'seed',
                 'resource_type': 'seed',
@@ -285,6 +292,7 @@ def expect_test_output(self):
                         'vars': {},
                         'column_types': {},
                         'persist_docs': {},
+                        'full_refresh': None,
                     },
                     'alias': 'not_null_outer_id',
                     'resource_type': 'test',
@@ -306,6 +314,7 @@ def expect_test_output(self):
                         'vars': {},
                         'column_types': {},
                         'persist_docs': {},
+                        'full_refresh': None,
                     },
                     'alias': 't',
                     'resource_type': 'test',
@@ -326,6 +335,7 @@ def expect_test_output(self):
                         'vars': {},
                         'column_types': {},
                         'persist_docs': {},
+                        'full_refresh': None,
                     },
                     'alias': 'unique_outer_id',
                     'resource_type': 'test',