Repository: dbt-labs/dbt-core

Issue:
Remove subquery from incremental models when no sql_where is specified
## Issue

### Issue description

Redshift Spectrum does not support (some?) nested queries. This includes [a subquery](https://github.com/fishtown-analytics/dbt/blob/v0.12.2/dbt/include/global_project/macros/materializations/incremental/incremental.sql#L61) used in the default incremental materialization.

The generated simple query for my incremental model looks like this:

```sql
  select
      "..." as surrogate_key
    , "etl-date" as etl_date
    , ...
 from spectrum.table
where "etl-date" < to_char(current_date, 'YYYY-MM-DD')
  and "etl_date" > coalesce((select max("etl_date") from "dev"."raw_table"), '0000-00-00')
```

With configuration in `dbt_project.yml` like so:

```yaml
    raw:
      materialized: incremental
      dist: surrogate_key
      sort: ['etl_date']
```

### Results

On runs with `--full-refresh` specified, the model and resulting incremental table is created and selected into as expected. On subsequent runs without `--full-refresh` Redshift complains that the nested query is not allowed. From `logs/dbt.log`:

```
Database Error in model raw_activity (models/raw/raw_activity.sql)
  Spectrum nested query error
  DETAIL:
    -----------------------------------------------
    error:  Spectrum nested query error
    code:      8001
    context:   Expression is not supported by Spectrum.
    query:     0
    location:  nested_query_rewriter.cpp:683
    process:   padbmaster [pid=7955]
```

### Workaround

The nested query in question is on [L61](https://github.com/fishtown-analytics/dbt/blob/v0.12.2/dbt/include/global_project/macros/materializations/incremental/incremental.sql#L61) of the incremental materialization.

```sql
       {% set tmp_table_sql -%}
         {# We are using a subselect instead of a CTE here to allow PostgreSQL to use indexes. -#}
         select * from (
           {{ sql }}
         ) as dbt_incr_sbq

         {% if sql_where %}
         where ({{ sql_where }})
           or ({{ sql_where }}) is null
         {% endif %}
       {%- endset %}
```

The subquery as `dbt_incr_sbq` appears unused in the case where `sql_where` is not present. If `sql_where` is not used, this issue can be worked around by creating a custom materialization that replaces the above with simply:

```sql
       {% set tmp_table_sql -%}
         {{ sql }}
       {%- endset %}
```

### System information

```
(venv) $ dbt --version
installed version: 0.12.2
   latest version: 0.12.2

Up to date!
(venv) $ python --version
Python 2.7.10
(venv) $ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.3
BuildVersion:	18D42
```



Discussion:
Thanks for the report @ianterrell! I think your workaround sounds like a reasonable one, though the best solution is probably to omit the `sql_where` config in favor of using the new [is_incremental()](https://docs.getdbt.com/docs/configuring-incremental-models#section-filtering-rows-on-an-incremental-run) pattern.

Can you try omitting the `sql_where` flag in favor of the `is_incremental()` pattern, then try running the model incrementally? That should remove the invalid spectrum subquery pattern

Hi @drewbanin!

My queries were already written with `is_incremental()` and had omitted the `sql_where` flag. 

> That should remove the invalid spectrum subquery pattern

It appears to me that the [`dbt_incr_sbq` subquery](https://github.com/fishtown-analytics/dbt/blob/v0.12.2/dbt/include/global_project/macros/materializations/incremental/incremental.sql#L61-L63) is generated regardless, and it's only the subsequent where clause that is dependent on the `sql_where` flag.
Got it - thanks @ianterrell. Let's remove the subquery if no `sql_where` is specified. Thanks for the additional info!

Resolving patch:
diff --git a/core/dbt/deprecations.py b/core/dbt/deprecations.py
--- a/core/dbt/deprecations.py
+++ b/core/dbt/deprecations.py
@@ -30,16 +30,6 @@ class DBTRepositoriesDeprecation(DBTDeprecation):
   """
 
 
-class SqlWhereDeprecation(DBTDeprecation):
-    name = "sql_where"
-    description = """\
-The `sql_where` option for incremental models is deprecated and will be
-  removed in a future release. Check the docs for more information
-
-  {}
-  """.format(dbt.links.IncrementalDocs)
-
-
 class SeedDropExistingDeprecation(DBTDeprecation):
     name = 'drop-existing'
     description = """The --drop-existing argument to `dbt seed` has been
@@ -83,7 +73,6 @@ def warn(name, *args, **kwargs):
 deprecations_list = [
     DBTRepositoriesDeprecation(),
     SeedDropExistingDeprecation(),
-    SqlWhereDeprecation(),
 ]
 
 deprecations = {d.name: d for d in deprecations_list}
diff --git a/core/dbt/loader.py b/core/dbt/loader.py
--- a/core/dbt/loader.py
+++ b/core/dbt/loader.py
@@ -214,8 +214,6 @@ def _warn_for_unused_resource_config_paths(manifest, config):
 def _warn_for_deprecated_configs(manifest):
     for unique_id, node in manifest.nodes.items():
         is_model = node.resource_type == NodeType.Model
-        if is_model and 'sql_where' in node.config:
-            deprecations.warn('sql_where')
 
 
 def _check_manifest(manifest, config):
diff --git a/core/dbt/parser/source_config.py b/core/dbt/parser/source_config.py
--- a/core/dbt/parser/source_config.py
+++ b/core/dbt/parser/source_config.py
@@ -13,7 +13,6 @@ class SourceConfig(object):
         'schema',
         'enabled',
         'materialized',
-        'sql_where',
         'unique_key',
         'database',
     }


Test patch:
diff --git a/test/integration/012_deprecation_tests/models/sql_where.sql b/test/integration/012_deprecation_tests/models/sql_where.sql
deleted file mode 100644
--- a/test/integration/012_deprecation_tests/models/sql_where.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-{{ config(sql_where='id > (select max(id) from {{this}})')}}
-
-select 1 as id
diff --git a/test/integration/012_deprecation_tests/test_deprecations.py b/test/integration/012_deprecation_tests/test_deprecations.py
--- a/test/integration/012_deprecation_tests/test_deprecations.py
+++ b/test/integration/012_deprecation_tests/test_deprecations.py
@@ -29,6 +29,6 @@ def test_postgres_deprecations_fail(self):
     @use_profile('postgres')
     def test_postgres_deprecations(self):
         self.assertEqual(deprecations.active_deprecations, set())
-        results = self.run_dbt(strict=False)
-        self.assertEqual({'adapter:already_exists', 'sql_where'},
+        self.run_dbt(strict=False)
+        self.assertEqual({'adapter:already_exists'},
                          deprecations.active_deprecations)
diff --git a/test/unit/test_config.py b/test/unit/test_config.py
--- a/test/unit/test_config.py
+++ b/test/unit/test_config.py
@@ -42,7 +42,6 @@ def temp_cd(path):
                 'sort': 'timestamp',
                 'materialized': 'incremental',
                 'dist': 'user_id',
-                'sql_where': 'created_at > (select max(created_at) from {{ this }})',
                 'unique_key': 'id'
             },
             'base': {
diff --git a/test/unit/test_graph.py b/test/unit/test_graph.py
--- a/test/unit/test_graph.py
+++ b/test/unit/test_graph.py
@@ -208,7 +208,6 @@ def test__model_incremental(self):
                 "test_models_compile": {
                     "model_one": {
                         "materialized": "incremental",
-                        "sql_where": "created_at",
                         "unique_key": "id"
                     },
                 }