Repository: dbt-labs/dbt-core

Issue:
issues with concurrent transaction
The proper way to atomically replace a view/table in Postgres/Redshift is:

1. begin transaction
2. rename model to "model_old"
3. rename new model to "model'"
4. commit
5. drop table "model_old"

Currently, dbt drops the table inside of the transaction, which can lead to errors like:

```
ERROR: table 12345 dropped by concurrent transaction
```

or 

```
could not complete because of conflict with concurrent transaction
```

Via @teej
See also: [this SO post](https://stackoverflow.com/questions/20692518/how-can-i-ensure-synchronous-ddl-operations-on-a-table-that-is-being-replaced)

To fix this, dbt's [table](https://github.com/fishtown-analytics/dbt/blob/development/dbt/include/global_project/macros/materializations/table.sql) and [view](https://github.com/fishtown-analytics/dbt/blob/development/dbt/include/global_project/macros/materializations/view.sql) materializations need to be updated to drop the old model _outside_ of the transaction.

This PR will ultimately require < 5 lines of code change, but since it involves the table and view materializations, we should be very cautious here. Is there a good way to write an integration test for this?


Discussion:
You might be able to simulate this behavior by:

* First, run dbt and build a model.
* Second, start a second dbt run which builds a model in a thread
* Finally, in a concurrent thread, attempt to access the results of the model.

However I think the model would have to take long enough to build so as to ensure that the test doesn't sporadically pass. 

You can force this condition to trigger by adding a sleep in the model's SQL. You can sleep in Redshift with:

```SQL
CREATE OR REPLACE FUNCTION f_sleep (x float)
RETURNS bool IMMUTABLE
AS
$$
  from time import sleep
  sleep(x)
  return True
$$ LANGUAGE plpythonu;
```
oooh nice idea @teej 
We only started seeing this issue when updating from `0.8.0` to `0.9.1`.

Is there any workaround? And, do we know which version introduced this issue?
@igrayson we overhauled how model materializations work in 0.9.0, but it looks like the core SQL logic was unchanged.

Ultimately, this is going to be a small code change, but it's a little daunting to make because Redshift's transaction management is pretty finicky.

Happy to point someone in the right direction if they're interested in tackling this, and if not, I think we'll be able to hit it pretty soon. @igrayson @teej @abelsonlive are you folks able to help test an alpha version of this fix when one is ready? That would go a long way towards making me feel good about any prospective fix.
@drewbanin can definitely help test anything you put out... have been a bit swamped but might still take a stab at this if my schedule clears up


thanks @abelsonlive -- we're focused on getting 0.9.2 out, we'll tackle this one for 0.9.3
Sounds good - as discussed on the Slack channel, this could potentially also fix an issue I've been having where DBT is forced to wait indefinitely due to Redshift failing to execute some statements (e.g. `DROP VIEW`) within a transaction block. When I check the `stv_recents` Redshift table, the query is left in a 'running' state and doesn't complete.
This change is going to apply to Redshift / Postgres / Snowflake

Resolving patch:
diff --git a/dbt/adapters/default/impl.py b/dbt/adapters/default/impl.py
--- a/dbt/adapters/default/impl.py
+++ b/dbt/adapters/default/impl.py
@@ -162,7 +162,8 @@ def drop_relation(cls, profile, project_cfg, relation, model_name=None):
 
         sql = 'drop {} if exists {} cascade'.format(relation.type, relation)
 
-        connection, cursor = cls.add_query(profile, sql, model_name)
+        connection, cursor = cls.add_query(profile, sql, model_name,
+                                           auto_begin=False)
 
     @classmethod
     def truncate(cls, profile, project_cfg, schema, table, model_name=None):


Test patch:
diff --git a/test.env.sample b/test.env.sample
--- a/test.env.sample
+++ b/test.env.sample
@@ -15,3 +15,9 @@ BIGQUERY_AUTH_URI=
 BIGQUERY_TOKEN_URI=
 BIGQUERY_AUTH_PROVIDER_X509_CERT_URL=
 BIGQUERY_CLIENT_X509_CERT_URL=
+
+REDSHIFT_TEST_HOST=
+REDSHIFT_TEST_USER=
+REDSHIFT_TEST_PASSWORD=
+REDSHIFT_TEST_PORT=
+REDSHIFT_TEST_DBNAME=
diff --git a/test/integration/032_concurrent_transaction_test/README.md b/test/integration/032_concurrent_transaction_test/README.md
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/README.md
@@ -0,0 +1,33 @@
+
+This test warrants some explanation. In dbt <=0.10.1, Redshift table and view materializations suffered from issues around concurrent transactions. In order to reliably reproduce this error, a query needs to select from a dbt model as the table is being rebuilt. Critically, this concurrent select needs to query the table during the drop/swap portition of the materialization. This looks like:
+
+```sql
+begin;
+create table as (...);
+drop table old_table cascade;
+// <---- The concurrent query needs to be running here!
+alter table new_table rename to old_table;
+commit;
+```
+
+In order to reliably reproduce this failure, the model shown above needs to block for a long time between the `drop` and `alter` statements. We can't just stick a sleep() call in there, as this code is defined in the materialization. Instead, we can reliably reproduce the failure by:
+
+1) creating a view that depends on this model
+2) issuing a long-running query on the view before `dbt run` is invoked
+3) issuing _another_ long-running query against the original model
+
+Since long-running query (step 2) is selecting from the view, Redshift blocks on the `drop ... cascade`, of the materialization, which causes the query from step 3 time to overlap with the critical section of the materialization between the `drop` and `alter` statements.
+
+In dbt v0.10.1, this integration test results in:
+
+```
+======================================================================
+FAIL: test__redshift__concurrent_transaction (test_concurrent_transaction.TestConcurrentTransaction)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/usr/src/app/test/integration/032_concurrent_transaction_test/test_concurrent_transaction.py", line 84, in test__redshift__concurrent_transaction
+    self.assertEqual(self.query_state['model_1'], 'good')
+AssertionError: 'error: table 3379442 dropped by concurrent transaction\n' != 'good'
+- error: table 3379442 dropped by concurrent transaction
++ good
+```
diff --git a/test/integration/032_concurrent_transaction_test/macros/udfs.sql b/test/integration/032_concurrent_transaction_test/macros/udfs.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/macros/udfs.sql
@@ -0,0 +1,13 @@
+
+{% macro create_udfs() %}
+
+CREATE OR REPLACE FUNCTION {{ target.schema }}.f_sleep (x float)
+RETURNS bool IMMUTABLE
+AS
+$$
+  from time import sleep
+  sleep(x)
+  return True
+$$ LANGUAGE plpythonu;
+
+{% endmacro %}
diff --git a/test/integration/032_concurrent_transaction_test/models-incremental/model_1.sql b/test/integration/032_concurrent_transaction_test/models-incremental/model_1.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/models-incremental/model_1.sql
@@ -0,0 +1,5 @@
+
+{{ config(materialized='incremental', sql_where=True, unique_key='id') }}
+
+-- incremental model
+select 1 as id
diff --git a/test/integration/032_concurrent_transaction_test/models-incremental/view_model.sql b/test/integration/032_concurrent_transaction_test/models-incremental/view_model.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/models-incremental/view_model.sql
@@ -0,0 +1,3 @@
+
+
+select * from {{ ref('model_1') }}
diff --git a/test/integration/032_concurrent_transaction_test/models-table/model_1.sql b/test/integration/032_concurrent_transaction_test/models-table/model_1.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/models-table/model_1.sql
@@ -0,0 +1,5 @@
+
+{{ config(materialized='table') }}
+
+-- table model
+select 1 as id
diff --git a/test/integration/032_concurrent_transaction_test/models-table/view_model.sql b/test/integration/032_concurrent_transaction_test/models-table/view_model.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/models-table/view_model.sql
@@ -0,0 +1,3 @@
+
+
+select * from {{ ref('model_1') }}
diff --git a/test/integration/032_concurrent_transaction_test/models-view/model_1.sql b/test/integration/032_concurrent_transaction_test/models-view/model_1.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/models-view/model_1.sql
@@ -0,0 +1,5 @@
+
+{{ config(materialized='view') }}
+
+-- view model
+select 1 as id
diff --git a/test/integration/032_concurrent_transaction_test/models-view/view_model.sql b/test/integration/032_concurrent_transaction_test/models-view/view_model.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/models-view/view_model.sql
@@ -0,0 +1,3 @@
+
+
+select * from {{ ref('model_1') }}
diff --git a/test/integration/032_concurrent_transaction_test/test_concurrent_transaction.py b/test/integration/032_concurrent_transaction_test/test_concurrent_transaction.py
new file mode 100644
--- /dev/null
+++ b/test/integration/032_concurrent_transaction_test/test_concurrent_transaction.py
@@ -0,0 +1,111 @@
+from nose.plugins.attrib import attr
+from test.integration.base import DBTIntegrationTest
+import threading
+
+class BaseTestConcurrentTransaction(DBTIntegrationTest):
+
+    def reset(self):
+        self.query_state = {
+            'view_model': 'wait',
+            'model_1': 'wait',
+        }
+
+    @property
+    def schema(self):
+        return "concurrent_transaction_032"
+
+    @property
+    def project_config(self):
+        return {
+            "macro-paths": ["test/integration/032_concurrent_transaction_test/macros"],
+            "on-run-start": [
+                "{{ create_udfs() }}",
+            ],
+        }
+
+    def run_select_and_check(self, rel, sql):
+        try:
+            res = self.run_sql(sql, fetch='one')
+
+            # The result is the output of f_sleep(), which is True
+            if res[0] == True:
+                self.query_state[rel] = 'good'
+            else:
+                self.query_state[rel] = 'bad'
+
+        except Exception as e:
+            if 'concurrent transaction' in str(e):
+                self.query_state[rel] = 'error: {}'.format(e)
+            else:
+                self.query_state[rel] = 'error: {}'.format(e)
+
+    def async_select(self, rel, sleep=10):
+        # Run the select statement in a thread. When the query returns, the global
+        # query_state will be update with a state of good/bad/error, and the associated
+        # error will be reported if one was raised.
+
+        schema = self.unique_schema()
+        query = '''
+        -- async_select: {rel}
+        select {schema}.f_sleep({sleep}) from {schema}.{rel}
+        '''.format(
+                schema=schema,
+                sleep=sleep,
+                rel=rel)
+
+        thread = threading.Thread(target=lambda: self.run_select_and_check(rel, query))
+        thread.start()
+        return thread
+
+    def run_test(self):
+        self.use_profile("redshift")
+
+        # First run the project to make sure the models exist
+        self.run_dbt(args=['run'])
+
+        # Execute long-running queries in threads
+        t1 = self.async_select('view_model', 10)
+        t2 = self.async_select('model_1', 5)
+
+        # While the queries are executing, re-run the project
+        res = self.run_dbt(args=['run', '--threads', '8'])
+
+        # Finally, wait for these threads to finish
+        t1.join()
+        t2.join()
+
+        self.assertTrue(len(res) > 0)
+
+        # If the query succeeded, the global query_state should be 'good'
+        self.assertEqual(self.query_state['view_model'], 'good')
+        self.assertEqual(self.query_state['model_1'], 'good')
+
+class TableTestConcurrentTransaction(BaseTestConcurrentTransaction):
+    @property
+    def models(self):
+        return "test/integration/032_concurrent_transaction_test/models-table"
+
+    @attr(type="redshift")
+    def test__redshift__concurrent_transaction_table(self):
+        self.reset()
+        self.run_test()
+
+class ViewTestConcurrentTransaction(BaseTestConcurrentTransaction):
+    @property
+    def models(self):
+        return "test/integration/032_concurrent_transaction_test/models-view"
+
+    @attr(type="redshift")
+    def test__redshift__concurrent_transaction_view(self):
+        self.reset()
+        self.run_test()
+
+class IncrementalTestConcurrentTransaction(BaseTestConcurrentTransaction):
+    @property
+    def models(self):
+        return "test/integration/032_concurrent_transaction_test/models-incremental"
+
+    @attr(type="redshift")
+    def test__redshift__concurrent_transaction_incremental(self):
+        self.reset()
+        self.run_test()
diff --git a/test/integration/base.py b/test/integration/base.py
--- a/test/integration/base.py
+++ b/test/integration/base.py
@@ -64,6 +64,28 @@ def postgres_profile(self):
             }
         }
 
+    def redshift_profile(self):
+        return {
+            'config': {
+                'send_anonymous_usage_stats': False
+            },
+            'test': {
+                'outputs': {
+                    'default2': {
+                        'type': 'redshift',
+                        'threads': 1,
+                        'host': os.getenv('REDSHIFT_TEST_HOST'),
+                        'port': os.getenv('REDSHIFT_TEST_PORT'),
+                        'user': os.getenv('REDSHIFT_TEST_USER'),
+                        'pass': os.getenv('REDSHIFT_TEST_PASS'),
+                        'dbname': os.getenv('REDSHIFT_TEST_DBNAME'),
+                        'schema': self.unique_schema()
+                    }
+                },
+                'target': 'default2'
+            }
+        }
+
     def snowflake_profile(self):
         return {
             'config': {
@@ -137,6 +159,8 @@ def get_profile(self, adapter_type):
             return self.snowflake_profile()
         elif adapter_type == 'bigquery':
             return self.bigquery_profile()
+        elif adapter_type == 'redshift':
+            return self.redshift_profile()
 
     def setUp(self):
         self.adapter_type = 'postgres'