Repository: dbt-labs/dbt-core

Issue:
Disable most transactional logic on Snowflake
### Describe the feature

Today, dbt treats Snowflake as it would any transactional database (e.g. Postgres, Redshift): `begin` + `commit` sandwich all DDL/DML for materializations, tests, snapshot-freshness queries, etc. This has the effect of running more queries than strictly necessary (#2748). More recently, we've come to understand that transactional logic weights heavily in the Cloud Services layer.

The simplest solution here would be:
1. Disable transactional logic in Snowflake by default, let DDL/DML statements auto-commit themselves wherever possible
2. Add explicit `begin` + `commit` in only the places where it's needed, i.e. multiple DDL/DML statements that must be executed within a single transaction

Step 1 could be as simple as replicating the code in dbt-bigquery:
https://github.com/fishtown-analytics/dbt/blob/eb46bfc3d6fd70ee5352fa758470da03dc3700d1/plugins/bigquery/dbt/adapters/bigquery/connections.py#L147-L148

https://github.com/fishtown-analytics/dbt/blob/eb46bfc3d6fd70ee5352fa758470da03dc3700d1/plugins/bigquery/dbt/adapters/bigquery/connections.py#L196-L200

My sense is that the code changes for this are actually quite easy. The trickiness comes in considering and testing all possible permutations / edge cases, and validating that Snowflake's empirical behavior in each case matches its documented behavior. To that end, it's worth _really_ digging into how `autocommit` works on Snowflake ([docs](https://docs.snowflake.com/en/sql-reference/transactions.html#autocommit)), and understanding _exactly_ how dbt interacts with each note below:

> Snowflake supports an AUTOCOMMIT parameter. The default setting for AUTOCOMMIT is on.
> 
> While AUTOCOMMIT is enabled:
> 
> - Each statement outside an explicit transaction is treated as though it is inside its own implicit single-statement transaction. In other words, that statement is automatically committed if it succeeds, and automatically rolled back if it fails. Statements inside an explicit transaction are not affected by AUTOCOMMIT. For example, statements inside an explicit BEGIN TRANSACTION ... ROLLBACK are rolled back even if AUTOCOMMIT is TRUE.
> 
> While AUTOCOMMIT is disabled:
> 
> - An implicit BEGIN TRANSACTION is executed at:
> 
>     - The first DML statement or query statement after a transaction ends. This is true regardless of what ended the preceding transaction (e.g. a DDL statement, or an explicit commit or rollback).
> 
>     - The first DML statement or query statement after disabling AUTOCOMMIT.
> 
> - An implicit COMMIT is executed at the following (if a transaction is already active):
> 
>     - The execution of a DDL statement.
> 
>     - The execution of an ALTER SESSION SET AUTOCOMMIT statement, regardless of whether the new value is TRUE or FALSE, and regardless of whether or not the new value is different from the previous value. For example, even if you set AUTOCOMMIT to FALSE when it is already FALSE, an implicit COMMIT is executed.
> 
> - An implicit ROLLBACK is executed at the following (if a transaction is already active):
> 
>     - The end of a session.
> 
>     - The end of a stored procedure. Regardless of whether the stored procedure’s active transaction was started explicitly or implicitly, Snowflake rolls back the active transaction and issues an error message.

Here's what sticks out to me:
- `AUTOCOMMIT` is on by default, but it could be off. Should dbt demand to work with `AUTOCOMMIT` on? Should we add this as an option in `profiles.yml`?
- Since v0.17.2, dbt closes connections on Snowflake once it's done using them. This has the effect of ending the session, i.e. implicitly rolling back all transactions if `AUTOCOMMIT` is off.
- The `delete+insert` incremental strategy requires running two DML statements within the same transaction. We'll need explicit `begin` + `end` logic there.

Another trade-off here: `auto_begin` (for statements) and `inside_transaction` (for pre- and post-hooks) would no longer work. But it's not clear to me that they're working well today, either. We could advise users to explicitly specify `begin` + `commit` within their statements or hook definitions, rather than relying on dbt's built-in methods to open up or close out transactions on their behalf.

### Describe alternatives you've considered
- Leaving all the unnecessary `begin` + `commit` in place, to the tune of much unnecessary credit expenditure
- Reconsidering dbt's default behavior around transactions, as the majority of dbt projects increasingly work with non-transactional databases

### Who will this benefit?
dbt + Snowflake users


Discussion:

Resolving patch:
diff --git a/plugins/snowflake/dbt/adapters/snowflake/connections.py b/plugins/snowflake/dbt/adapters/snowflake/connections.py
--- a/plugins/snowflake/dbt/adapters/snowflake/connections.py
+++ b/plugins/snowflake/dbt/adapters/snowflake/connections.py
@@ -224,7 +224,7 @@ def open(cls, connection):
                 schema=creds.schema,
                 warehouse=creds.warehouse,
                 role=creds.role,
-                autocommit=False,
+                autocommit=True,
                 client_session_keep_alive=creds.client_session_keep_alive,
                 application='dbt',
                 **creds.auth_args()
@@ -275,6 +275,23 @@ def get_response(cls, cursor) -> AdapterResponse:
             code=code
         )
 
+    # disable transactional logic by default on Snowflake
+    # except for DML statements where explicitly defined
+    def add_begin_query(self, *args, **kwargs):
+        pass
+
+    def add_commit_query(self, *args, **kwargs):
+        pass
+
+    def begin(self):
+        pass
+
+    def commit(self):
+        pass
+
+    def clear_transaction(self):
+        pass
+
     @classmethod
     def _split_queries(cls, sql):
         "Splits sql statements at semicolons into discrete queries"
@@ -352,15 +369,3 @@ def add_query(self, sql, auto_begin=True,
             )
 
         return connection, cursor
-
-    @classmethod
-    def _rollback_handle(cls, connection):
-        """On snowflake, rolling back the handle of an aborted session raises
-        an exception.
-        """
-        try:
-            connection.handle.rollback()
-        except snowflake.connector.errors.ProgrammingError as e:
-            msg = str(e)
-            if 'Session no longer exists' not in msg:
-                raise


Test patch:
diff --git a/test/unit/test_snowflake_adapter.py b/test/unit/test_snowflake_adapter.py
--- a/test/unit/test_snowflake_adapter.py
+++ b/test/unit/test_snowflake_adapter.py
@@ -134,8 +134,11 @@ def test_quoting_on_truncate(self):
         )
         self.adapter.truncate_relation(relation)
 
+        # no query comment because wrapped in begin; + commit; for explicit DML
         self.mock_execute.assert_has_calls([
-            mock.call('/* dbt */\ntruncate table test_database."test_schema".test_table', None)
+            mock.call('/* dbt */\nbegin;', None),
+            mock.call('truncate table test_database."test_schema".test_table\n  ;', None),
+            mock.call('commit;', None)
         ])
 
     def test_quoting_on_rename(self):
@@ -258,7 +261,7 @@ def test_client_session_keep_alive_false_by_default(self):
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', private_key=None, application='dbt')
@@ -274,7 +277,7 @@ def test_client_session_keep_alive_true(self):
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=True, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', private_key=None, application='dbt')
@@ -291,7 +294,7 @@ def test_user_pass_authentication(self):
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 password='test_password', role=None, schema='public',
                 user='test_user', warehouse='test_warehouse', private_key=None,
@@ -310,7 +313,7 @@ def test_authenticator_user_pass_authentication(self):
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 password='test_password', role=None, schema='public',
                 user='test_user', warehouse='test_warehouse',
@@ -329,7 +332,7 @@ def test_authenticator_externalbrowser_authentication(self):
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', authenticator='externalbrowser',
@@ -348,7 +351,7 @@ def test_authenticator_oauth_authentication(self):
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', authenticator='oauth', token='my-oauth-token',
@@ -369,7 +372,7 @@ def test_authenticator_private_key_authentication(self, mock_get_private_key):
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', private_key='test_key',
@@ -390,7 +393,7 @@ def test_authenticator_private_key_authentication_no_passphrase(self, mock_get_p
         conn.handle
         self.snowflake.assert_has_calls([
             mock.call(
-                account='test_account', autocommit=False,
+                account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', private_key='test_key',