Repository: dbt-labs/dbt-core

Issue:
Support CopyJobs in BigQuery via a macro 
### Describe the feature
Support a `copy_table` macro in BigQuery, which takes advantage of BigQuery's [CopyJob](https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.CopyJob.html) which is better optimized than, for instance `select * {{source('my_source')}}`.

The write disposition on the resulting table can be either `append` or `truncate` corresponding to materializations of `incremental` or `table` here, depending on how you want to modify the resulting table.

This should look like the following in a model:

```
{{ config(materialized='table') }}

{{ copy_table(ref('my_table')) }}
``` 

### Describe alternatives you've considered
Just use `select *` here.

### Additional context
Only interested in BigQuery for this issue, but if there are analogous features in other databases, they can be implemented as well, and possible this can become a global macro.

### Who will this benefit?
Any BigQuery users who have step(s) in their SQL pipeline which simply copy data (ex. a release step, or curation that may leave some tables unchanged)

### Are you interested in contributing this feature?
I have already written this and maintained it as a patch on a fork of dbt, but would love to upstream & share with anyone who can take advantage.



Discussion:
Thanks for the issue @kconvey, I'm supportive of adding this as an adapter method for BigQuery. I completely agree that it should take a relation as its argument. I imagine there should also be a check to confirm that the relation exists as a table (as opposed to a view or ephemeral model).

I'd be curious to hear a little more about the ways in which this is better optimized than `select * from table`, if you know them. I couldn't find anything after briefly checking the docs.

I think the closest comparable feature in another database is zero-copy cloning on Snowflake. The two feel different enough that we shouldn't try to implement them as adapter-specific versions of the same macro. (Also, Snowflake has a DDL interface for cloning, so it doesn't _need_ to be an adapter macro.)
+1 to checking that the relation is `table` or `incremental`. 

I don't know of any specific BigQuery implementation details that make `copy_table` taster than `select * from table`, but several months back I found it was a bit better when trying to speed up a pipeline which had an initial model that was a massive copy step, from which all other models derived. The pipeline also was originally written in python (before conversion to dbt), and included both BQ sql jobs and copy jobs at various points, so it was useful to have this feature parity in our fork of dbt when convincing the pipeline owners to switch to dbt.

Going to create a PR for this. Thanks for the triage!

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/connections.py b/plugins/bigquery/dbt/adapters/bigquery/connections.py
--- a/plugins/bigquery/dbt/adapters/bigquery/connections.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/connections.py
@@ -26,6 +26,8 @@
 
 BQ_QUERY_JOB_SPLIT = '-----Query Job SQL Follows-----'
 
+WRITE_TRUNCATE = google.cloud.bigquery.job.WriteDisposition.WRITE_TRUNCATE
+
 REOPENABLE_ERRORS = (
     ConnectionResetError,
     ConnectionError,
@@ -336,7 +338,7 @@ def create_table(self, database, schema, table_name, sql):
 
         table_ref = self.table_ref(database, schema, table_name, conn)
         job_params = {'destination': table_ref,
-                      'write_disposition': 'WRITE_TRUNCATE'}
+                      'write_disposition': WRITE_TRUNCATE}
 
         timeout = self.get_timeout(conn)
 
@@ -352,6 +354,32 @@ def callback(table):
         self.create_bigquery_table(database, schema, table_name, callback,
                                    'CREATE DAY PARTITIONED TABLE')
 
+    def copy_bq_table(self, source, destination, write_disposition):
+        conn = self.get_thread_connection()
+        client = conn.handle
+
+        source_ref = self.table_ref(
+            source.database, source.schema, source.table, conn)
+        destination_ref = self.table_ref(
+            destination.database, destination.schema, destination.table, conn)
+
+        logger.debug(
+            'Copying table "{}" to "{}" with disposition: "{}"',
+            source_ref.path, destination_ref.path, write_disposition)
+
+        def copy_and_results():
+            job_config = google.cloud.bigquery.CopyJobConfig(
+                write_disposition=write_disposition)
+            copy_job = client.copy_table(
+                source_ref, destination_ref, job_config=job_config)
+            iterator = copy_job.result(timeout=self.get_timeout(conn))
+            return copy_job, iterator
+
+        self._retry_and_handle(
+            msg='copy table "{}" to "{}"'.format(
+                source_ref.path, destination_ref.path),
+            conn=conn, fn=copy_and_results)
+
     @staticmethod
     def dataset(database, schema, conn):
         dataset_ref = conn.handle.dataset(schema, database)
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -32,6 +32,10 @@
 import agate
 import json
 
+# Write dispositions for bigquery.
+WRITE_APPEND = google.cloud.bigquery.job.WriteDisposition.WRITE_APPEND
+WRITE_TRUNCATE = google.cloud.bigquery.job.WriteDisposition.WRITE_TRUNCATE
+
 
 def sql_escape(string):
     if not isinstance(string, str):
@@ -417,6 +421,23 @@ def _materialize_as_table(
 
         return "CREATE TABLE"
 
+    @available.parse(lambda *a, **k: '')
+    def copy_table(self, source, destination, materialization):
+        if materialization == 'incremental':
+            write_disposition = WRITE_APPEND
+        elif materialization == 'table':
+            write_disposition = WRITE_TRUNCATE
+        else:
+            dbt.exceptions.raise_compiler_error(
+                'Copy table materialization must be "copy" or "table", but '
+                f"config.get('copy_materialization', 'table') was "
+                f'{materialization}')
+
+        self.connections.copy_bq_table(
+            source, destination, write_disposition)
+
+        return "COPY TABLE with materialization: {}".format(materialization)
+
     @classmethod
     def poll_until_job_completes(cls, job, timeout):
         retry_count = timeout


Test patch:
diff --git a/test/integration/022_bigquery_test/copy-failing-models/copy_bad_materialization.sql b/test/integration/022_bigquery_test/copy-failing-models/copy_bad_materialization.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/copy-failing-models/copy_bad_materialization.sql
@@ -0,0 +1,2 @@
+{{ config(copy_materialization='view') }}
+{{ ref('original') }}
\ No newline at end of file
diff --git a/test/integration/022_bigquery_test/copy-failing-models/original.sql b/test/integration/022_bigquery_test/copy-failing-models/original.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/copy-failing-models/original.sql
@@ -0,0 +1 @@
+select 1 as id
\ No newline at end of file
diff --git a/test/integration/022_bigquery_test/copy-models/copy_as_incremental.sql b/test/integration/022_bigquery_test/copy-models/copy_as_incremental.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/copy-models/copy_as_incremental.sql
@@ -0,0 +1,2 @@
+{{ config(copy_materialization='incremental') }}
+{{ ref('original') }}
\ No newline at end of file
diff --git a/test/integration/022_bigquery_test/copy-models/copy_as_table.sql b/test/integration/022_bigquery_test/copy-models/copy_as_table.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/copy-models/copy_as_table.sql
@@ -0,0 +1,2 @@
+{{ config(copy_materialization='table') }}
+{{ ref('original') }}
\ No newline at end of file
diff --git a/test/integration/022_bigquery_test/copy-models/original.sql b/test/integration/022_bigquery_test/copy-models/original.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/copy-models/original.sql
@@ -0,0 +1 @@
+select 1 as id
\ No newline at end of file
diff --git a/test/integration/022_bigquery_test/test_bigquery_copy_failing_models.py b/test/integration/022_bigquery_test/test_bigquery_copy_failing_models.py
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/test_bigquery_copy_failing_models.py
@@ -0,0 +1,36 @@
+from test.integration.base import DBTIntegrationTest, use_profile
+import textwrap
+import yaml
+
+
+class TestBigqueryCopyTableFails(DBTIntegrationTest):
+
+    @property
+    def schema(self):
+        return "bigquery_test_022"
+
+    @property
+    def models(self):
+        return "copy-failing-models"
+
+    @property
+    def profile_config(self):
+        return self.bigquery_profile()
+
+    @property
+    def project_config(self):
+        return yaml.safe_load(textwrap.dedent('''\
+        config-version: 2
+        models:
+            test:
+                original:
+                    materialized: table
+                copy_bad_materialization:
+                    materialized: copy
+        '''))
+
+    @use_profile('bigquery')
+    def test__bigquery_copy_table_fails(self):
+        results = self.run_dbt(expect_pass=False)
+        self.assertEqual(len(results), 2)
+        self.assertTrue(results[1].error)
diff --git a/test/integration/022_bigquery_test/test_bigquery_copy_models.py b/test/integration/022_bigquery_test/test_bigquery_copy_models.py
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/test_bigquery_copy_models.py
@@ -0,0 +1,37 @@
+from test.integration.base import DBTIntegrationTest, use_profile
+import textwrap
+import yaml
+
+
+class TestBigqueryCopyTable(DBTIntegrationTest):
+
+    @property
+    def schema(self):
+        return "bigquery_test_022"
+
+    @property
+    def models(self):
+        return "copy-models"
+
+    @property
+    def profile_config(self):
+        return self.bigquery_profile()
+
+    @property
+    def project_config(self):
+        return yaml.safe_load(textwrap.dedent('''\
+        config-version: 2
+        models:
+            test:
+                original:
+                    materialized: table
+                copy_as_table:
+                    materialized: copy
+                copy_as_incremental:
+                    materialized: copy
+        '''))
+
+    @use_profile('bigquery')
+    def test__bigquery_copy_table(self):
+        results = self.run_dbt()
+        self.assertEqual(len(results), 3)
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -4,7 +4,7 @@
 import unittest
 from contextlib import contextmanager
 from requests.exceptions import ConnectionError
-from unittest.mock import patch, MagicMock, Mock, create_autospec
+from unittest.mock import patch, MagicMock, Mock, create_autospec, ANY
 
 import hologram
 
@@ -21,8 +21,9 @@
 import dbt.exceptions
 from dbt.logger import GLOBAL_LOGGER as logger  # noqa
 
-from .utils import config_from_parts_or_dicts, inject_adapter, TestAdapterConversions
+import google.cloud.bigquery
 
+from .utils import config_from_parts_or_dicts, inject_adapter, TestAdapterConversions
 
 def _bq_conn():
     conn = MagicMock()
@@ -516,8 +517,65 @@ def test_query_and_results(self, mock_bq):
         self.mock_client.query.assert_called_once_with(
           'sql', job_config=mock_bq.QueryJobConfig())
 
+    
+    def test_copy_bq_table_appends(self):
+        self._copy_table(
+            write_disposition=dbt.adapters.bigquery.impl.WRITE_APPEND)
+        args, kwargs = self.mock_client.copy_table.call_args
+        self.mock_client.copy_table.assert_called_once_with(
+            self._table_ref('project', 'dataset', 'table1', None),
+            self._table_ref('project', 'dataset', 'table2', None),
+            job_config=ANY)
+        args, kwargs = self.mock_client.copy_table.call_args
+        self.assertEqual(
+            kwargs['job_config'].write_disposition,
+            dbt.adapters.bigquery.impl.WRITE_APPEND)
+
+    def test_copy_bq_table_truncates(self):
+        self._copy_table(
+            write_disposition=dbt.adapters.bigquery.impl.WRITE_TRUNCATE)
+        args, kwargs = self.mock_client.copy_table.call_args
+        self.mock_client.copy_table.assert_called_once_with(
+            self._table_ref('project', 'dataset', 'table1', None),
+            self._table_ref('project', 'dataset', 'table2', None),
+            job_config=ANY)
+        args, kwargs = self.mock_client.copy_table.call_args
+        self.assertEqual(
+            kwargs['job_config'].write_disposition,
+            dbt.adapters.bigquery.impl.WRITE_TRUNCATE)
+
+    def _table_ref(self, proj, ds, table, conn):
+        return google.cloud.bigquery.table.TableReference.from_string(
+            '{}.{}.{}'.format(proj, ds, table))
+
+    def _copy_table(self, write_disposition):
+
+        self.connections.table_ref = self._table_ref
+        source = BigQueryRelation.create(
+            database='project', schema='dataset', identifier='table1')
+        destination = BigQueryRelation.create(
+            database='project', schema='dataset', identifier='table2')
+        self.connections.copy_bq_table(source, destination, write_disposition)
+
+
+class TestBigQueryAdapter(BaseTestBigQueryAdapter):
+
+    def test_copy_table_materialization_table(self):
+        adapter = self.get_adapter('oauth')
+        adapter.connections = MagicMock()
+        adapter.copy_table('source', 'destination', 'table')
+        adapter.connections.copy_bq_table.assert_called_once_with(
+            'source', 'destination',
+            dbt.adapters.bigquery.impl.WRITE_TRUNCATE)
+
+    def test_copy_table_materialization_incremental(self):
+        adapter = self.get_adapter('oauth')
+        adapter.connections = MagicMock()
+        adapter.copy_table('source', 'destination', 'incremental')
+        adapter.connections.copy_bq_table.assert_called_once_with(
+            'source', 'destination',
+            dbt.adapters.bigquery.impl.WRITE_APPEND)
 
-class TestBigQueryTableOptions(BaseTestBigQueryAdapter):
     def test_parse_partition_by(self):
         adapter = self.get_adapter('oauth')