Repository: dbt-labs/dbt-core

Issue:
persist_docs functionality doesn't persist descriptions on nested bigquery columns
### Describe the bug
With the release of dbt 0.17.0, the persist_docs functionality was enhanced to also apply column descriptions to bigquery when they are supplied in a schema.yml file. However, it seems that it only takes into account top-level columns and there is no way to persist a description to a nested column (ie a field that is part of a RECORD type bigquery column).

### Steps To Reproduce
Suppose you create a simple test model using dbt in bigquery:
``` SQL
SELECT
    STRUCT(
        STRUCT(
            1 AS level_3_a,
            2 AS level_3_b
        ) AS level_2
    ) AS level_1
```

This will create a view/table in bigquery with the following schema:
![image](https://user-images.githubusercontent.com/1922169/84779953-4ea48d80-afe5-11ea-98ea-18d7a686647e.png)

We can describe this table in a schema.yml file and add descriptions as follows:
```yaml
version: 2

models:
  - name: test_nested
  
    columns:
      - name: level_1
        description: level_1 description
      - name: level_1.level_2.level_3_a
        description: level_3_a description
```

When we now perform a `dbt run` on this model, only the description on the top level field is added and no warnings or errors are generated by dbt:
![image](https://user-images.githubusercontent.com/1922169/84780414-d5f20100-afe5-11ea-869a-386b8518dd3f.png)

### Expected behavior
When providing descriptions for nested fields (preferably using the dotted syntax as shown above), the descriptions should be correctly persisted on nested fields.

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [x] bigquery
- [ ] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
installed version: 0.17.0
   latest version: 0.17.0

Up to date!

Plugins:
  - snowflake: 0.17.0
  - postgres: 0.17.0
  - bigquery: 0.17.0
  - redshift: 0.17.0
```

**The operating system you're using:**
Mac OS 10.15.3

**The output of `python --version`:**
Python 3.7.5



Discussion:

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -564,6 +564,36 @@ def get_table_ref_from_relation(self, conn, relation):
                                           relation.identifier,
                                           conn)
 
+    def _update_column_dict(self, bq_column_dict, dbt_columns, parent=''):
+        """
+        Helper function to recursively traverse the schema of a table in the
+        update_column_descriptions function below.
+
+        bq_column_dict should be a dict as obtained by the to_api_repr()
+        function of a SchemaField object.
+        """
+        if parent:
+            dotted_column_name = '{}.{}'.format(parent, bq_column_dict['name'])
+        else:
+            dotted_column_name = bq_column_dict['name']
+
+        if dotted_column_name in dbt_columns:
+            column_config = dbt_columns[dotted_column_name]
+            bq_column_dict['description'] = column_config.get('description')
+
+        new_fields = []
+        for child_col_dict in bq_column_dict.get('fields', list()):
+            new_child_column_dict = self._update_column_dict(
+                child_col_dict,
+                dbt_columns,
+                parent=dotted_column_name
+            )
+            new_fields.append(new_child_column_dict)
+
+        bq_column_dict['fields'] = new_fields
+
+        return bq_column_dict
+
     @available.parse_none
     def update_column_descriptions(self, relation, columns):
         if len(columns) == 0:
@@ -574,13 +604,13 @@ def update_column_descriptions(self, relation, columns):
         table = conn.handle.get_table(table_ref)
 
         new_schema = []
-        for column in table.schema:
-            if column.name in columns:
-                column_config = columns[column.name]
-                column_dict = column.to_api_repr()
-                column_dict['description'] = column_config.get('description')
-                column = SchemaField.from_api_repr(column_dict)
-            new_schema.append(column)
+        for bq_column in table.schema:
+            bq_column_dict = bq_column.to_api_repr()
+            new_bq_column_dict = self._update_column_dict(
+                bq_column_dict,
+                columns
+            )
+            new_schema.append(SchemaField.from_api_repr(new_bq_column_dict))
 
         new_table = google.cloud.bigquery.Table(table_ref, schema=new_schema)
         conn.handle.update_table(new_table, ['schema'])


Test patch:
diff --git a/test/integration/029_docs_generate_tests/bq_models/nested_table.sql b/test/integration/029_docs_generate_tests/bq_models/nested_table.sql
--- a/test/integration/029_docs_generate_tests/bq_models/nested_table.sql
+++ b/test/integration/029_docs_generate_tests/bq_models/nested_table.sql
@@ -10,6 +10,6 @@ select
     3 as field_3,
 
     struct(
-        4 as field_4,
-        5 as field_5
+        5 as field_5,
+        6 as field_6
     ) as nested_field
diff --git a/test/integration/029_docs_generate_tests/test_docs_generate.py b/test/integration/029_docs_generate_tests/test_docs_generate.py
--- a/test/integration/029_docs_generate_tests/test_docs_generate.py
+++ b/test/integration/029_docs_generate_tests/test_docs_generate.py
@@ -619,10 +619,10 @@ def expected_bigquery_complex_catalog(self):
                 'type': 'INT64',
                 'comment': None
             },
-            'nested_field.field_4': {
-                'name': 'nested_field.field_4',
+            'nested_field': {
+                'name': 'nested_field',
                 'index': 4,
-                'type': 'INT64',
+                'type': 'STRUCT<field_5 INT64, field_6 INT64>',
                 'comment': None
             },
             'nested_field.field_5': {
@@ -630,6 +630,12 @@ def expected_bigquery_complex_catalog(self):
                 'index': 5,
                 'type': 'INT64',
                 'comment': None
+            },
+            'nested_field.field_6': {
+                'name': 'nested_field.field_6',
+                'index': 6,
+                'type': 'INT64',
+                'comment': None
             }
         }
 
diff --git a/test/integration/060_persist_docs_tests/models-bigquery-nested/schema.yml b/test/integration/060_persist_docs_tests/models-bigquery-nested/schema.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/060_persist_docs_tests/models-bigquery-nested/schema.yml
@@ -0,0 +1,19 @@
+version: 2
+
+models:
+  - name: table_model_nested
+    columns:
+      - name: level_1
+        description: level_1 column description
+      - name: level_1.level_2
+        description: level_2 column description
+      - name: level_1.level_2.level_3_a
+        description: level_3 column description
+  - name: view_model_nested
+    columns:
+      - name: level_1
+        description: level_1 column description
+      - name: level_1.level_2
+        description: level_2 column description
+      - name: level_1.level_2.level_3_a
+        description: level_3 column description
\ No newline at end of file
diff --git a/test/integration/060_persist_docs_tests/models-bigquery-nested/table_model_nested.sql b/test/integration/060_persist_docs_tests/models-bigquery-nested/table_model_nested.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/060_persist_docs_tests/models-bigquery-nested/table_model_nested.sql
@@ -0,0 +1,8 @@
+{{ config(materialized='table') }}
+SELECT
+    STRUCT(
+        STRUCT(
+            1 AS level_3_a,
+            2 AS level_3_b
+        ) AS level_2
+    ) AS level_1
\ No newline at end of file
diff --git a/test/integration/060_persist_docs_tests/models-bigquery-nested/view_model_nested.sql b/test/integration/060_persist_docs_tests/models-bigquery-nested/view_model_nested.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/060_persist_docs_tests/models-bigquery-nested/view_model_nested.sql
@@ -0,0 +1,8 @@
+{{ config(materialized='view') }}
+SELECT
+    STRUCT(
+        STRUCT(
+            1 AS level_3_a,
+            2 AS level_3_b
+        ) AS level_2
+    ) AS level_1
\ No newline at end of file
diff --git a/test/integration/060_persist_docs_tests/test_persist_docs.py b/test/integration/060_persist_docs_tests/test_persist_docs.py
--- a/test/integration/060_persist_docs_tests/test_persist_docs.py
+++ b/test/integration/060_persist_docs_tests/test_persist_docs.py
@@ -33,13 +33,15 @@ def _assert_has_table_comments(self, table_node):
         assert table_id_comment.startswith('id Column description')
 
         table_name_comment = table_node['columns']['name']['comment']
-        assert table_name_comment.startswith('Some stuff here and then a call to')
+        assert table_name_comment.startswith(
+            'Some stuff here and then a call to')
 
         self._assert_common_comments(
             table_comment, table_id_comment, table_name_comment
         )
 
-    def _assert_has_view_comments(self, view_node, has_node_comments=True, has_column_comments=True):
+    def _assert_has_view_comments(self, view_node, has_node_comments=True,
+                                  has_column_comments=True):
         view_comment = view_node['metadata']['comment']
         if has_node_comments:
             assert view_comment.startswith('View model description')
@@ -156,3 +158,76 @@ def test_snowflake_persist_docs(self):
     @use_profile('bigquery')
     def test_bigquery_persist_docs(self):
         self.run_dbt()
+
+
+class TestPersistDocsNested(BasePersistDocsTest):
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'models': {
+                'test': {
+                    '+persist_docs': {
+                        "relation": True,
+                        "columns": True,
+                    },
+                }
+            }
+        }
+
+    @property
+    def models(self):
+        return 'models-bigquery-nested'
+
+    @use_profile('bigquery')
+    def test_bigquery_persist_docs(self):
+        """
+        run dbt and use the bigquery client from the adapter to check if the
+        colunmn descriptions are persisted on the test model table and view.
+
+        Next, generate the catalog and check if the comments are also included.
+        """
+        self.run_dbt()
+
+        self.run_dbt(['docs', 'generate'])
+        with open('target/catalog.json') as fp:
+            catalog_data = json.load(fp)
+        assert 'nodes' in catalog_data
+        assert len(catalog_data['nodes']) == 2  # table and view model
+
+        for node_id in ['table_model_nested', 'view_model_nested']:
+            # check the descriptions using the api
+            with self.adapter.connection_named('_test'):
+                client = self.adapter.connections \
+                    .get_thread_connection().handle
+
+                table_id = "{}.{}.{}".format(
+                    self.default_database,
+                    self.unique_schema(),
+                    node_id
+                )
+                bq_schema = client.get_table(table_id).schema
+
+                level_1_field = bq_schema[0]
+                assert level_1_field.description == \
+                       "level_1 column description"
+
+                level_2_field = level_1_field.fields[0]
+                assert level_2_field.description == \
+                       "level_2 column description"
+
+                level_3_field = level_2_field.fields[0]
+                assert level_3_field.description == \
+                       "level_3 column description"
+
+            # check the descriptions in the catalog
+            node = catalog_data['nodes']['model.test.{}'.format(node_id)]
+
+            level_1_column = node['columns']['level_1']
+            assert level_1_column['comment'] == "level_1 column description"
+
+            level_2_column = node['columns']['level_1.level_2']
+            assert level_2_column['comment'] == "level_2 column description"
+
+            level_3_column = node['columns']['level_1.level_2.level_3_a']
+            assert level_3_column['comment'] == "level_3 column description"