Repository: dbt-labs/dbt-core

Issue:
Generating docs take and long time
When running `dbt docs generate` is take over 10 mins to complete and uses lots of CPU resources producing the catalog.

When running with debug turned on the following gets written out:

```
2019-06-20 16:57:55,546 (MainThread): On "<None>": cache miss for schema "my_project.my_dataset", this is inefficient
```

along with:

```
2019-06-20 16:57:56,829 (MainThread): with schema=99439170, model_name=None, relations=[
<BigQueryRelation my_project.my_dataset.`cloudaudit_googleapis_com_data_access_20161114`>, 
<BigQueryRelation my_project.my_dataset.`cloudaudit_googleapis_com_data_access_20161115`>
.....
```

which is a date sharded table containing hundreds of shards.

The interesting thing is the `cloudaudit_googleapis_com_data_access` is the only one in the dataset that is not used or referenced by DBT. In the same dataset there is 2 other date sharded tables containing over a thousand shards, which are referenced as a source in DBT, but they are not printed out

```
sources:
  - name: google_analytics
    database: my_project
    schema: my_dataset
    loader: ga360
    tables:
      - name: ga_sessions_*
      - name: ga_sessions_intraday_*
```


Discussion:
Thanks for the report @whittid4 - we'll check it out! My initial thinking:
- high cpu usage sounds like an `agate` problem? I recall this being an issue with seeds, but not sure if that's exactly what's happening here
- we're probably iterating over the individual date shards, which is probably undersirable
I have run into a similar problem. I also have DBT setup to read from a BQ database with a lot of date sharded tables. Running `dbt docs generate` takes over an hour for me. There is no real noticable cpu load though

In the dbt logs the following is repeated over 50,000 times: 
```
019-06-26 15:34:11,918 (MainThread): /home/venv/lib/python3.6/site-packages/dbt/adapters/bigquery/impl.py:465: PendingDeprecationWarning: This method will be deprecated in future versions. Please use Table.time_partitioning.type_ instead.
```


Thanks for the additional info @EricLeer!

It's pretty clear to me that we need dbt to be smarter about catalog generation with date sharded tables on BQ. I don't think it makes sense for dbt to know about every single individual date shard -- these are probably source tables, and the BQ interface is probably suited for exploring these shards than dbt docs is, at least currently.

Presently, dbt is using an API method to fetch all of the tables/views in every dataset that dbt touches. This means that dbt will fundamentally need to pull down every single one of these tables, then maybe distill the date-sharded tables down to a single source table in-memory.

Maybe an alternative approach is to use BQ's new-ish information schema? I think with some clever SQL, we can push a lot of this filtering into the BQ layer. This should make `dbt docs generate` a lot snappier, as well as bring BQ in line with the rest of dbt's plugins (for the most part).

I just played around with some code here, what do you guys think of something like this?
```sql
with base as (

  select *,
    REGEXP_CONTAINS(table_name, '^.+[0-9]{8}$') as is_date_shard,
    REGEXP_EXTRACT(table_name, '^(.+)[0-9]{8}$') as base_name,
    REGEXP_EXTRACT(table_name, '^.+([0-9]{8})$') as shard_name

  FROM dbt_dbanin.INFORMATION_SCHEMA.TABLES

),

extracted as (

  select *,
    coalesce(base_name, table_name) as root_table_name

  from base

),

unsharded as (

  select
    table_catalog,
    table_schema,
    root_table_name as table_name,
    
    row_number() over (partition by root_table_name order by shard_name desc) as shard_index,
    min(shard_name) over (partition by root_table_name) as first_shard,
    max(shard_name) over (partition by root_table_name) as last_shard,
    count(*) over (partition by root_table_name) as num_shards

  from extracted
  
)

select *
from unsharded
where shard_index = 1
order by table_name
```

This query will find all of the tables/views in a given dataset, then squash down date shards into a single "root" record. We can pluck out the individual shard names if necessary, maybe aggregating them into an array in SQL? That would be pretty slick.

Curious to hear what you think!

cc @whittid4 
I think this would work. For one of my datasets this would reduce the amount of tables searched from 40000 to 400. 

On the other hand I think you would still run into the same problem if the tables are sharded in a different way then yyyymmdd date. In principle with [querying a wildcard table](https://cloud.google.com/bigquery/docs/querying-wildcard-tables) any pattern is possible and I think it would be best if this behaviour is also supported with this solution. Maybe the correct information of on what to colapse the shards can be gathered from the schema.yml file where the source is defined? For instance I have defined my source as:

```
name: dataset_name
    tables:
      - name: table_name
        identifier: table_name_*
``` 
and thus I would expect the shards to colapse on `table_name_` .

Finally what is the reason that dbt fetches all tables/views in a dataset it touches? Wouldn't it make more sense to only fetch the tables/views that are actually defined as a source?
That's a really good point - I didn't realize that you could shard on strings other than an 8-char date suffix. I agree - generating some sort of glob query from the specified `identifier` is probably a good idea.

dbt tries to find all of the tables in a given BQ dataset that match up with models and sources defined in the active dbt project. By querying for all of the relations in a dataset at once, dbt can make one query per dataset referenced in the dbt project. If dbt instead queried for each source/model individually, then dbt would need to execute orders of magnitude more queries like this in the general case. Imagine you had 30 source tables defined in a single dataset --  all things considered, I'd rather make one query against that dataset than 30!
Sorry for the delay in getting back to you.

@drewbanin, the example SQL you shared above will work for our case as we are only using the standard data sharding method, but as @EricLeer points out it would be nice to make this solution fit most use cases


I tried to point to a specific table during doc generation with dbt 0.14.2 but it didn't help.
It takes hours to process a referenced dataset with thousand (date) sharded tables even if you only reference a specific non-wildcard table in that dataset.
In my case it's a way too long to generate docs for both `dataset_1` and `dataset_2` (please see example YAML below). Both contain more than a thousand sharded tables.
``` yaml
sources:

  - name: dataset_1
    database: main_database
    schema: dataset_1_{{ var("tenant_id") }}
    tables:

      - name: table_1
        identifier: table_1_{{ var("tenant_id") }}_{{ "*" if not var("doc_gen", False) else "20190901" }}

  - name: dataset_2
    database: main_database
    schema: dataset_2
    tables:

      - name: table_2
        identifier: table_2
        columns:
          - name: table_id
```
Hey @mr2dark - I spent some time yesterday pulling together a more completed "catalog" query for BigQuery which leverages the `INFORMATION_SCHEMA`. TBD if we're able to slot that in for the 0.15.0 release (we removed it from the milestone, for now) but I'd love to ship it if we can!

Are you able/interested to help test out a branch of dbt which introduces this logic?
@drewbanin I'll be glad to help. How can I test that?

BTW it looks like my estimation is wrong. It can take 30 minutes and the cases where I didn't receive results for hours were probably caused by power/network state issues when my laptop went to sleep and/or was disconnected from network for a while. I'll try to reproduce that when I'll have some spare time.
But anyway building of a catalog for 30 minutes when you actually use a couple of tables feels like an overkill. 
Cool! I'll follow up with a branch name and some more info when I have something to show for myself :)
hey @mr2dark - check out the PR here: https://github.com/fishtown-analytics/dbt/pull/1795

This needs a little more love around automated testing, but the idea is there. Let me know if you're able to give it a spin locally. My hope is that the time to generate docs drops from 30 mins to a single-digit number of seconds.

Thanks!
I've left a comment in #1795

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -1,5 +1,3 @@
-import copy
-
 import dbt.deprecations
 import dbt.exceptions
 import dbt.flags as flags
@@ -387,127 +385,10 @@ def load_dataframe(self, database, schema, table_name, agate_table,
         with self.connections.exception_handler("LOAD TABLE"):
             self.poll_until_job_completes(job, timeout)
 
-    ###
-    # The get_catalog implementation for bigquery
-    ###
-    def _flat_columns_in_table(self, table):
-        """An iterator over the flattened columns for a given schema and table.
-        Resolves child columns as having the name "parent.child".
-        """
-        for col in self._get_dbt_columns_from_bq_table(table):
-            yield from col.flatten()
-
-    @classmethod
-    def _get_stats_column_names(cls):
-        """Construct a tuple of the column names for stats. Each stat has 4
-        columns of data.
-        """
-        columns = []
-        stats = ('num_bytes', 'num_rows', 'location', 'partitioning_type',
-                 'clustering_fields')
-        stat_components = ('label', 'value', 'description', 'include')
-        for stat_id in stats:
-            for stat_component in stat_components:
-                columns.append('stats:{}:{}'.format(stat_id, stat_component))
-        return tuple(columns)
-
-    @classmethod
-    def _get_stats_columns(cls, table, relation_type):
-        """Given a table, return an iterator of key/value pairs for stats
-        column names/values.
-        """
-        column_names = cls._get_stats_column_names()
-
-        # agate does not handle the array of column names gracefully
-        clustering_value = None
-        if table.clustering_fields is not None:
-            clustering_value = ','.join(table.clustering_fields)
-        # cast num_bytes/num_rows to str before they get to agate, or else
-        # agate will incorrectly decide they are booleans.
-        column_values = (
-            'Number of bytes',
-            str(table.num_bytes),
-            'The number of bytes this table consumes',
-            relation_type == 'table',
-
-            'Number of rows',
-            str(table.num_rows),
-            'The number of rows in this table',
-            relation_type == 'table',
-
-            'Location',
-            table.location,
-            'The geographic location of this table',
-            True,
-
-            'Partitioning Type',
-            table.partitioning_type,
-            'The partitioning type used for this table',
-            relation_type == 'table',
-
-            'Clustering Fields',
-            clustering_value,
-            'The clustering fields for this table',
-            relation_type == 'table',
-        )
-        return zip(column_names, column_values)
-
-    def get_catalog(self, manifest):
-        connection = self.connections.get_thread_connection()
-        client = connection.handle
-
-        schemas = manifest.get_used_schemas()
-
-        column_names = (
-            'table_database',
-            'table_schema',
-            'table_name',
-            'table_type',
-            'table_comment',
-            # does not exist in bigquery, but included for consistency
-            'table_owner',
-            'column_name',
-            'column_index',
-            'column_type',
-            'column_comment',
-        )
-        all_names = column_names + self._get_stats_column_names()
-        columns = []
+    def _catalog_filter_table(self, table, manifest):
+        # BigQuery doesn't allow ":" chars in column names -- remap them here.
+        table = table.rename(column_names={
+            col.name: col.name.replace('__', ':') for col in table.columns
+        })
 
-        for database_name, schema_name in schemas:
-            relations = self.list_relations(database_name, schema_name)
-            for relation in relations:
-
-                # This relation contains a subset of the info we care about.
-                # Fetch the full table object here
-                table_ref = self.connections.table_ref(
-                    database_name,
-                    relation.schema,
-                    relation.identifier,
-                    connection
-                )
-                table = client.get_table(table_ref)
-
-                flattened = self._flat_columns_in_table(table)
-                relation_stats = dict(self._get_stats_columns(table,
-                                                              relation.type))
-
-                for index, column in enumerate(flattened, start=1):
-                    column_data = (
-                        relation.database,
-                        relation.schema,
-                        relation.name,
-                        relation.type,
-                        None,
-                        None,
-                        column.name,
-                        index,
-                        column.data_type,
-                        None,
-                    )
-                    column_dict = dict(zip(column_names, column_data))
-                    column_dict.update(copy.deepcopy(relation_stats))
-
-                    columns.append(column_dict)
-
-        return dbt.clients.agate_helper.table_from_data(columns, all_names)
+        return super()._catalog_filter_table(table, manifest)
diff --git a/plugins/bigquery/dbt/adapters/bigquery/relation.py b/plugins/bigquery/dbt/adapters/bigquery/relation.py
--- a/plugins/bigquery/dbt/adapters/bigquery/relation.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/relation.py
@@ -5,6 +5,10 @@
     BaseRelation, ComponentName
 )
 from dbt.utils import filter_null_values
+from typing import TypeVar
+
+
+Self = TypeVar('Self', bound='BigQueryRelation')
 
 
 @dataclass(frozen=True, eq=False, repr=False)
@@ -40,3 +44,33 @@ def project(self):
     @property
     def dataset(self):
         return self.schema
+
+    def information_schema(self: Self, identifier=None) -> Self:
+        # BigQuery (usually) addresses information schemas at the dataset
+        # level. This method overrides the BaseRelation method to return an
+        # Information Schema relation as project.dataset.information_schem
+
+        include_policy = self.include_policy.replace(
+            database=self.database is not None,
+            schema=self.schema is not None,
+            identifier=True
+        )
+
+        # Quote everything on BigQuery -- identifiers are case-sensitive,
+        # even when quoted.
+        quote_policy = self.quote_policy.replace(
+            database=True,
+            schema=True,
+            identifier=True,
+        )
+
+        path = self.path.replace(
+            schema=self.schema,
+            identifier='INFORMATION_SCHEMA'
+        )
+
+        return self.replace(
+            quote_policy=quote_policy,
+            include_policy=include_policy,
+            path=path,
+        )


Test patch:
diff --git a/test/integration/029_docs_generate_tests/test_docs_generate.py b/test/integration/029_docs_generate_tests/test_docs_generate.py
--- a/test/integration/029_docs_generate_tests/test_docs_generate.py
+++ b/test/integration/029_docs_generate_tests/test_docs_generate.py
@@ -5,7 +5,7 @@
 from unittest.mock import ANY, patch
 
 from test.integration.base import DBTIntegrationTest, use_profile, AnyFloat, \
-    AnyStringWith, normalize
+    AnyString, AnyStringWith, normalize
 
 
 def _read_file(path):
@@ -225,39 +225,48 @@ def _bigquery_stats(self, is_table, partition=None, cluster=None):
                 'value': 'US',
                 'description': 'The geographic location of this table',
                 'include': True,
-            },
+            }
         }
         if is_table:
             stats.update({
                 'num_bytes': {
                     'id': 'num_bytes',
-                    'label': 'Number of bytes',
+                    'label': AnyString(),
                     'value': AnyFloat(),
-                    'description': 'The number of bytes this table consumes',
+                    'description': AnyString(),
                     'include': True,
                 },
                 'num_rows': {
                     'id': 'num_rows',
-                    'label': 'Number of rows',
+                    'label': AnyString(),
                     'value': AnyFloat(),
-                    'description': 'The number of rows in this table',
+                    'description': AnyString(),
                     'include': True,
-                },
+                }
+            })
+
+        if partition is not None:
+            stats.update({
                 'partitioning_type': {
                     'id': 'partitioning_type',
-                    'label': 'Partitioning Type',
+                    'label': AnyString(),
                     'value': partition,
-                    'description': 'The partitioning type used for this table',
-                    'include': True,
-                },
+                    'description': AnyString(),
+                    'include': True
+                }
+            })
+
+        if cluster is not None:
+            stats.update({
                 'clustering_fields': {
                     'id': 'clustering_fields',
-                    'label': 'Clustering Fields',
+                    'label': AnyString(),
                     'value': cluster,
-                    'description': 'The clustering fields for this table',
-                    'include': True,
-                },
+                    'description': AnyString(),
+                    'include': True
+                }
             })
+
         return stats
 
     def _expected_catalog(self, id_type, text_type, time_type, view_type,
@@ -535,10 +544,10 @@ def expected_bigquery_complex_catalog(self):
         my_schema_name = self.unique_schema()
         role = self.get_role()
         table_stats = self._bigquery_stats(True)
-        clustering_stats = self._bigquery_stats(True, partition='DAY',
+        clustering_stats = self._bigquery_stats(True, partition='updated_at',
                                                 cluster='first_name')
-        multi_clustering_stats = self._bigquery_stats(True, partition='DAY',
-                                                      cluster='first_name,email')
+        multi_clustering_stats = self._bigquery_stats(True, partition='updated_at',
+                                                      cluster='first_name, email')
         nesting_columns = {
             'field_1': {
                 'name': 'field_1',
diff --git a/test/integration/base.py b/test/integration/base.py
--- a/test/integration/base.py
+++ b/test/integration/base.py
@@ -1157,6 +1157,13 @@ def __eq__(self, other):
         return isinstance(other, float)
 
 
+class AnyString:
+    """Any string. Use this in assertEqual() calls to assert that it is a string.
+    """
+    def __eq__(self, other):
+        return isinstance(other, str)
+
+
 class AnyStringWith:
     def __init__(self, contains=None):
         self.contains = contains