Repository: dbt-labs/dbt-core

Issue:
YAML implicitly converts fields to other datatypes
### Describe the bug

With Google Analytics, some tables are numeric, these table names are automagically converted to bool/int/Decimal.

```
MacBook-Pro-van-Fokko:dbt-source fokkodriesprong$ python3
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 26 2018, 19:50:54)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>>
>>> yaml.load("name: 19.25")
{'name': 19.25}
>>> data = yaml.load("name: 19.25")
>>> type(data['name'])
<class 'float'>

>>> data = yaml.load("name: 1925")
>>> type(data['name'])
<class 'int'>

>>> data = yaml.load("name: 'no'")
>>> type(data['name'])
<class 'str'>

>>> data = yaml.load("name: no")
>>> type(data['name'])
<class 'bool'>
```

### Steps To Reproduce

Have a source with a Decimal name.
```
  - name: 19.25
    schema: "{{ var('test_run_schema') }}"
    quoting:
      identifier: True
    tables:
      - name: test_table
        identifier: other_source_table
```

### Screenshots and log output

```
2020-03-02 09:10:33,888249 (MainThread): 'decimal.Decimal' object has no attribute 'lower'
2020-03-02 09:10:33,892358 (MainThread): Traceback (most recent call last):
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/main.py", line 80, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/main.py", line 158, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/main.py", line 210, in run_from_args
    results = task.run()
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/task/generate.py", line 201, in run
    catalog_table = adapter.get_catalog(self.manifest)
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/adapters/base/impl.py", line 1016, in get_catalog
    results = self._catalog_filter_table(table, manifest)
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/adapters/bigquery/impl.py", line 493, in _catalog_filter_table
    return super()._catalog_filter_table(table, manifest)
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/adapters/base/impl.py", line 998, in _catalog_filter_table
    return table.where(_catalog_filter_schemas(manifest))
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/agate/table/where.py", line 25, in where
    if test(row):
  File "/usr/local/Cellar/dbt/0.15.2_1/libexec/lib/python3.7/site-packages/dbt/adapters/base/impl.py", line 82, in test
    return (table_database.lower(), table_schema.lower()) in schemas
AttributeError: 'decimal.Decimal' object has no attribute 'lower'
```

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [x] bigquery
- [ ] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
2020-03-02 09:10:29,518713 (MainThread): On generate_catalog: /* {"app": "dbt", "dbt_version": "0.15.2", "profile_name": "XXXX", "target_name": "prod", "connection_name": "generate_catalog"} */
```

**The operating system you're using:**

OSX

**The output of `python --version`:**

```
MacBook-Pro-van-Fokko:dbt-source fokkodriesprong$ python3 --version
Python 3.6.6
```


Discussion:
hey @Fokko - thanks for opening this issue! I see how a numeric table name could cause issues with the code that dbt uses to operate on relations:
```
return (table_database.lower(), table_schema.lower()) in schemas
```

Do you have control over the schema.yml files that declare these sources? My recommendation would be to supply the table name as a string, rather than as a decimal. You're not seeing implicit conversion in the example shared above:
```
  - name: 19.25
    schema: "{{ var('test_run_schema') }}"
    quoting:
      identifier: True
    tables:
      - name: test_table
        identifier: other_source_table
```

In this case, you're just supplying a number where you should be supplying a string. Can you do something like:
```
  - name: "19.25" # <----- note the quotes here!
    schema: "{{ var('test_run_schema') }}"
    quoting:
      identifier: True
    tables:
      - name: test_table
        identifier: other_source_table
```

If I understand everything correctly, this should solve your problem! 
Thanks for pointing this out Drew. Actually I'm helping someone on the devlist: https://getdbt.slack.com/archives/C2JRRQDTL/p1582909962173000

The weird thing that I'm not able to reproduce the issue. It seems that there is already code in place that enforces this, hence the failing CI.
That person is me 👋.

I have the same "System connection" except I'm using Python 3.7.6.

I have stripped the project right back to have:

- The out-the-box `dbt_project.yml` template that links to my `~/.dbt/profiles.yml` file
- An empty `schema.yml` file
- A single model that uses no references like so:

```
SELECT * FROM `frontend-data.200111222.sessions_*`
```

But still running `dbt docs generate` fails with the same:

```
Encountered an error:
'decimal.Decimal' object has no attribute 'lower'
```
I’ve installed 0.16.0-rc2. When I run `dbt docs generate` I still get the same error however it also is able to write down the `catalog.json` file and therefore running `dbt docs serve` works:
```
✗ dbt docs generate
Running with dbt=0.16.0-rc2
Found 5 models, 32 tests, 0 snapshots, 0 analyses, 268 macros, 0 operations, 0 seed files, 1 source
11:56:13 | Concurrency: 1 threads (target='prod')
11:56:13 |
11:56:14 | Done.
11:56:14 | Building catalog
Encountered an error while generating catalog: 'decimal.Decimal' object has no attribute 'lower'
dbt encountered 1 failure while writing the catalog
11:56:26 | Catalog written to /Users/DbtProjects/data-pipelines/frontend/target/catalog.json
```

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -3,7 +3,7 @@
 from contextlib import contextmanager
 from datetime import datetime
 from typing import (
-    Optional, Tuple, Callable, Container, FrozenSet, Type, Dict, Any, List,
+    Optional, Tuple, Callable, Iterable, FrozenSet, Type, Dict, Any, List,
     Mapping, Iterator, Union, Set
 )
 
@@ -18,7 +18,7 @@
 import dbt.flags
 
 from dbt import deprecations
-from dbt.clients.agate_helper import empty_table, merge_tables
+from dbt.clients.agate_helper import empty_table, merge_tables, table_from_rows
 from dbt.clients.jinja import MacroGenerator
 from dbt.contracts.graph.compiled import CompileResultNode, CompiledSeedNode
 from dbt.contracts.graph.manifest import Manifest
@@ -53,21 +53,6 @@ def _expect_row_value(key: str, row: agate.Row):
     return row[key]
 
 
-def _relations_filter_schemas(
-    schemas: Container[str]
-) -> Callable[[agate.Row], bool]:
-    def test(row):
-        referenced_schema = _expect_row_value('referenced_schema', row)
-        dependent_schema = _expect_row_value('dependent_schema', row)
-        # handle the null schema
-        if referenced_schema is not None:
-            referenced_schema = referenced_schema.lower()
-        if dependent_schema is not None:
-            dependent_schema = dependent_schema.lower()
-        return referenced_schema in schemas or dependent_schema in schemas
-    return test
-
-
 def _catalog_filter_schemas(manifest: Manifest) -> Callable[[agate.Row], bool]:
     """Return a function that takes a row and decides if the row should be
     included in the catalog output.
@@ -913,6 +898,7 @@ def execute_macro(
         context_override: Optional[Dict[str, Any]] = None,
         kwargs: Dict[str, Any] = None,
         release: bool = False,
+        text_only_columns: Optional[Iterable[str]] = None,
     ) -> agate.Table:
         """Look macro_name up in the manifest and execute its results.
 
@@ -976,6 +962,12 @@ def _catalog_filter_table(
         """Filter the table as appropriate for catalog entries. Subclasses can
         override this to change filtering rules on a per-adapter basis.
         """
+        # force database + schema to be strings
+        table = table_from_rows(
+            table.rows,
+            table.column_names,
+            text_only_columns=['table_database', 'table_schema', 'table_name']
+        )
         return table.where(_catalog_filter_schemas(manifest))
 
     def _get_one_catalog(
diff --git a/core/dbt/clients/agate_helper.py b/core/dbt/clients/agate_helper.py
--- a/core/dbt/clients/agate_helper.py
+++ b/core/dbt/clients/agate_helper.py
@@ -4,7 +4,7 @@
 import datetime
 import isodate
 import json
-from typing import Iterable, List, Dict, Union
+from typing import Iterable, List, Dict, Union, Optional, Any
 
 from dbt.exceptions import RuntimeException
 
@@ -34,7 +34,7 @@ def cast(self, d):
         )
 
 
-def build_type_tester(text_columns: Iterable[str]):
+def build_type_tester(text_columns: Iterable[str]) -> agate.TypeTester:
     types = [
         agate.data_types.Number(null_values=('null', '')),
         agate.data_types.Date(null_values=('null', ''),
@@ -57,7 +57,19 @@ def build_type_tester(text_columns: Iterable[str]):
 DEFAULT_TYPE_TESTER = build_type_tester(())
 
 
-def table_from_data(data, column_names):
+def table_from_rows(
+    rows: List[Any],
+    column_names: Iterable[str],
+    text_only_columns: Optional[Iterable[str]] = None,
+) -> agate.Table:
+    if text_only_columns is None:
+        column_types = DEFAULT_TYPE_TESTER
+    else:
+        column_types = build_type_tester(text_only_columns)
+    return agate.Table(rows, column_names, column_types=column_types)
+
+
+def table_from_data(data, column_names: Iterable[str]) -> agate.Table:
     "Convert list of dictionaries into an Agate table"
 
     # The agate table is generated from a list of dicts, so the column order
@@ -72,7 +84,7 @@ def table_from_data(data, column_names):
         return table.select(column_names)
 
 
-def table_from_data_flat(data, column_names):
+def table_from_data_flat(data, column_names: Iterable[str]) -> agate.Table:
     "Convert list of dictionaries into an Agate table"
 
     rows = []
@@ -85,7 +97,7 @@ def table_from_data_flat(data, column_names):
                 row.append(value)
         rows.append(row)
 
-    return agate.Table(rows, column_names, column_types=DEFAULT_TYPE_TESTER)
+    return table_from_rows(rows=rows, column_names=column_names)
 
 
 def empty_table():


Test patch:
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -1,3 +1,5 @@
+import agate
+import decimal
 import re
 import unittest
 from contextlib import contextmanager
@@ -12,6 +14,7 @@
 from dbt.adapters.bigquery import BigQueryRelation
 from dbt.adapters.bigquery.connections import BigQueryConnectionManager
 from dbt.adapters.base.query_headers import MacroQueryStringSetter
+from dbt.clients import agate_helper
 import dbt.exceptions
 from dbt.logger import GLOBAL_LOGGER as logger  # noqa
 
@@ -446,3 +449,27 @@ def test_parse_partition_by(self):
                 }
             }
         )
+
+
+class TestBigQueryFilterCatalog(unittest.TestCase):
+    def test__catalog_filter_table(self):
+        manifest = MagicMock()
+        manifest.get_used_schemas.return_value = [['a', 'B'], ['a', '1234']]
+        column_names = ['table_name', 'table_database', 'table_schema', 'something']
+        rows = [
+            ['foo', 'a', 'b', '1234'],  # include
+            ['foo', 'a', '1234', '1234'],  # include, w/ table schema as str
+            ['foo', 'c', 'B', '1234'],  # skip
+            ['1234', 'A', 'B', '1234'],  # include, w/ table name as str
+        ]
+        table = agate.Table(
+            rows, column_names, agate_helper.DEFAULT_TYPE_TESTER
+        )
+
+        result = BigQueryAdapter._catalog_filter_table(table, manifest)
+        assert len(result) == 3
+        for row in result.rows:
+            assert isinstance(row['table_schema'], str)
+            assert isinstance(row['table_database'], str)
+            assert isinstance(row['table_name'], str)
+            assert isinstance(row['something'], decimal.Decimal)
diff --git a/test/unit/test_postgres_adapter.py b/test/unit/test_postgres_adapter.py
--- a/test/unit/test_postgres_adapter.py
+++ b/test/unit/test_postgres_adapter.py
@@ -1,3 +1,5 @@
+import agate
+import decimal
 import unittest
 from unittest import mock
 
@@ -6,13 +8,12 @@
 
 from dbt.adapters.base.query_headers import MacroQueryStringSetter
 from dbt.adapters.postgres import PostgresAdapter
-from dbt.config import ConfigRenderer
+from dbt.clients import agate_helper
 from dbt.exceptions import ValidationException, DbtConfigError
 from dbt.logger import GLOBAL_LOGGER as logger  # noqa
 from dbt.parser.results import ParseResult
 from psycopg2 import extensions as psycopg2_extensions
 from psycopg2 import DatabaseError
-import agate
 
 from .utils import config_from_parts_or_dicts, inject_adapter, mock_connection
 
@@ -394,3 +395,27 @@ def test_dbname_verification_is_case_insensitive(self):
         self.adapter.cleanup_connections()
         self._adapter = PostgresAdapter(self.config)
         self.adapter.verify_database('postgres')
+
+
+class TestPostgresFilterCatalog(unittest.TestCase):
+    def test__catalog_filter_table(self):
+        manifest = mock.MagicMock()
+        manifest.get_used_schemas.return_value = [['a', 'B'], ['a', '1234']]
+        column_names = ['table_name', 'table_database', 'table_schema', 'something']
+        rows = [
+            ['foo', 'a', 'b', '1234'],  # include
+            ['foo', 'a', '1234', '1234'],  # include, w/ table schema as str
+            ['foo', 'c', 'B', '1234'],  # skip
+            ['1234', 'A', 'B', '1234'],  # include, w/ table name as str
+        ]
+        table = agate.Table(
+            rows, column_names, agate_helper.DEFAULT_TYPE_TESTER
+        )
+
+        result = PostgresAdapter._catalog_filter_table(table, manifest)
+        assert len(result) == 3
+        for row in result.rows:
+            assert isinstance(row['table_schema'], str)
+            assert isinstance(row['table_database'], str)
+            assert isinstance(row['table_name'], str)
+            assert isinstance(row['something'], decimal.Decimal)