Repository: dbt-labs/dbt-core

Issue:
ValueError when column_types defined in dbt_project.yaml but null/empty shows up in seed file
### Describe the bug
`dbt seed --full-refresh` caused the issue and resulted in `max() arg is an empty sequence`.

### Steps To Reproduce
Provide seed definitions in `dbt_project.yaml` with something along the lines of :
```
seeds:
  project_name:
    enabled: true
    schema: seeds_schema
    foo_barr_seed:
      column_types:
        column_that_breaks: varchar(128)
```

Then create a seed file where at least one row has an empty string "" in the `column_that_breaks` column.

The final line in the debug log file from dbt cloud shows that a null value isn't being handled well in Python:
`max_len = max(lens) if lens else 64` 

### Expected behavior
`dbt seed` should either handle null/empty values when a column_type is defined or exit with a more graceful message

### Screenshots and log output
Full log attached. Screenshot of the last lines of the stack trace as well as the error below:
[debug (1).log](https://github.com/fishtown-analytics/dbt/files/4388935/debug.1.log)
<img width="800" alt="Screen Shot 2020-03-26 at 10 50 47 AM" src="https://user-images.githubusercontent.com/22972908/77679606-00d13980-6f50-11ea-9931-3916e89d0a51.png">
<img width="1442" alt="Screen Shot 2020-03-26 at 10 50 54 AM" src="https://user-images.githubusercontent.com/22972908/77679607-029afd00-6f50-11ea-9425-ce6c336e5b97.png">


### System information
**Which database are you using dbt with?**
- [ ] postgres
- [x] redshift
- [ ] bigquery
- [ ] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
installed version: 0.16.0
   latest version: 0.16.0

Up to date!
```

**The operating system you're using:**
macOS Catalina
10.15.3

Also fails in dbt cloud.

**The output of `python --version`:**
```Python 3.7.3```

### Additional context
From the #support channel in the dbt slack community: https://getdbt.slack.com/archives/C2JRRQDTL/p1585243307097400


As an addition, thanks to you all for maintaining an amazing and responsive slack community. I always have confidence that I'll get help whenever I have an issue with dbt. Much love from Envoy!


Discussion:
triage note: we can fix this issue by just making `lens` a list comprehension instead of a generator expression, but this also exposes a need to unit test these conversion methods - this seems like an easy fix to inadvertently revert!
Thanks for this really thorough report @jasonwang592! 

@beckjake should we sneak this in for 0.16.1 or is this an 0.17.0 fix? I'm mostly curious if it's an 0.16.0 regression or a long-standing bug
@drewbanin If it's a regression, it's of the "other changes to dbt exposed an existing bad behavior" variety, because the code in question hasn't changed. The user on slack reported it in the context of "I updated to dbt 0.16.0 and ...", and it's believable that the issue was exposed by the changes to how dbt handles seeds with user-defined types in 0.16.0.

I think we should get it into 0.16.1 regardless - it seems like it's a pretty trivial fix with low/no risk. 
Cool, sounds good to me, just added this to the 0.16.1 milestone - let's make it happen

Resolving patch:
diff --git a/plugins/redshift/dbt/adapters/redshift/impl.py b/plugins/redshift/dbt/adapters/redshift/impl.py
--- a/plugins/redshift/dbt/adapters/redshift/impl.py
+++ b/plugins/redshift/dbt/adapters/redshift/impl.py
@@ -35,7 +35,9 @@ def drop_relation(self, relation):
     @classmethod
     def convert_text_type(cls, agate_table, col_idx):
         column = agate_table.columns[col_idx]
-        lens = (len(d.encode("utf-8")) for d in column.values_without_nulls())
+        # `lens` must be a list, so this can't be a generator expression,
+        # because max() raises ane exception if its argument has no members.
+        lens = [len(d.encode("utf-8")) for d in column.values_without_nulls()]
         max_len = max(lens) if lens else 64
         return "varchar({})".format(max_len)
 


Test patch:
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -18,7 +18,7 @@
 import dbt.exceptions
 from dbt.logger import GLOBAL_LOGGER as logger  # noqa
 
-from .utils import config_from_parts_or_dicts, inject_adapter
+from .utils import config_from_parts_or_dicts, inject_adapter, TestAdapterConversions
 
 
 def _bq_conn():
@@ -473,3 +473,73 @@ def test__catalog_filter_table(self):
             assert isinstance(row['table_database'], str)
             assert isinstance(row['table_name'], str)
             assert isinstance(row['something'], decimal.Decimal)
+
+
+class TestBigQueryAdapterConversions(TestAdapterConversions):
+    def test_convert_text_type(self):
+        rows = [
+            ['', 'a1', 'stringval1'],
+            ['', 'a2', 'stringvalasdfasdfasdfa'],
+            ['', 'a3', 'stringval3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Text)
+        expected = ['string', 'string', 'string']
+        for col_idx, expect in enumerate(expected):
+            assert BigQueryAdapter.convert_text_type(agate_table, col_idx) == expect
+
+    def test_convert_number_type(self):
+        rows = [
+            ['', '23.98', '-1'],
+            ['', '12.78', '-2'],
+            ['', '79.41', '-3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Number)
+        expected = ['int64', 'float64', 'int64']
+        for col_idx, expect in enumerate(expected):
+            assert BigQueryAdapter.convert_number_type(agate_table, col_idx) == expect
+
+    def test_convert_boolean_type(self):
+        rows = [
+            ['', 'false', 'true'],
+            ['', 'false', 'false'],
+            ['', 'false', 'true'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Boolean)
+        expected = ['bool', 'bool', 'bool']
+        for col_idx, expect in enumerate(expected):
+            assert BigQueryAdapter.convert_boolean_type(agate_table, col_idx) == expect
+
+    def test_convert_datetime_type(self):
+        rows = [
+            ['', '20190101T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190102T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190103T01:01:01Z', '2019-01-01 01:01:01'],
+        ]
+        agate_table = self._make_table_of(rows, [agate.DateTime, agate_helper.ISODateTime, agate.DateTime])
+        expected = ['datetime', 'datetime', 'datetime']
+        for col_idx, expect in enumerate(expected):
+            assert BigQueryAdapter.convert_datetime_type(agate_table, col_idx) == expect
+
+    def test_convert_date_type(self):
+        rows = [
+            ['', '2019-01-01', '2019-01-04'],
+            ['', '2019-01-02', '2019-01-04'],
+            ['', '2019-01-03', '2019-01-04'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Date)
+        expected = ['date', 'date', 'date']
+        for col_idx, expect in enumerate(expected):
+            assert BigQueryAdapter.convert_date_type(agate_table, col_idx) == expect
+
+    def test_convert_time_type(self):
+        # dbt's default type testers actually don't have a TimeDelta at all.
+        agate.TimeDelta
+        rows = [
+            ['', '120s', '10s'],
+            ['', '3m', '11s'],
+            ['', '1h', '12s'],
+        ]
+        agate_table = self._make_table_of(rows, agate.TimeDelta)
+        expected = ['time', 'time', 'time']
+        for col_idx, expect in enumerate(expected):
+            assert BigQueryAdapter.convert_time_type(agate_table, col_idx) == expect
diff --git a/test/unit/test_postgres_adapter.py b/test/unit/test_postgres_adapter.py
--- a/test/unit/test_postgres_adapter.py
+++ b/test/unit/test_postgres_adapter.py
@@ -15,7 +15,7 @@
 from psycopg2 import extensions as psycopg2_extensions
 from psycopg2 import DatabaseError
 
-from .utils import config_from_parts_or_dicts, inject_adapter, mock_connection
+from .utils import config_from_parts_or_dicts, inject_adapter, mock_connection, TestAdapterConversions
 
 
 class TestPostgresAdapter(unittest.TestCase):
@@ -419,3 +419,73 @@ def test__catalog_filter_table(self):
             assert isinstance(row['table_database'], str)
             assert isinstance(row['table_name'], str)
             assert isinstance(row['something'], decimal.Decimal)
+
+
+class TestPostgresAdapterConversions(TestAdapterConversions):
+    def test_convert_text_type(self):
+        rows = [
+            ['', 'a1', 'stringval1'],
+            ['', 'a2', 'stringvalasdfasdfasdfa'],
+            ['', 'a3', 'stringval3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Text)
+        expected = ['text', 'text', 'text']
+        for col_idx, expect in enumerate(expected):
+            assert PostgresAdapter.convert_text_type(agate_table, col_idx) == expect
+
+    def test_convert_number_type(self):
+        rows = [
+            ['', '23.98', '-1'],
+            ['', '12.78', '-2'],
+            ['', '79.41', '-3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Number)
+        expected = ['integer', 'float8', 'integer']
+        for col_idx, expect in enumerate(expected):
+            assert PostgresAdapter.convert_number_type(agate_table, col_idx) == expect
+
+    def test_convert_boolean_type(self):
+        rows = [
+            ['', 'false', 'true'],
+            ['', 'false', 'false'],
+            ['', 'false', 'true'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Boolean)
+        expected = ['boolean', 'boolean', 'boolean']
+        for col_idx, expect in enumerate(expected):
+            assert PostgresAdapter.convert_boolean_type(agate_table, col_idx) == expect
+
+    def test_convert_datetime_type(self):
+        rows = [
+            ['', '20190101T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190102T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190103T01:01:01Z', '2019-01-01 01:01:01'],
+        ]
+        agate_table = self._make_table_of(rows, [agate.DateTime, agate_helper.ISODateTime, agate.DateTime])
+        expected = ['timestamp without time zone', 'timestamp without time zone', 'timestamp without time zone']
+        for col_idx, expect in enumerate(expected):
+            assert PostgresAdapter.convert_datetime_type(agate_table, col_idx) == expect
+
+    def test_convert_date_type(self):
+        rows = [
+            ['', '2019-01-01', '2019-01-04'],
+            ['', '2019-01-02', '2019-01-04'],
+            ['', '2019-01-03', '2019-01-04'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Date)
+        expected = ['date', 'date', 'date']
+        for col_idx, expect in enumerate(expected):
+            assert PostgresAdapter.convert_date_type(agate_table, col_idx) == expect
+
+    def test_convert_time_type(self):
+        # dbt's default type testers actually don't have a TimeDelta at all.
+        agate.TimeDelta
+        rows = [
+            ['', '120s', '10s'],
+            ['', '3m', '11s'],
+            ['', '1h', '12s'],
+        ]
+        agate_table = self._make_table_of(rows, agate.TimeDelta)
+        expected = ['time', 'time', 'time']
+        for col_idx, expect in enumerate(expected):
+            assert PostgresAdapter.convert_time_type(agate_table, col_idx) == expect
diff --git a/test/unit/test_redshift_adapter.py b/test/unit/test_redshift_adapter.py
--- a/test/unit/test_redshift_adapter.py
+++ b/test/unit/test_redshift_adapter.py
@@ -1,15 +1,17 @@
+import string
 import unittest
 from unittest import mock
+import agate
 
-import dbt.adapters
+import dbt.adapters  # noqa
 import dbt.flags as flags
-import dbt.utils
 
 from dbt.adapters.redshift import RedshiftAdapter
+from dbt.clients import agate_helper
 from dbt.exceptions import FailedToConnectException
 from dbt.logger import GLOBAL_LOGGER as logger  # noqa
 
-from .utils import config_from_parts_or_dicts, mock_connection
+from .utils import config_from_parts_or_dicts, mock_connection, TestAdapterConversions
 
 
 @classmethod
@@ -212,7 +214,7 @@ def test_search_path_with_space(self, psycopg2):
             password='password',
             port=5439,
             connect_timeout=10,
-            options="-c search_path=test\ test",
+            options=r"-c search_path=test\ test",
             keepalives_idle=240)
 
     @mock.patch('dbt.adapters.postgres.connections.psycopg2')
@@ -261,3 +263,79 @@ def test_dbname_verification_is_case_insensitive(self):
         self.adapter.cleanup_connections()
         self._adapter = RedshiftAdapter(self.config)
         self.adapter.verify_database('redshift')
+
+
+class TestRedshiftAdapterConversions(TestAdapterConversions):
+    def test_convert_text_type(self):
+        rows = [
+            ['', 'a1', 'stringval1'],
+            ['', 'a2', 'stringvalasdfasdfasdfa'],
+            ['', 'a3', 'stringval3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Text)
+        expected = ['varchar(64)', 'varchar(2)', 'varchar(22)']
+        for col_idx, expect in enumerate(expected):
+            assert RedshiftAdapter.convert_text_type(agate_table, col_idx) == expect
+
+    def test_convert_number_type(self):
+        rows = [
+            ['', '23.98', '-1'],
+            ['', '12.78', '-2'],
+            ['', '79.41', '-3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Number)
+        expected = ['integer', 'float8', 'integer']
+        for col_idx, expect in enumerate(expected):
+            assert RedshiftAdapter.convert_number_type(agate_table, col_idx) == expect
+
+    def test_convert_boolean_type(self):
+        rows = [
+            ['', 'false', 'true'],
+            ['', 'false', 'false'],
+            ['', 'false', 'true'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Boolean)
+        expected = ['boolean', 'boolean', 'boolean']
+        for col_idx, expect in enumerate(expected):
+            assert RedshiftAdapter.convert_boolean_type(agate_table, col_idx) == expect
+
+    def test_convert_datetime_type(self):
+        rows = [
+            ['', '20190101T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190102T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190103T01:01:01Z', '2019-01-01 01:01:01'],
+        ]
+        agate_table = self._make_table_of(rows, [agate.DateTime, agate_helper.ISODateTime, agate.DateTime])
+        expected = ['timestamp without time zone', 'timestamp without time zone', 'timestamp without time zone']
+        for col_idx, expect in enumerate(expected):
+            assert RedshiftAdapter.convert_datetime_type(agate_table, col_idx) == expect
+
+    def test_convert_date_type(self):
+        rows = [
+            ['', '2019-01-01', '2019-01-04'],
+            ['', '2019-01-02', '2019-01-04'],
+            ['', '2019-01-03', '2019-01-04'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Date)
+        expected = ['date', 'date', 'date']
+        for col_idx, expect in enumerate(expected):
+            assert RedshiftAdapter.convert_date_type(agate_table, col_idx) == expect
+
+    def test_convert_time_type(self):
+        # dbt's default type testers actually don't have a TimeDelta at all.
+        agate.TimeDelta
+        rows = [
+            ['', '120s', '10s'],
+            ['', '3m', '11s'],
+            ['', '1h', '12s'],
+        ]
+        agate_table = self._make_table_of(rows, agate.TimeDelta)
+        expected = ['varchar(24)', 'varchar(24)', 'varchar(24)']
+        for col_idx, expect in enumerate(expected):
+            assert RedshiftAdapter.convert_time_type(agate_table, col_idx) == expect
+
+
+# convert_boolean_type
+# convert_datetime_type
+# convert_date_type
+# convert_time_type
diff --git a/test/unit/test_snowflake_adapter.py b/test/unit/test_snowflake_adapter.py
--- a/test/unit/test_snowflake_adapter.py
+++ b/test/unit/test_snowflake_adapter.py
@@ -1,3 +1,4 @@
+import agate
 import unittest
 from contextlib import contextmanager
 from unittest import mock
@@ -6,11 +7,12 @@
 
 from dbt.adapters.snowflake import SnowflakeAdapter
 from dbt.adapters.base.query_headers import MacroQueryStringSetter
+from dbt.clients import agate_helper
 from dbt.logger import GLOBAL_LOGGER as logger  # noqa
 from dbt.parser.results import ParseResult
 from snowflake import connector as snowflake_connector
 
-from .utils import config_from_parts_or_dicts, inject_adapter, mock_connection
+from .utils import config_from_parts_or_dicts, inject_adapter, mock_connection, TestAdapterConversions
 
 
 class TestSnowflakeAdapter(unittest.TestCase):
@@ -376,3 +378,73 @@ def test_authenticator_private_key_authentication_no_passphrase(self, mock_get_p
                 warehouse='test_warehouse', private_key='test_key',
                 application='dbt')
         ])
+
+
+class TestSnowflakeAdapterConversions(TestAdapterConversions):
+    def test_convert_text_type(self):
+        rows = [
+            ['', 'a1', 'stringval1'],
+            ['', 'a2', 'stringvalasdfasdfasdfa'],
+            ['', 'a3', 'stringval3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Text)
+        expected = ['text', 'text', 'text']
+        for col_idx, expect in enumerate(expected):
+            assert SnowflakeAdapter.convert_text_type(agate_table, col_idx) == expect
+
+    def test_convert_number_type(self):
+        rows = [
+            ['', '23.98', '-1'],
+            ['', '12.78', '-2'],
+            ['', '79.41', '-3'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Number)
+        expected = ['integer', 'float8', 'integer']
+        for col_idx, expect in enumerate(expected):
+            assert SnowflakeAdapter.convert_number_type(agate_table, col_idx) == expect
+
+    def test_convert_boolean_type(self):
+        rows = [
+            ['', 'false', 'true'],
+            ['', 'false', 'false'],
+            ['', 'false', 'true'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Boolean)
+        expected = ['boolean', 'boolean', 'boolean']
+        for col_idx, expect in enumerate(expected):
+            assert SnowflakeAdapter.convert_boolean_type(agate_table, col_idx) == expect
+
+    def test_convert_datetime_type(self):
+        rows = [
+            ['', '20190101T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190102T01:01:01Z', '2019-01-01 01:01:01'],
+            ['', '20190103T01:01:01Z', '2019-01-01 01:01:01'],
+        ]
+        agate_table = self._make_table_of(rows, [agate.DateTime, agate_helper.ISODateTime, agate.DateTime])
+        expected = ['timestamp without time zone', 'timestamp without time zone', 'timestamp without time zone']
+        for col_idx, expect in enumerate(expected):
+            assert SnowflakeAdapter.convert_datetime_type(agate_table, col_idx) == expect
+
+    def test_convert_date_type(self):
+        rows = [
+            ['', '2019-01-01', '2019-01-04'],
+            ['', '2019-01-02', '2019-01-04'],
+            ['', '2019-01-03', '2019-01-04'],
+        ]
+        agate_table = self._make_table_of(rows, agate.Date)
+        expected = ['date', 'date', 'date']
+        for col_idx, expect in enumerate(expected):
+            assert SnowflakeAdapter.convert_date_type(agate_table, col_idx) == expect
+
+    def test_convert_time_type(self):
+        # dbt's default type testers actually don't have a TimeDelta at all.
+        agate.TimeDelta
+        rows = [
+            ['', '120s', '10s'],
+            ['', '3m', '11s'],
+            ['', '1h', '12s'],
+        ]
+        agate_table = self._make_table_of(rows, agate.TimeDelta)
+        expected = ['time', 'time', 'time']
+        for col_idx, expect in enumerate(expected):
+            assert SnowflakeAdapter.convert_time_type(agate_table, col_idx) == expect
diff --git a/test/unit/utils.py b/test/unit/utils.py
--- a/test/unit/utils.py
+++ b/test/unit/utils.py
@@ -3,10 +3,12 @@
 Note that all imports should be inside the functions to avoid import/mocking
 issues.
 """
+import string
 import os
 from unittest import mock
 from unittest import TestCase
 
+import agate
 from hologram import ValidationError
 
 
@@ -163,3 +165,25 @@ def generate_name_macros(package):
             macro_sql=sql,
         )
         yield pm
+
+
+class TestAdapterConversions(TestCase):
+    def _get_tester_for(self, column_type):
+        from dbt.clients import agate_helper
+        if column_type is agate.TimeDelta:  # dbt never makes this!
+            return agate.TimeDelta()
+
+        for instance in agate_helper.DEFAULT_TYPE_TESTER._possible_types:
+            if type(instance) is column_type:
+                return instance
+
+        raise ValueError(f'no tester for {column_type}')
+
+    def _make_table_of(self, rows, column_types):
+        column_names = list(string.ascii_letters[:len(rows[0])])
+        if isinstance(column_types, type):
+            column_types = [self._get_tester_for(column_types) for _ in column_names]
+        else:
+            column_types = [self._get_tester_for(typ) for typ in column_types]
+        table = agate.Table(rows, column_names=column_names, column_types=column_types)
+        return table