Repository: spotify/luigi

Issue:
Redshift COPY fails in luigi 2.7.1 when columns are not provided
Running Redshift COPY jobs with `columns = None` to prohibit table creation fails in luigi 2.7.1 with 
```
TypeError: object of type 'NoneType' has no len()
```

The root cause seems to be https://github.com/spotify/luigi/pull/2245/files#diff-778ea3db4cccaf4de6564889c5eb670fR338

A possible solution would be to change the line to
```python
if self.columns and len(self.columns) > 0:
```
unless I am missing some reason to explicitly ask only for `len(self.columns)`.


<!---
We use GitHub issues mainly for tracking bugs and feature requests.
Questions for how to use luigi can be sent to the mailing list.

Currently, there are no strict procedures or guidelines for submitting issues.
In short, please just use common sense.

Common sense includes this at bare-minimum:

 * search for similar issues posted before creating a new issue.
 * Use markdown to format all code/logs. Issues which are hard to read
   when rendered on GitHub might be closed with a friendly reminder of this.
 * If applicable, reading relevant parts of the documentation.

Also, add steps to reproduce the bug, if applicable. Sample code would be nice too :)

For more information on how to submit valuable contributions,
see https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution
-->



Discussion:
This seems like a reasonable solution. The motivation here was to handle the [default `column`](https://github.com/spotify/luigi/blob/master/luigi/contrib/rdbms.py#L71) definition or some structure with length > 0. Really, we should ensure that it is not `None` and an iterable with length greater than zero.

Feel free to PR!

Resolving patch:
diff --git a/luigi/contrib/redshift.py b/luigi/contrib/redshift.py
--- a/luigi/contrib/redshift.py
+++ b/luigi/contrib/redshift.py
@@ -335,7 +335,7 @@ def copy(self, cursor, f):
         """
         logger.info("Inserting file: %s", f)
         colnames = ''
-        if len(self.columns) > 0:
+        if self.columns and len(self.columns) > 0:
             colnames = ",".join([x[0] for x in self.columns])
             colnames = '({})'.format(colnames)
 


Test patch:
diff --git a/test/contrib/redshift_test.py b/test/contrib/redshift_test.py
--- a/test/contrib/redshift_test.py
+++ b/test/contrib/redshift_test.py
@@ -40,9 +40,11 @@ class DummyS3CopyToTableBase(luigi.contrib.redshift.S3CopyToTable):
     user = 'dummy_user'
     password = 'dummy_password'
     table = luigi.Parameter(default='dummy_table')
-    columns = (
-        ('some_text', 'varchar(255)'),
-        ('some_int', 'int'),
+    columns = luigi.TupleParameter(
+        default=(
+            ('some_text', 'varchar(255)'),
+            ('some_int', 'int'),
+        )
     )
 
     copy_options = ''
@@ -214,6 +216,120 @@ def test_s3_copy_to_temp_table(self, mock_redshift_target, mock_copy):
             (task.table,),
         )
 
+    @mock.patch("luigi.contrib.redshift.RedshiftTarget")
+    def test_s3_copy_with_valid_columns(self, mock_redshift_target):
+        task = DummyS3CopyToTableKey()
+        task.run()
+
+        # The mocked connection cursor passed to
+        # S3CopyToTable.copy(self, cursor, f).
+        mock_cursor = (mock_redshift_target.return_value
+                                           .connect
+                                           .return_value
+                                           .cursor
+                                           .return_value)
+
+        # `mock_redshift_target` is the mocked `RedshiftTarget` object
+        # returned by S3CopyToTable.output(self).
+        mock_redshift_target.assert_called_once_with(
+            database=task.database,
+            host=task.host,
+            update_id=task.task_id,
+            user=task.user,
+            table=task.table,
+            password=task.password,
+        )
+
+        # To get the proper intendation in the multiline `COPY` statement the
+        # SQL string was copied from redshift.py.
+        mock_cursor.execute.assert_called_with("""
+         COPY {table} {colnames} from '{source}'
+         CREDENTIALS '{creds}'
+         {options}
+         ;""".format(
+            table='dummy_table',
+            colnames='(some_text,some_int)',
+            source='s3://bucket/key',
+            creds='aws_access_key_id=key;aws_secret_access_key=secret',
+            options='')
+        )
+
+    @mock.patch("luigi.contrib.redshift.RedshiftTarget")
+    def test_s3_copy_with_default_columns(self, mock_redshift_target):
+        task = DummyS3CopyToTableKey(columns=[])
+        task.run()
+
+        # The mocked connection cursor passed to
+        # S3CopyToTable.copy(self, cursor, f).
+        mock_cursor = (mock_redshift_target.return_value
+                                           .connect
+                                           .return_value
+                                           .cursor
+                                           .return_value)
+
+        # `mock_redshift_target` is the mocked `RedshiftTarget` object
+        # returned by S3CopyToTable.output(self).
+        mock_redshift_target.assert_called_once_with(
+            database=task.database,
+            host=task.host,
+            update_id=task.task_id,
+            user=task.user,
+            table=task.table,
+            password=task.password,
+        )
+
+        # To get the proper intendation in the multiline `COPY` statement the
+        # SQL string was copied from redshift.py.
+        mock_cursor.execute.assert_called_with("""
+         COPY {table} {colnames} from '{source}'
+         CREDENTIALS '{creds}'
+         {options}
+         ;""".format(
+            table='dummy_table',
+            colnames='',
+            source='s3://bucket/key',
+            creds='aws_access_key_id=key;aws_secret_access_key=secret',
+            options='')
+        )
+
+    @mock.patch("luigi.contrib.redshift.RedshiftTarget")
+    def test_s3_copy_with_nonetype_columns(self, mock_redshift_target):
+        task = DummyS3CopyToTableKey(columns=None)
+        task.run()
+
+        # The mocked connection cursor passed to
+        # S3CopyToTable.copy(self, cursor, f).
+        mock_cursor = (mock_redshift_target.return_value
+                                           .connect
+                                           .return_value
+                                           .cursor
+                                           .return_value)
+
+        # `mock_redshift_target` is the mocked `RedshiftTarget` object
+        # returned by S3CopyToTable.output(self).
+        mock_redshift_target.assert_called_once_with(
+            database=task.database,
+            host=task.host,
+            update_id=task.task_id,
+            user=task.user,
+            table=task.table,
+            password=task.password,
+        )
+
+        # To get the proper intendation in the multiline `COPY` statement the
+        # SQL string was copied from redshift.py.
+        mock_cursor.execute.assert_called_with("""
+         COPY {table} {colnames} from '{source}'
+         CREDENTIALS '{creds}'
+         {options}
+         ;""".format(
+            table='dummy_table',
+            colnames='',
+            source='s3://bucket/key',
+            creds='aws_access_key_id=key;aws_secret_access_key=secret',
+            options='')
+        )
+
 
 class TestS3CopyToSchemaTable(unittest.TestCase):
     @mock.patch("luigi.contrib.redshift.S3CopyToTable.copy")