Repository: spotify/luigi

Issue:
Handle UnicodeDecodeError on write to Postgres.
Unfortunately, I don't have a good automated test to show this, but the following simple example returns a UnicodeDecodeError with the current `postgres.py` in master when it attempts to write the third row (or any text with extended Unicode characters).

```
#coding: utf-8

import luigi
import luigi.postgres
import logging

logger = logging.getLogger("luigi-interface")

testdata = ([99, 'My fake plants died because I did not pretend to water them.'],
       [100, 'I always arrive late at the office, but I make up for it by leaving early.'],
       [101, 'ஸ்றீனிவாஸ ராமானுஜன் ஐயங்கார்'])

class TrivialPgLoad(luigi.postgres.CopyToTable):
    host = 'localhost'
    database = 'spotify'
    user = 'spotify'
    password = 'guest'
    table = 'trivium'
    columns = [("trival_id", "INT"),
               ("description", "TEXT")]

    def rows(self):
        for row in testdata:
            yield row

if __name__ == '__main__':
    luigi.run(main_task_cls=TrivialPgLoad)
```

The traceback looks like:

```
Traceback (most recent call last):
  File "/Users/thomas/Dropbox/envs/luigi-pg-unicode/lib/python2.7/site-packages/luigi/worker.py", line 137, in run
    new_deps = self._run_get_new_deps()
  File "/Users/thomas/Dropbox/envs/luigi-pg-unicode/lib/python2.7/site-packages/luigi/worker.py", line 88, in _run_get_new_deps
    task_gen = self.task.run()
  File "/Users/thomas/Dropbox/envs/luigi-pg-unicode/lib/python2.7/site-packages/luigi/postgres.py", line 318, in run
    tmp_file.write(rowstr.encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 4: ordinal not in range(128)
```

The load to Postgres succeeds with my fix.



Discussion:
Not thinking through this fully, but I would rather make the conversion explicit – it seems like either all the rows are utf-8, or none of them are.

Yes, it actually works just as well for me to take the encode() call out of line 318 completely. I used the try/except block because I didn't feel 100% confident that doing it that wouldn't break other use cases. 

There was some discussion of this previously over here: https://groups.google.com/forum/m/#!searchin/luigi-user/Unicodedecodeerror/luigi-user/2UyGTJHFx_8 ... ISTM the issue has to do with the data getting converted to utf-8 twice in some cases. The issue appears if the source data is already Unicode. Pure Python str inserts work with the current code. 

The issue here (like at many place in Luigi) is that Binary and String data are missed. The postgres task is build to handle string and not binary data. However, in the example that you gave you use binary data  (I assume that you are using python2). The easy fix would be to use `unicode` instead of `str`.

``` python
testdata = ([99, u'My fake plants died because I did not pretend to water them.'],
       [100, u'I always arrive late at the office, but I make up for it by leaving early.'],
       [101, u'ஸ்றீனிவாஸ ராமானுஜன் ஐயங்கார்'])
```

Again, it is precisely the case where the data are already Unicode that fails the current code. Str data load fine as is. 

I had a closer look and there is indeed an issue, but it's not with the line in the traceback, but with this [one](https://github.com/spotify/luigi/blob/master/luigi/postgres.py#L265). I'll send a PR soon.

@erikbern I wouldn't require the user to encode explicitly the data since the text need to be encoded in utf-8 since it's what the connection [expect](https://github.com/spotify/luigi/blob/master/luigi/postgres.py#L197).  The current implementation doesn't seems to handle binary data, so the users should provide text data which is `unicode` for python2. Doing otherwise would create issue if we want to handle binary data.

@gpoulin - thanks! I knew about line 265—that's what I was trying to say about unicode lines having encode() called twice. Since you seem to have a clearer idea how to fix this than I do, I will look forward to your PR, and we can close this one now. 


Resolving patch:
diff --git a/luigi/postgres.py b/luigi/postgres.py
--- a/luigi/postgres.py
+++ b/luigi/postgres.py
@@ -256,12 +256,8 @@ def map_column(self, value):
         """
         if value in self.null_values:
             return r'\N'
-        elif six.PY3:
-            return default_escape(str(value))
-        elif isinstance(value, unicode):
-            return default_escape(value).encode('utf8')
         else:
-            return default_escape(str(value))
+            return default_escape(six.text_type(value))
 
 # everything below will rarely have to be overridden
 


Test patch:
diff --git a/test/postgres_with_server_test.py b/test/postgres_with_server_test.py
--- a/test/postgres_with_server_test.py
+++ b/test/postgres_with_server_test.py
@@ -48,6 +48,8 @@
         password=password,
     )
     conn.close()
+    psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
+    psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY)
 except Exception:
     raise unittest.SkipTest('Unable to connect to postgres')
 
@@ -76,6 +78,7 @@ def rows(self):
         yield 'foo', 123, 123.45
         yield None, '-100', '5143.213'
         yield '\t\n\r\\N', 0, 0
+        yield u'éцү我', 0, 0
 
 
 class MetricBase(CopyToTestDB):
@@ -132,8 +135,9 @@ def test_repeat(self):
         self.assertEqual(rows, (
             ('foo', 123, 123.45),
             (None, -100, 5143.213),
-            ('\t\n\r\\N', 0.0, 0))
-        )
+            ('\t\n\r\\N', 0.0, 0),
+            (u'éцү我', 0, 0),
+        ))
 
     def test_multimetric(self):
         metrics = MetricBase()