Repository: weecology/retriever

Issue:
GWDD failing to install due to non-latin-1 character
```
ethan@stice:~$ retriever install --debug csv GWDD
=> Installing Zanne et al. Global wood density database.
'latin-1' codec can't encode character u'\u2563' in position 157: ordinal not in range(256)
Traceback (most recent call last):
  File "/home/ethan/anaconda/bin/retriever", line 9, in <module>
    load_entry_point('retriever===1.9.0-de', 'console_scripts', 'retriever')()
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/__main__.py", line 135, in main
    dataset.download(engine, debug=debug)
  File "/home/ethan/.retriever/scripts/gwdd.py", line 99, in download
    this_line[label] = format_value(row[num])
  File "/home/ethan/.retriever/scripts/gwdd.py", line 96, in format_value
    s = Excel.cell_value(s)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/excel.py", line 26, in cell_value
    return (cell.value).encode().strip()
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2563' in position 157: ordinal not in range(256)

```

This dataset worked fine before the Python 3 upgrade even with the default `latin-1` encoding, so I suspect some change in the encoding decoding behavior.

@goelakash - is this something you have some time to look into?



Discussion:
Just updated the original issue with the full `--debug` trace.

@ethanwhite We could add an `error=ignore` for this case, as this is not a general issue (Most files decode well with `latin-1` as we discussed earlier in #553 . What do you think?

I'd rather not drop characters if at all possible. Would it be possible to change the encoding from the default on a per dataset level? So, by default we use `latin-1`, but if we need something else we set an `encoding` variable in the script such that encoding gets set like `encoding = 'utf8'` to whatever encoding is best.


Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -104,7 +104,9 @@ def source_gen():
         pos = 0
         for line in real_lines:
             if not self.table.fixed_width:
-                line = line.strip()
+                # This replaces end of line characters that exist in a single line
+                # eg. "one \nline has multiple end of lines\n"
+                line = line.replace('\n', '').strip()
             if line:
                 self.table.record_id += 1
 
@@ -564,12 +566,14 @@ def format_insert_value(self, value, datatype, escape=True):
             else:
                 return "null"
         elif datatype in ("double", "decimal"):
-            if strvalue:
+            if strvalue.strip():
                 try:
                     decimals = float(strvalue)
                     return str(decimals)
                 except:
                     return "null"
+            else:
+                return "null"
         elif datatype == "char":
             if strvalue.lower() in nulls:
                 return "null"
diff --git a/lib/excel.py b/lib/excel.py
--- a/lib/excel.py
+++ b/lib/excel.py
@@ -23,5 +23,5 @@ def empty_cell(cell):
     def cell_value(cell):
         """Returns the string value of an excel spreadsheet cell"""
         if (cell.value).__class__.__name__ == 'unicode':
-            return (cell.value).encode().strip()
+            return (str(cell.value).encode()).strip()
         return str(cell.value).strip()


Test patch: