Repository: weecology/retriever

Issue:
Python 2 encoding issues in Python interface only
In the Python interface some datasets (at least `airports` and `gentry-forest-transects`) report encoding issues and fail. E.g.,

```
In [11]: rt.install_sqlite('gentry-forest-transects')
=> Installing gentry-forest-transects
'ascii' codec can't encode characters in position 5-6: ordinal not in range(128)
```

But these same datasets install fine from the CLI.


Discussion:
May I go ahead with this? @ethanwhite 
Go for it. If you have questions @goelakash may have some ideas since he dealt with a lot of the issues involving encoding in the past. 
Thank you. 
@ethanwhite 
According to docs
`The csv module doesn’t directly support reading and writing Unicode, but it is 8-bit-clean save for some problems with ASCII NUL characters. So you can write functions or classes that handle the encoding and decoding for you as long as you avoid encodings like UTF-16 that use NULs. UTF-8 is recommended.`

[Link to docs](https://docs.python.org/2/library/csv.html)
A possible fix to this would be adding this in engine.py
```python
import sys
from imp import reload
from retriever.lib.defaults import ENCODING
encoding = ENCODING.lower()
reload(sys)
if hasattr(sys, 'setdefaultencoding'):
    sys.setdefaultencoding(encoding)
```
or use the way that is recommended in docs but then we will have to check unicode strings everywhere.

If no one is working on this issue I would like to take this up.
@apoorvaeternity Please go ahead :)
@ethanwhite @henrykironde How shall I proceed with this? The setdefaultencoding method is not considered a good practice. 
I think the big question (which maybe @apoorvaeternity answered, but I didn't follow) is why this is only happening in the Python interface not the CLI. The should both be running largely the same code for the actual data processing and installation. Once we understand that then I think we'll understand the best fix.
@ethanwhite In CLI we use `__main__.py`. In it we have set the default encoding as
```python
from retriever.lib.defaults import sample_script, CITATION, ENCODING, SCRIPT_SEARCH_PATHS
encoding = ENCODING.lower()
# sys removes the setdefaultencoding method at startup; reload to get it back
reload(sys)
if hasattr(sys, 'setdefaultencoding'):
    sys.setdefaultencoding(encoding)
```
Removing these lines give the same error in CLI too. In python interface we have not set the default encoding so the error occurs.
@apoorvaeternity thats sounds right. I was expecting a PR on that. I think @ss-is-master-chief was working on that. We could confirm from @ss-is-master-chief else you could go ahead and put in a fix.
@henrykironde I already asked him.  [See](https://github.com/weecology/retriever/issues/1097#issuecomment-368449124) . Shall I work on the fix?
Yes, thanks

Resolving patch:
diff --git a/retriever/__main__.py b/retriever/__main__.py
--- a/retriever/__main__.py
+++ b/retriever/__main__.py
@@ -12,7 +12,6 @@
 import os
 import sys
 from builtins import input
-from imp import reload
 
 from retriever.engines import engine_list, choose_engine
 from retriever.lib.datapackage import create_json, edit_json, delete_json, get_script_filename
@@ -23,12 +22,6 @@
 from retriever.lib.scripts import SCRIPT_LIST, get_script
 from retriever.lib.engine_tools import name_matches, reset_retriever
 
-encoding = ENCODING.lower()
-# sys removes the setdefaultencoding method at startup; reload to get it back
-reload(sys)
-if hasattr(sys, 'setdefaultencoding'):
-    sys.setdefaultencoding(encoding)
-
 
 def main():
     """This function launches the Data Retriever."""
@@ -135,7 +128,7 @@ def main():
                 print("Available datasets : {}\n".format(len(all_scripts)))
                 from retriever import lscolumns
                 lscolumns.printls(all_scripts)
-            
+
             elif type(args.v) is list:
                 if args.v:
                     try:
@@ -156,7 +149,7 @@ def main():
                         script.citation
                     ))
                     count += 1
- 
+
             else:
                 param_licenses = args.l if args.l else None
                 keywords = args.k if args.k else None
@@ -170,10 +163,10 @@ def main():
                     count = 1
                     for script in searched_scripts:
                         print("{}. {}\n{}\n{}\n{}\n".format(
-                                count, script.title,
-                                script.name,
-                                script.keywords,
-                                str(script.licenses[0]['name'])
+                            count, script.title,
+                            script.name,
+                            script.keywords,
+                            str(script.licenses[0]['name'])
                         ))
                         count += 1
             return
diff --git a/retriever/lib/engine.py b/retriever/lib/engine.py
--- a/retriever/lib/engine.py
+++ b/retriever/lib/engine.py
@@ -22,9 +22,16 @@
 from tqdm import tqdm
 from urllib.request import urlretrieve
 from retriever.lib.tools import open_fr, open_fw, open_csvw
-from retriever.lib.defaults import DATA_SEARCH_PATHS, DATA_WRITE_PATH
+from retriever.lib.defaults import DATA_SEARCH_PATHS, DATA_WRITE_PATH, ENCODING
 from retriever.lib.cleanup import no_cleanup
 from retriever.lib.warning import Warning
+from imp import reload
+
+encoding = ENCODING.lower()
+# sys removes the setdefaultencoding method at startup; reload to get it back
+reload(sys)
+if hasattr(sys, 'setdefaultencoding'):
+    sys.setdefaultencoding(encoding)
 
 
 class Engine(object):
@@ -85,8 +92,8 @@ def add_to_table(self, data_source):
         types = self.table.get_column_datatypes()
         multiple_values = []
         progbar = tqdm(desc='Installing {}'.format(self.table_name()),
-                   total=real_line_length,
-                   unit='rows')
+                       total=real_line_length,
+                       unit='rows')
         for line in real_lines:
             if line:
                 # Only process non empty lines
@@ -779,9 +786,11 @@ def gen_from_source(source):
 def reporthook(tqdm_inst):
     """tqdm wrapper to generate progress bar for urlretriever"""
     last_block = [0]
+
     def update_to(count=1, block_size=1, total_size=None):
         if total_size is not None:
             tqdm_inst.total = total_size
         tqdm_inst.update((count - last_block[0]) * block_size)
         last_block[0] = count
+
     return update_to


Test patch: