Repository: weecology/retriever

Issue:
Support unicode for json creation from .script
An example of this issue can be replicated if we want to add data sets having french character 

```
name: Croche understory vegetation data set
shortname: paquette2007
citation: Alain Paquette, Etienne Laliberté, André Bouchard, Sylvie de Blois, Pierre Legendre, and Jacques Brisson. 2007. Lac Croche understory vegetation data set (1998–2006). Ecology 88:3209.
description: The Lac Croche data set covers a nine-year period (1998–2006) of detailed understory vegetation sampling of a temperate North American forest located in the Station de Biologie des Laurentides (SBL), Québec, Canada. 
tags: Taxon > plants
ref: http://esapubs.org/archive/ecol/E088/197/
# tables
table: croche_vegCover, http://esapubs.org/archive/ecol/E088/197/Croche_VegCover.txt
table: croche_seedDens, http://esapubs.org/archive/ecol/E088/197/Croche_SeedDens.txt
table: croche_env, http://esapubs.org/archive/ecol/E088/197/Croche_Env.txt
table: croche_ba, http://esapubs.org/archive/ecol/E088/197/Croche_BA.txt
*nulls: '-999'
```

trying to run `parse_script_to_json.py` will create the json 

```
{
    "citation": "Alain Paquette, Etienne Lalibert\u00e9, Andr\u00e9 Bouchard, Sylvie de Blois, Pierre Legendre, and Jacques Brisson. 2007. Lac Croche understory vegetation data set (1998\u20132006). Ecology 88:3209.",
    "description": "The Lac Croche data set covers a nine-year period (1998\u20132006) of detailed understory vegetation sampling of a temperate North American forest located in the Station de Biologie des Laurentides (SBL), Qu\u00e9bec, Canada.",
    "homepage": "http://esapubs.org/archive/ecol/E088/197/",
    "keywords": [
        "Taxon > plants"
    ],
    "name": "paquette2007",
    "resources": [
        {
            "dialect": {},
            "name": "croche_vegCover",
            "schema": {},
            "url": "http://esapubs.org/archive/ecol/E088/197/Croche_VegCover.txt"
        },
        {
            "dialect": {},
            "name": "croche_seedDens",
            "schema": {},
            "url": "http://esapubs.org/archive/ecol/E088/197/Croche_SeedDens.txt"
        },
        {
            "dialect": {},
            "name": "croche_env",
            "schema": {},
            "url": "http://esapubs.org/archive/ecol/E088/197/Croche_Env.txt"
        },
        {
            "dialect": {
                "nulls": [
                    "-999"
                ]
            },
            "name": "croche_ba",
            "schema": {},
            "url": "http://esapubs.org/archive/ecol/E088/197/Croche_BA.txt"
        }
    ],
    "retriever": "True",
    "retriever_minimum_version": 2.0,
    "script_version": 1.0,
    "title": "Croche understory vegetation data set",
    "urls": {
        "croche_ba": "http://esapubs.org/archive/ecol/E088/197/Croche_BA.txt",
        "croche_env": "http://esapubs.org/archive/ecol/E088/197/Croche_Env.txt",
        "croche_seedDens": "http://esapubs.org/archive/ecol/E088/197/Croche_SeedDens.txt",
        "croche_vegCover": "http://esapubs.org/archive/ecol/E088/197/Croche_VegCover.txt"
    }
}

```

This will not compile the json file to python file `paquette2007.py`



Discussion:
@goelakash if you have time could you look at this issue

The function `compile_json` is able to compile the JSON (I tried manually using ipython). But for some reason, retriever doesn't compile it.


Resolving patch:
diff --git a/__init__.py b/__init__.py
--- a/__init__.py
+++ b/__init__.py
@@ -78,7 +78,7 @@ def MODULE_LIST(force_compile=False):
 
         files = [file for file in os.listdir(search_path)
                  if file[-3:] == ".py" and file[0] != "_" and
-                 '#retriever' in open(join(search_path, file), 'r').readline().lower()]
+                 '#retriever' in ' '.join(open(join(search_path, file), 'r').readlines()[:2]).lower()]
 
         for script in files:
             script_name = '.'.join(script.split('.')[:-1])
diff --git a/lib/compile.py b/lib/compile.py
--- a/lib/compile.py
+++ b/lib/compile.py
@@ -1,5 +1,9 @@
 from builtins import str
 import json
+import sys
+if sys.version_info[0] < 3:
+    from codecs import open
+
 script_templates = {
     "default": """#retriever
 from retriever.lib.templates import BasicTextTemplate
@@ -287,6 +291,7 @@ def compile_json(json_file):
         template = "default"
     script_contents = (script_templates[template] % script_desc)
 
-    new_script = open(json_file + '.py', 'w')
+    new_script = open(json_file + '.py', 'w', encoding='utf-8')
+    new_script.write('# -*- coding: latin-1 -*-\n')
     new_script.write(script_contents)
     new_script.close()


Test patch: