Repository: weecology/retriever

Issue:
Stop bad scripts from causing errors
Currently when `compile_json` gets run if something goes wrong the retriever errors out even on commands not running the script (e.g., `retriever ls`). What it should do is ignore the bad script and possibly report back that there is an issue with the script but keep running normally otherwise.



Discussion:
@ethanwhite Can you post an example error trace so I can look into the issue?

To reproduce:
1. Change the `"retriever_minimum_version"` value in `MoM2003.json` to `2.0` from `"2.0.0-dev"`.
2. Run `retriever ls`

```
ethan@croryx:~$ retriever ls
Traceback (most recent call last):
  File "/home/ethan/anaconda/bin/retriever", line 9, in <module>
    load_entry_point('retriever===2.0.0-de', 'console_scripts', 'retriever')()
  File "/home/ethan/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/pkg_resources/__init__.py", line 549, in load_entry_point
  File "/home/ethan/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/pkg_resources/__init__.py", line 2542, in load_entry_point
  File "/home/ethan/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/pkg_resources/__init__.py", line 2202, in load
  File "/home/ethan/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/pkg_resources/__init__.py", line 2208, in resolve
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/__main__.py", line 29, in <module>
    from retriever.lib.datapackage import create_json, edit_json
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/datapackage.py", line 8, in <module>
    short_names = [script.shortname.lower() for script in SCRIPT_LIST()]
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/__init__.py", line 111, in SCRIPT_LIST
    return [module.SCRIPT for module in MODULE_LIST(force_compile)]
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/__init__.py", line 78, in MODULE_LIST
    compile_json(join(search_path, script_name))
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/compile.py", line 241, in compile_json
    values["retriever_minimum_version"] = "\"" + value + "\""
TypeError: cannot concatenate 'str' and 'float' objects
```

When running a local python environment via virtualenv or using `sudo` with system python, I don't get the error. Instead, the exception is handled already:

```
Failed to load script: MoM2003 (scripts)
Exception: name 'dev' is not defined 
Failed to load script: MoM2003 (scripts)
Exception: name 'dev' is not defined 
Available datasets : 47

Adler2007        BBS           EltonTraits        leaf_herbivory  Matter2014       Petraitis2008          SonoranPerennials   
AmnioteDB        BBS50         FIA                leda            MCDB             PhytoplankonBiovolume  Steppe_plants_2013  
Anderson2011     Bioclim       FishParasiteHosts  MammalDIET      McGlinn2010      PlantTaxonomy          TreeWesternGhats    
ArcticBreedBird  CForBioData   FrayJorge          MammalLH        MiscAbundanceDB  PortalMammals          vertnet             
AvianBodyMass    Clark2006     Gentry             MammalMR2010    NPN              PRISM                  Woods2009           
AvianBodySize    DelMoral2010  GWDD               mammsupertree   Palmer2007       Ramesh2010             Zachmann2010        
BAAD             eBirdOD       HomeRanges         MarineSize      Pantheria       
```

I think I can explicitly cast `value` to `str` before concatenating. See if that helps.

It's interesting that this is the result you get since this is what I thought should be happening already following a718c8d798903f1e096c4fb5ca0da9354a1dd536 from a few years ago, but I don't get this behavior in either my normal anaconda install or using Ubuntu's core Python install (with or without sudo) without the virtualenv.

Did you introduce exactly the same issue I did or something different? Python 2 or 3?

Python 2, with the modification you suggested. I tried both, a system install and local install of retriever before modifying and subsequently calling `retriever ls`.

@henrykironde - can you let us know what you see when you do this when you get the chance?

I still get the same error
1. Change the "retriever_minimum_version" value in MoM2003.json to 2.0 from "2.0.0-dev".
2. Run retriever ls

```
henrykironde$ retriever ls
Traceback (most recent call last):
  File "/usr/local/bin/retriever", line 9, in <module>
    load_entry_point('retriever===2.0.0-de', 'console_scripts', 'retriever')()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/site-packages/retriever-2.0.0_de-py2.7.egg/retriever/__main__.py", line 29, in <module>
    from retriever.lib.datapackage import create_json, edit_json
  File "/usr/local/lib/python2.7/site-packages/retriever-2.0.0_de-py2.7.egg/retriever/lib/datapackage.py", line 8, in <module>
    short_names = [script.shortname.lower() for script in SCRIPT_LIST()]
  File "/usr/local/lib/python2.7/site-packages/retriever-2.0.0_de-py2.7.egg/retriever/__init__.py", line 111, in SCRIPT_LIST
    return [module.SCRIPT for module in MODULE_LIST(force_compile)]
  File "/usr/local/lib/python2.7/site-packages/retriever-2.0.0_de-py2.7.egg/retriever/__init__.py", line 78, in MODULE_LIST
    compile_json(join(search_path, script_name))
  File "/usr/local/lib/python2.7/site-packages/retriever-2.0.0_de-py2.7.egg/retriever/lib/compile.py", line 241, in compile_json
    values["retriever_minimum_version"] = "\"" + value + "\""
TypeError: cannot concatenate 'str' and 'float' objects

```


Resolving patch:
diff --git a/lib/compile.py b/lib/compile.py
--- a/lib/compile.py
+++ b/lib/compile.py
@@ -238,7 +238,7 @@ def compile_json(json_file):
             values["tags"] = value
 
         elif key == "retriever_minimum_version":
-            values["retriever_minimum_version"] = "\"" + value + "\""
+            values["retriever_minimum_version"] = "\"" + str(value) + "\""
 
         elif key == "resources":
             # Array of table objects


Test patch: