Repository: weecology/retriever

Issue:
retriever update downloads all scripts twice
The retriever appears to be downloading all the scripts twice here is the output:

```
$ retriever update
Downloading script: MammalSuperTree.py
0.0-MammalSuperTree.py 100.0-MammalSuperTree.py Downloading script: EA_capellini2010.script
0.0-EA_capellini2010.script 100.0-EA_capellini2010.script Downloading script: EA_avianbodysize2007.script
0.0-EA_avianbodysize2007.script 100.0-EA_avianbodysize2007.script Downloading script: EA_mom2003.script
0.0-EA_mom2003.script 100.0-EA_mom2003.script Downloading script: EA_pantheria.script
0.0-EA_pantheria.script 100.0-EA_pantheria.script Downloading script: bbs.py
0.0-bbs.py 80.7-bbs.py 100.0-bbs.py Downloading script: EA_del_moral_2010.script
0.0-EA_del_moral_2010.script 100.0-EA_del_moral_2010.script Downloading script:
EA_mcglinn2010.script
0.0-EA_mcglinn2010.script 100.0-EA_mcglinn2010.script Downloading script: EA_barnes2008.script
0.0-EA_barnes2008.script 100.0-EA_barnes2008.script Downloading script: EA_ramesh2010.script
0.0-EA_ramesh2010.script 100.0-EA_ramesh2010.script Downloading script: sev_small_mammals.py
0.0-sev_small_mammals.py 100.0-sev_small_mammals.py Downloading script: bioclim_2pt5.py
0.0-bioclim_2pt5.py 100.0-bioclim_2pt5.py Downloading script: EA_thibault2011.script
0.0-EA_thibault2011.script 100.0-EA_thibault2011.script Downloading script: npn.py
0.0-npn.py 100.0-npn.py Downloading script: EA_ernest2003.script
0.0-EA_ernest2003.script 100.0-EA_ernest2003.script Downloading script: bbs50stop.py
0.0-bbs50stop.py 70.9-bbs50stop.py 100.0-bbs50stop.py Downloading script: EA_woods2009.script
0.0-EA_woods2009.script 100.0-EA_woods2009.script Downloading script: fia.py
0.0-fia.py 100.0-fia.py Downloading script: EA_clark2006.py
0.0-EA_clark2006.py 100.0-EA_clark2006.py Downloading script: EA_zachmann2010.script
0.0-EA_zachmann2010.script 100.0-EA_zachmann2010.script Downloading script: gentry.py
0.0-gentry.py 75.9-gentry.py 100.0-gentry.py Downloading script: USDA_plants.script
0.0-USDA_plants.script 100.0-USDA_plants.script Downloading script: EA_adler2007.script
0.0-EA_adler2007.script 100.0-EA_adler2007.script Downloading script: EA_portal_mammals.script
0.0-EA_portal_mammals.script 100.0-EA_portal_mammals.script Downloading script:
MammalDiet.script
0.0-MammalDiet.script 100.0-MammalDiet.script Downloading script: EA_petraitis2008.py
0.0-EA_petraitis2008.py 100.0-EA_petraitis2008.py Downloading script: EA_capellini2010.script
0.0-EA_capellini2010.script 100.0-EA_capellini2010.script Downloading script: EA_avianbodysize2007.script
0.0-EA_avianbodysize2007.script 100.0-EA_avianbodysize2007.script Downloading script: EA_mom2003.script
0.0-EA_mom2003.script 100.0-EA_mom2003.script Downloading script: EA_pantheria.script
0.0-EA_pantheria.script 100.0-EA_pantheria.script Downloading script: bbs.py
0.0-bbs.py 80.7-bbs.py 100.0-bbs.py Downloading script: EA_del_moral_2010.script
0.0-EA_del_moral_2010.script 100.0-EA_del_moral_2010.script Downloading script:
EA_mcglinn2010.script
0.0-EA_mcglinn2010.script 100.0-EA_mcglinn2010.script Downloading script: EA_barnes2008.script
0.0-EA_barnes2008.script 100.0-EA_barnes2008.script Downloading script: EA_ramesh2010.script
0.0-EA_ramesh2010.script 100.0-EA_ramesh2010.script Downloading script: EA_thibault2011.script
0.0-EA_thibault2011.script 100.0-EA_thibault2011.script Downloading script: npn.py
0.0-npn.py 100.0-npn.py Downloading script: EA_ernest2003.script
0.0-EA_ernest2003.script 100.0-EA_ernest2003.script Downloading script: bbs50stop.py
0.0-bbs50stop.py 70.9-bbs50stop.py 100.0-bbs50stop.py Downloading script: EA_woods2009.script
0.0-EA_woods2009.script 100.0-EA_woods2009.script Downloading script: fia.py
0.0-fia.py 100.0-fia.py Downloading script: EA_clark2006.py
0.0-EA_clark2006.py 100.0-EA_clark2006.py Downloading script: EA_zachmann2010.script
0.0-EA_zachmann2010.script 100.0-EA_zachmann2010.script Downloading script: gentry.py
0.0-gentry.py 75.9-gentry.py 100.0-gentry.py Downloading script: USDA_plants.script
0.0-USDA_plants.script 100.0-USDA_plants.script Downloading script: EA_adler2007.script
0.0-EA_adler2007.script 100.0-EA_adler2007.script Downloading script: EA_portal_mammals.script
0.0-EA_portal_mammals.script 100.0-EA_portal_mammals.script Downloading script:
MammalDiet.script
0.0-MammalDiet.script 100.0-MammalDiet.script Downloading script: EA_petraitis2008.py
0.0-EA_petraitis2008.py 100.0-EA_petraitis2008.py
```



Discussion:
Yes, that's correct. The issue is that `version.py` doesn't check for duplicates, which makes it easy to accidentally create a release in such a way that results in two copies of a script (one for the repo and one from the develops installed copy of the retriever) being listed in `version.txt`.

@bendmorris had done some work on this as part of a larger PR that never made it in, but I can't seem to track it down now.

Thanks for the info! I'll leave this open then as a reminder of something to fix down the line then.


Resolving patch:
diff --git a/__init__.py b/__init__.py
--- a/__init__.py
+++ b/__init__.py
@@ -86,8 +86,11 @@ def MODULE_LIST(force_compile=False):
             file, pathname, desc = imp.find_module(script_name, [search_path])
             try:
                 new_module = imp.load_module(script_name, file, pathname, desc)
-                new_module.SCRIPT.download
-                modules.append(new_module)
+                if not new_module in modules:
+                    # if the script wasn't found in an early search path
+                    # make sure it works and then add it
+                    new_module.SCRIPT.download
+                    modules.append(new_module)
             except:
                 sys.stderr.write("Failed to load script: %s (%s)" % (script_name, search_path))
 


Test patch: