internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: weecology/retriever

Issue:
Update FIA links
It's that time of year again where FIA puts out a new release and moves things around. See https://github.com/weecology/retriever/issues/865#issuecomment-325588307

We need to track down the most recent links and update the script.

Thanks to @nestorperez for reporting this.


Discussion:
Thanks to the entire Retriever's team for making available this quite useful package. 

Resolving patch:
diff --git a/scripts/forest_inventory_analysis.py b/scripts/forest_inventory_analysis.py
--- a/scripts/forest_inventory_analysis.py
+++ b/scripts/forest_inventory_analysis.py
@@ -20,10 +20,10 @@ def __init__(self, **kwargs):
         self.title = "Forest Inventory and Analysis"
         self.name = "forest-inventory-analysis"
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '1.4.0'
+        self.version = '1.4.1'
         self.ref = "http://fia.fs.fed.us/"
-        self.urls = {"main": "https://apps.fs.usda.gov/fiadb-downloads/CSV/",
-                     'species': 'https://apps.fs.usda.gov/fiadb-downloads/CSV/REF_SPECIES.csv'}
+        self.urls = {"main": "https://apps.fs.usda.gov/fia/datamart/CSV/",
+                     'species': 'https://apps.fs.usda.gov/fia/datamart/CSV/REF_SPECIES.csv'}
         self.keywords = ["plants", "continental-scale", "observational"]
         self.citation = "DATEOFDOWNLOAD. Forest Inventory and Analysis Database, St. Paul, MN: U.S. Department of Agriculture, Forest Service, Northern Research Station. [Available only on internet: http://apps.fs.fed.us/fiadb-downloads/datamart.html]"
         self.description = """WARNING: This dataset requires downloading many large files and will probably take several hours to finish installing."""


Test patch:
Repository: weecology/retriever

Issue:
Improve error message when a file to be downloaded is missing
If I file isn't present at the url when we try to download it the error that is reported isn't very useful:

```
cannot import name 'urlretrieve'
```

We get this in part because of joint support for Python 2/3 (we try the Python 3 call, fall back to Python 2, and so if issue isn't Python 3 vs 2 we get an unhelpful error message), but we need to catch this better and report something more useful to the user.

See, e.g., https://github.com/weecology/retriever/issues/865


Discussion:

Resolving patch:
diff --git a/retriever/lib/engine.py b/retriever/lib/engine.py
--- a/retriever/lib/engine.py
+++ b/retriever/lib/engine.py
@@ -136,7 +136,7 @@ def get_ct_line_length(self, lines):
         real_line_length = 0
         for values in lines:
             initial_cols = len(self.table.columns) - \
-                (3 if hasattr(self.table, "ct_names") else 2)
+                           (3 if hasattr(self.table, "ct_names") else 2)
             # add one if auto increment is not set to get the right initial columns
             if not self.table.columns[0][1][0] == "pk-auto":
                 initial_cols += 1
@@ -412,7 +412,7 @@ def download_file(self, url, filename):
             print("\nDownloading " + filename + "...")
             try:
                 urlretrieve(url, path, reporthook=reporthook)
-            except:
+            except ImportError:
                 # For some urls lacking filenames urlretrieve from the future
                 # package seems to fail. This issue occurred in the PlantTaxonomy
                 # script. If this happens, fall back to the standard Python 2 version.


Test patch:
Repository: weecology/retriever

Issue:
Incorrectly lower casing table_name for csv
It looks like we're lower casing manually set table/directory names, at least for csv but probably for all flat file engines.

```
$ mkdir TESTER
$ retriever install csv mammal-masses --table_name TESTER/test.csv
=> Installing mammal-masses
[Errno 2] No such file or directory: 'tester/test.csv'
Done!

$ mkdir tester
$ retriever install csv mammal-masses --table_name TESTER/test.csv
=> Installing mammal-masses
Progress: 5731/5731 rows inserted into tester/test.csv totaling 5731:

Done!
```

This is causing issues for the R package, see https://github.com/ropensci/rdataretriever/issues/131, but is also a general problem since directory names are case sensitive for 2/3 OSs.


Discussion:
This happens in the very first line of `__main__.py`:

```
sys.argv[1:] = [arg.lower() for arg in sys.argv[1:]]
```

This makes sense of course, but is an issue for that particular arg. I suspect that something changed in how R was creating it's temp dirs recently which is why this suddenly cropped up over there, but this is a bug regardless.
@henrykironde - do you remember the motivation for https://github.com/weecology/retriever/commit/33788bd236a04c77bb4de66688ff3c590427f0bf ?
That is definitely a bug, I think it was a something I added when working on `the retriever ls ... `. 

Resolving patch:
diff --git a/retriever/__main__.py b/retriever/__main__.py
--- a/retriever/__main__.py
+++ b/retriever/__main__.py
@@ -32,7 +32,6 @@
 
 def main():
     """This function launches the Data Retriever."""
-    sys.argv[1:] = [arg.lower() for arg in sys.argv[1:]]
     if len(sys.argv) == 1:
         # if no command line args are passed, show the help options
         parser.parse_args(['-h'])


Test patch:
Repository: weecology/retriever

Issue:
Add a description field(s) to setup.py
This populates the description on PYPI:

https://packaging.python.org/tutorials/distributing-packages/#description


Discussion:
@ethanwhite Shall I send a PR for this?
`The Data Retriever is a package manager for data. It downloads, cleans, and stores publicly available data, so that analysts spend less time cleaning and managing data, and more time analyzing it.`
Is this text fine? It's the same as on http://www.data-retriever.org/
Looks great. Go for it!

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -29,6 +29,7 @@
 def clean_version(v):
     return parse_version(v).__repr__().lstrip("<Version('").rstrip("')>")
 
+
 includes = [
                'xlrd',
                'future',
@@ -52,6 +53,10 @@ def clean_version(v):
 setup(name='retriever',
       version=clean_version(__version__),
       description='Data Retriever',
+      long_description=('The Data Retriever is a package manager for data. '
+                        'It downloads, cleans, and stores publicly available data, '
+                        'so that analysts spend less time cleaning and managing data, '
+                        'and more time analyzing it.'),
       author='Ben Morris, Shivam Negi, Akash Goel, Andrew Zhang, Henry Senyondo, Ethan White',
       author_email='ethan@weecology.org',
       url='https://github.com/weecology/retriever',


Test patch:
Repository: weecology/retriever

Issue:
Breed-Bird-Survey Raw Data Location Change
The raw data for breed-bird-survey has been moved to a different location, so none of the ftp addresses in the breed-bird-survey work currently. The ftp addresses need to be switched over to the newest ones found here: 

ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/Archivefiles/


Discussion:
Looks like the old path is still there (or maybe they just added it back):
ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/

Of course they couldn't update the site without at least one capitalization change in a file name :roll_eyes:. I'm testing a fix now and will send a PR shortly.
My read is that the current data is in `DataFiles`, which is what we want and older versions of the data (which is a cool addition) are in `Archivefiles`.

Resolving patch:
diff --git a/scripts/breed_bird_survey.py b/scripts/breed_bird_survey.py
--- a/scripts/breed_bird_survey.py
+++ b/scripts/breed_bird_survey.py
@@ -35,10 +35,10 @@ def __init__(self, **kwargs):
         self.ref = "http://www.pwrc.usgs.gov/BBS/"
         self.keywords = ["birds", "continental-scale"]
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '1.4.2'
+        self.version = '1.4.3'
         self.urls = {
                      "counts": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/States/",
-                     "routes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/Routes.zip",
+                     "routes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/routes.zip",
                      "weather": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/Weather.zip",
                      "region_codes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/RegionCodes.txt",
                      "species": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/SpeciesList.txt"
diff --git a/scripts/breed_bird_survey_50stop.py b/scripts/breed_bird_survey_50stop.py
--- a/scripts/breed_bird_survey_50stop.py
+++ b/scripts/breed_bird_survey_50stop.py
@@ -36,10 +36,10 @@ def __init__(self, **kwargs):
         self.ref = "http://www.pwrc.usgs.gov/BBS/"
         self.keywords = ["birds", "continental-scale"]
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '1.4.2'
+        self.version = '1.4.3'
         self.urls = {
                      "counts": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/50-StopData/1997ToPresent_SurveyWide/",
-                     "routes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/Routes.zip",
+                     "routes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/routes.zip",
                      "weather": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/Weather.zip",
                      "region_codes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/RegionCodes.txt",
                      "species": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/SpeciesList.txt"


Test patch:
Repository: weecology/retriever

Issue:
Gentry site_code incorrectly includes directory name
At some point a behavior changed that caused site names extracted from the file names in gentry-forest-transects to including the directory in which they were located. This leads to site_code values like `africa/NDAKANNI`, which isn't what we want.


Discussion:

Resolving patch:
diff --git a/scripts/gentry_forest_transects.py b/scripts/gentry_forest_transects.py
--- a/scripts/gentry_forest_transects.py
+++ b/scripts/gentry_forest_transects.py
@@ -1,9 +1,6 @@
 #retriever
-"""Retriever script for Alwyn H. Gentry Forest Transect Dataset
+"""Retriever script for Alwyn H. Gentry Forest Transect Dataset"""
 
-"""
-from __future__ import print_function
-# from __future__ import unicode_literals
 from builtins import str
 from builtins import range
 
@@ -29,7 +26,7 @@ def __init__(self, **kwargs):
         self.title = "Alwyn H. Gentry Forest Transect Dataset"
         self.name = "gentry-forest-transects"
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '1.4.2'
+        self.version = '1.4.3'
         self.urls = {"stems": "http://www.mobot.org/mobot/gentry/123/all_Excel.zip",
                      "sites": "https://ndownloader.figshare.com/files/5515373",
                      "species": "",
@@ -74,7 +71,6 @@ def download(self, engine=None, debug=False):
         lines = []
         tax = []
         for filename in filelist:
-            print("Extracting data from " + filename + "...")
             book = xlrd.open_workbook(self.engine.format_filename(filename))
             sh = book.sheet_by_index(0)
             rows = sh.nrows
@@ -130,7 +126,8 @@ def download(self, engine=None, debug=False):
                         this_line["stems"] = [row[c]
                                               for c in cn["stems"]
                                               if not Excel.empty_cell(row[c])]
-                        this_line["site"] = filename[0:-4]
+                        site_code, _ = os.path.splitext(os.path.basename(filename))
+                        this_line["site"] = site_code
 
                         # Manually correct CEDRAL data, which has a single line
                         # that is shifted by one to the left starting at Liana
@@ -166,17 +163,11 @@ def download(self, engine=None, debug=False):
         tax_count = 0
 
         # Get all unique families/genera/species
-        print("\n")
         for group in tax:
             if not (group in unique_tax):
                 unique_tax.append(group)
                 tax_count += 1
                 tax_dict[group[0:3]] = tax_count
-                if tax_count % 10 == 0:
-                    msg = "Generating taxonomic groups: " + str(tax_count) + " / " + str(TAX_GROUPS)
-                    sys.stdout.flush()
-                    sys.stdout.write(msg + "\b" * len(msg))
-        print("\n")
         # Create species table
         table = Table("species", delimiter=",")
         table.columns=[("species_id"            ,   ("pk-int",)    ),


Test patch:
Repository: weecology/retriever

Issue:
UnboundLocalError when deleting an item from keywords key in json scripts
I tried deleting an item from the list of the key `keywords` in the dataset scripts, but it threw an exception `UnboundLocalError: local variable 'del_key' referenced before assignment`. 



Discussion:
@henrykironde if possible, I would like to work on it.

Resolving patch:
diff --git a/retriever/lib/datapackage.py b/retriever/lib/datapackage.py
--- a/retriever/lib/datapackage.py
+++ b/retriever/lib/datapackage.py
@@ -356,7 +356,8 @@ def edit_dict(obj, tabwidth=0):
                         if del_val not in obj[key]:
                             print("Invalid value: Not found.")
                             continue
-                        print("Removed " + str(obj[key].pop(del_key)))
+                        obj[key].remove(del_val)
+                        print("Removed " + str(del_val))
 
                     elif selection == '3':
                         do_remove = clean_input(


Test patch:
Repository: weecology/retriever

Issue:
fetch method should return an ordered dict (not a dict)
Hello,

I noticed that `fetch` method returns a dict of dataframes.
To preserve order of tables (which may have a meaning) it should return an ordered dict.
[Datapackage](https://frictionlessdata.io/specs/data-package/) ressources are stored in a list so it's ordered.

Kind regards


Discussion:
That would  be great  👍 @scls19fr 

@henrykironde  @scls19fr 
For this to be implemented are the following changes enough in `retriever\retriever\engines\sqlite.py` ?

```diff
+from collections import OrderedDict

@@ -44,10 +45,10 @@ class engine(Engine):
     def fetch_tables(self, dataset, table_names):
         """Return sqlite dataset as list of pandas dataframe."""
         connection = self.get_connection()
-        data = {table[len(dataset) + 1:]: pd.read_sql_query("SELECT * "
+        data = OrderedDict({table[len(dataset) + 1:]: pd.read_sql_query("SELECT * "
                                                             "FROM {};".format(table),
                                                             connection)
-                for table in table_names}
+                for table in table_names})
         return data
```
Thank @Jaffrey98, I am not sure, we shall have to test this. Could you put in a PR and we could follow up with some testing. 
Sure !

Resolving patch:
diff --git a/retriever/engines/sqlite.py b/retriever/engines/sqlite.py
--- a/retriever/engines/sqlite.py
+++ b/retriever/engines/sqlite.py
@@ -1,6 +1,7 @@
 import os
 import pandas as pd
 from builtins import range
+from collections import OrderedDict
 
 from retriever.lib.defaults import DATA_DIR
 from retriever.lib.models import Engine, no_cleanup
@@ -44,10 +45,12 @@ def create_db(self):
     def fetch_tables(self, dataset, table_names):
         """Return sqlite dataset as list of pandas dataframe."""
         connection = self.get_connection()
-        data = {table[len(dataset) + 1:]: pd.read_sql_query("SELECT * "
-                                                            "FROM {};".format(table),
-                                                            connection)
-                for table in table_names}
+        sql_query = "SELECT * FROM {};"
+        data = OrderedDict({
+                    table[len(dataset) + 1:]
+                    :pd.read_sql_query(sql_query.format(table), connection)
+                    for table in table_names
+               })
         return data
 
     def get_bulk_insert_statement(self):


Test patch:
Repository: weecology/retriever

Issue:
Default REPO_URL leads to 400: Invalid request
The current [REPO_URL](https://raw.github.com/weecology/retriever/)  defined in `retriever.lib.defaults.py` leads to a 400: Invalid request.


Discussion:

Resolving patch:
diff --git a/retriever/lib/defaults.py b/retriever/lib/defaults.py
--- a/retriever/lib/defaults.py
+++ b/retriever/lib/defaults.py
@@ -4,7 +4,7 @@
 
 VERSION = __version__
 COPYRIGHT = "Copyright (C) 2011-2016 Weecology University of Florida"
-REPO_URL = "https://raw.github.com/weecology/retriever/"
+REPO_URL = "https://raw.githubusercontent.com/weecology/retriever/"
 MASTER_BRANCH = REPO_URL + "master/"
 REPOSITORY = MASTER_BRANCH
 ENCODING = 'ISO-8859-1'
diff --git a/retriever/lib/engine_tools.py b/retriever/lib/engine_tools.py
--- a/retriever/lib/engine_tools.py
+++ b/retriever/lib/engine_tools.py
@@ -98,7 +98,7 @@ def reset_retriever(scope="all", ask_permission=True):
     """Remove stored information on scripts and data."""
     warning_messages = {
         'all': "\nThis will remove existing scripts and cached data."
-               "\nSpecifically it will remove the scripts and raw_data folders"
+               "\nSpecifically it will remove the scripts and raw_data folders "
                "in {}\nDo you want to proceed? (y/N)\n",
         'scripts': "\nThis will remove existing scripts."
                    + "\nSpecifically it will remove the scripts folder in {}."


Test patch:
Repository: weecology/retriever

Issue:
Error when running retriever license with no dataset as argument
Currently, when running `retriever license` and providing no dataset as an argument it results into `KeyError: 'No dataset named: None'`
![retriever_ls](https://user-images.githubusercontent.com/21103831/56208546-069b7b00-606f-11e9-84e5-9952be6cc097.png)

In case of `retriever citation` if no dataset is given we show the citation for retriever. 
![retriever_ct](https://user-images.githubusercontent.com/21103831/56208927-ef10c200-606f-11e9-89aa-056c1c126e33.png)

How should we handle this? Show the license for retriever by reading the `LICENSE` file or create a new `LICENSE` variable in `retriever.lib.defaults` and show it?


Discussion:
Hey @apoorvaeternity thanks for catching this!

I think we should keep with convention and create a LICENSE variable in defaults to show. 

Resolving patch:
diff --git a/retriever/__main__.py b/retriever/__main__.py
--- a/retriever/__main__.py
+++ b/retriever/__main__.py
@@ -12,7 +12,7 @@
 from retriever.engines import engine_list, choose_engine
 from retriever.lib.datapackage import create_json, edit_json, delete_json, get_script_filename
 from retriever.lib.datasets import datasets, dataset_names, license
-from retriever.lib.defaults import sample_script, CITATION, SCRIPT_SEARCH_PATHS
+from retriever.lib.defaults import sample_script, CITATION, SCRIPT_SEARCH_PATHS, LICENSE
 from retriever.lib.engine_tools import name_matches, reset_retriever
 from retriever.lib.get_opts import parser
 from retriever.lib.repository import check_for_updates
@@ -76,11 +76,14 @@ def main():
             return
 
         elif args.command == 'license':
-            dataset_license = license(args.dataset)
-            if dataset_license:
-                print(dataset_license)
+            if args.dataset is None:
+                print(LICENSE)
             else:
-                print("There is no license information for {}".format(args.dataset))
+                dataset_license = license(args.dataset)
+                if dataset_license:
+                    print(dataset_license)
+                else:
+                    print("There is no license information for {}".format(args.dataset))
             return
 
         elif args.command == 'new':
diff --git a/retriever/lib/defaults.py b/retriever/lib/defaults.py
--- a/retriever/lib/defaults.py
+++ b/retriever/lib/defaults.py
@@ -4,6 +4,7 @@
 
 VERSION = __version__
 COPYRIGHT = "Copyright (C) 2011-2016 Weecology University of Florida"
+LICENSE = "MIT"
 REPO_URL = "https://raw.githubusercontent.com/weecology/retriever/"
 MASTER_BRANCH = REPO_URL + "master/"
 REPOSITORY = MASTER_BRANCH


Test patch:
Repository: weecology/retriever

Issue:
Repair biotimesql
define NULL


Discussion:

Resolving patch:
diff --git a/scripts/biotimesql.py b/scripts/biotimesql.py
--- a/scripts/biotimesql.py
+++ b/scripts/biotimesql.py
@@ -27,7 +27,7 @@ def __init__(self, **kwargs):
         self.urls = {
             "sql_file": "https://zenodo.org/record/2602708/files/BioTIMESQL02_04_2018.sql?download=1",
         }
-        self.version = "1.0.0"
+        self.version = "1.0.1"
         self.ref = "https://zenodo.org/record/1095628#.WskN7dPwYyn"
         self.citation = "Dornelas M, Antão LH, Moyes F, et al. BioTIME: A database of biodiversity time series for the Anthropocene. Global Ecology & Biogeography. 2018; 00:1 - 26. https://doi.org/10.1111/geb.12729."
         self.description = "The BioTIME database has species identities and abundances in ecological assemblages through time."
@@ -51,6 +51,7 @@ def download(self, engine=None, debug=False):
         csv_writer = None
         csv_file = None
         table_name = None
+        NULL = None
         for line in sql_data:
             table_indicator = "-- Table structure for table "
             if line.startswith(table_indicator):


Test patch:
Repository: weecology/retriever

Issue:
Post release bump of version to 2.4.1-dev



Discussion:

Resolving patch:
diff --git a/retriever/_version.py b/retriever/_version.py
--- a/retriever/_version.py
+++ b/retriever/_version.py
@@ -1 +1 @@
-__version__ = 'v2.4.0'
+__version__ = 'v2.4.1.dev'
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -21,7 +21,7 @@
 if os.path.exists(app_data):
     os.system("rm -r {}".format(app_data))
 
-__version__ = 'v2.4.0'
+__version__ = 'v2.4.1.dev'
 with open(os.path.join("retriever", "_version.py"), "w") as version_file:
     version_file.write("__version__ = " + "'" + __version__ + "'\n")
     version_file.close()


Test patch: