internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: weecology/retriever

Issue:
download command should probably fail when specified path does not exist
A datsaet can be downloaded to a specific path with the function `download` while specifying the -p argument. For example `retriever download MCDB -p my_path` will download each of the MCDB files and then copy them to the directory my_path but if my_path does not exist a file called my_path is created and the files overwrite that file as each is copied from the download directory. It may be best if the retriever fails with a warning that the path does not exist. 



Discussion:
Alternatively, should we try to create the directory first, and only fail if there's an error creating the directory?

> Alternatively, should we try to create the directory first, and only fail if there's an error creating the directory?

+1


Resolving patch:
diff --git a/engines/download_only.py b/engines/download_only.py
--- a/engines/download_only.py
+++ b/engines/download_only.py
@@ -51,7 +51,18 @@ def final_cleanup(self):
                     print("Keeping existing copy.")
                 else:
                     print("Copying %s from %s" % (file_name_nopath, file_path))
-                    shutil.copy(file_name, self.opts['path'])
+                    if os.path.isdir(self.opts['path']):
+                        try:
+                            shutil.copy(file_name, self.opts['path'])
+                        except:
+                            print("Couldn't copy file to %s" % self.opts['path'])
+                    else:
+                        try:
+                            print("Creating directory %s" % self.opts['path'])
+                            os.mkdir(self.opts['path'])
+                            shutil.copy(file_name, self.opts['path'])
+                        except:
+                            print("Couldn't create directory %s" % self.opts['path'])
         self.all_files = set()
 
     def auto_create_table(self, table, url=None, filename=None, pk=None):


Test patch:
Repository: weecology/retriever

Issue:
Improve Windows installer version handling of default paths
The Windows Installer version of the retriever currently defaults to installing files from file based engines onto the Desktop. This was done because if the GUI is launched from the start menu it's working directory is a system path. This has caused confusion https://github.com/ropensci/ecoretriever/issues/43 and should be improved.

The simplest solution is probably to check to see if the retriever is running as the CLI, and if it is to not override the default path.



Discussion:
@bendmorris - is there an easy way to check whether we're running from the CLI or the GUI?


Resolving patch:
diff --git a/__init__.py b/__init__.py
--- a/__init__.py
+++ b/__init__.py
@@ -51,10 +51,14 @@
                          ]
 DATA_WRITE_PATH =       DATA_SEARCH_PATHS[-1]
 
-# create a default data directory for Windows since the Windows installer places
-# the executable in a place where users won't expect the data to be stored.
-user_desktop = os.path.join(os.path.expanduser('~'), 'Desktop')
-DATA_DIR = user_desktop if "win" in current_platform else '.'
+# Create default data directory
+isgui = len(sys.argv) == 1 or ((len(sys.argv) > 1 and sys.argv[1] == 'gui'))
+if "win" in current_platform and isgui:
+    # The run path for installer based GUI on Windows is a system path.
+    # Users won't expect the data to be stored there, so store it on the Desktop
+    DATA_DIR = os.path.join(os.path.expanduser('~'), 'Desktop')
+else:
+    DATA_DIR = '.'
 
 def MODULE_LIST(force_compile=False):
     """Load scripts from scripts directory and return list of modules."""


Test patch:
Repository: weecology/retriever

Issue:
Should "ref" or "url" be a default tag 
The new script generation approach creates a generic file that lacks the ref or url tag. I think this tag should be included on the generic script because all of the datasets are coming from someplace online. Examining some of the script files suggests to me that the tags "ref" and "url" can be used interchangeably. Essentially this tag ensures that when browsing for datasets using the retriever you also see a hyperlink to the dataset. 



Discussion:
Sounds good to me. Do you want to give adding it a go?


Resolving patch:
diff --git a/__init__.py b/__init__.py
--- a/__init__.py
+++ b/__init__.py
@@ -117,6 +117,7 @@ def set_proxy():
 shortname: MammalLH
 description: S. K. Morgan Ernest. 2003. Life history characteristics of placental non-volant mammals. Ecology 84:3402.
 tags: Taxon > Mammals, Data Type > Compilation
+url: http://esapubs.org/archive/ecol/E084/093/default.htm
 
 # tables
 table: species, http://esapubs.org/archive/ecol/E084/093/Mammal_lifehistories_v2.txt"""


Test patch:
Repository: weecology/retriever

Issue:
Add BAAD biomass and allometry database
http://esapubs.org/archive/ecol/E096/128/metadata.php

Archive with multiple tables.



Discussion:

Resolving patch:
diff --git a/scripts/baad.py b/scripts/baad.py
new file mode 100644
--- /dev/null
+++ b/scripts/baad.py
@@ -0,0 +1,39 @@
+#retriever
+
+"""Retriever script for direct download of data data"""
+from retriever.lib.models import Table, Cleanup, correct_invalid_value
+from retriever.lib.templates import Script
+
+
+class main(Script):
+    def __init__(self, **kwargs):
+        Script.__init__(self, **kwargs)
+        self.name = "BAAD: a Biomass And Allometry Database for woody plants"
+        self.shortname = "BAAD"
+        self.ref = "http://esapubs.org/archive/ecol/E096/128/"
+        self.urls = {"BAAD": "http://www.esapubs.org/archive/ecol/E096/128/baad_data.zip"}
+        self.description = "Falster, D.S., Duursma, R.A., Ishihara, M.I., Barneche, D.R., FitzJohn, R.G., Varhammar, A., Aiba, M., Ando, M., Anten, N., Aspinwall, M.J. and Baltzer, J.L., 2015. BAAD: a Biomass And Allometry Database for woody plants."
+        self.tags = ['Taxon > Plants', 'Data Type > Observational']
+
+    def download(self, engine=None, debug=False):
+        Script.download(self, engine, debug)
+        engine = self.engine
+
+        # files are nested in another baad_data folder
+        # important files considered (baad_data.csv,baad_methods.csv)
+        # relevant files can be added in the same manner
+
+        file_names = ["baad_data/baad_data.csv", "baad_data/baad_methods.csv"]
+        engine.download_files_from_archive(self.urls["BAAD"], file_names)
+
+        # creating data from baad_data.csv
+        engine.auto_create_table(Table("data", cleanup=Cleanup(correct_invalid_value, nulls=['NA'])),
+                                 filename="baad_data.csv")
+        engine.insert_data_from_file(engine.format_filename("baad_data.csv"))
+
+        # creating methods from baad_methods.csv
+        engine.auto_create_table(Table("methods", cleanup=Cleanup(correct_invalid_value, nulls=['NA'])),
+                                 filename="baad_methods.csv")
+        engine.insert_data_from_file(engine.format_filename("baad_methods.csv"))
+
+SCRIPT = main()
\ No newline at end of file


Test patch:
Repository: weecology/retriever

Issue:
Bioclim data missing hdr files when downloaded to path
 retriever download Bioclim -p



Discussion:

Resolving patch:
diff --git a/scripts/bioclim_2pt5.py b/scripts/bioclim_2pt5.py
--- a/scripts/bioclim_2pt5.py
+++ b/scripts/bioclim_2pt5.py
@@ -20,8 +20,12 @@ def download(self, engine=None, debug=False):
         if engine.name != "Download Only":
             raise Exception("The Bioclim dataset contains only non-tabular data files, and can only be used with the 'download only' engine.")
         Script.download(self, engine, debug)
-        file_names = ["bio%s.bil" % file_num for file_num in range(1, 20)]
+        file_names = []
+        for file_num in range(1, 20):
+            for ext in (['bil', 'hdr']):
+                file_names += ["bio{0}.{1}".format(file_num, ext)]
         self.engine.download_files_from_archive(self.urls["climate"], file_names)
         self.engine.register_files(file_names)
 
 SCRIPT = main()
+


Test patch:
Repository: weecology/retriever

Issue:
Can't download and extract Gentry dataset
If trying to download "Gentry Forest Transect Dataset" the retriever seems to download the data, but gets stuck when it comes in extracting AVALANCH.xls
Moreover force quit seems the only way to close the program. 
OS: OS X El Capitan Version 10.11.3 (15D21)
Machine: Macbook Pro Early 2015 13"



Discussion:

Resolving patch:
diff --git a/app/download_manager.py b/app/download_manager.py
--- a/app/download_manager.py
+++ b/app/download_manager.py
@@ -102,8 +102,9 @@ def progress(s):
 
             current_progress = progress(s)
             if current_progress:
-                (keepgoing, skip) = self.dialog.Update(current_progress, s)
-            else:
+            # download progress remaining-time disabled. causes bottle neck on Gentry ref: #396.
+            #     (keepgoing, skip) = self.dialog.Update(current_progress, s)
+            # else:
                 (keepgoing, skip) = self.dialog.Pulse(s)
 
             if not keepgoing:


Test patch:
Repository: weecology/retriever

Issue:
Add list of all available datasets to the documentation
It would be nice to have a page of the documentation that is an auto-generated list of all of the currently available datasets. Each item would have the text from the `name` attribute and be linked to the `url` or `ref` attribute if available.



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -1,5 +1,33 @@
-import sys, os
 from retriever import VERSION,COPYRIGHT
+from retriever.lib.repository import check_for_updates
+from retriever import SCRIPT_LIST
+
+# Create the .rst file for the available datasets
+datasetfile = open("datasets.rst", "w")
+datasetfile_title = """
+==================
+Datasets Available
+==================
+
+
+"""
+check_for_updates(graphical=False)
+script_list = SCRIPT_LIST()
+
+# write the title of dataset rst file
+datasetfile.write(datasetfile_title)
+
+# get info from the scripts
+for script_num, script in enumerate(script_list, start=1):
+    if script.ref.strip():
+        reference_link = script.ref
+    elif bool(script.urls.values()):
+        reference_link = script.urls.values()[0].rpartition('/')[0]
+    else:
+        reference_link = ""
+    datasetfile.write("| " + str(script_num) + ". **{}** \n| shortname: {}\n| reference: {}\n\n".format(script.name, script.shortname, reference_link))
+datasetfile.close()
+
 needs_sphinx = '1.3'
 
 # Add any Sphinx extension module names here, as strings.


Test patch:
Repository: weecology/retriever

Issue:
Change in FIA file naming
I just tried to install FIA for the first time in a while and got

```
=> Installing Forest Inventory and Analysis
Creating database FIA...
Downloading REF_SPECIES.CSV...
Creating table FIA.species...
Inserting rows to FIA.species: 2591 / 2591
Downloading /home/ethan/.retriever/raw_data/FIA/AL_SURVEY.ZIP...
"There is no item named 'AL_SURVEY.CSV' in the archive"
```

It appears that FIA changed from using `.CSV` to `.csv` inside their archives. Pull request coming shortly.



Discussion:

Resolving patch:
diff --git a/scripts/fia.py b/scripts/fia.py
--- a/scripts/fia.py
+++ b/scripts/fia.py
@@ -55,13 +55,13 @@ def download(self, engine=None, debug=False):
         for table in tablelist:
             for state, year in stateslist:
                 engine.download_files_from_archive(self.urls["main"] + state + "_" + table + ".ZIP",
-                                                   [state + "_" + table + ".CSV"])
+                                                   [state + "_" + table + ".csv"])
 
         for table in tablelist:
             print "Scanning data for table %s..." % table
             prep_file_name = "%s.csv" % table
             prep_file = open(engine.format_filename(prep_file_name), "wb")
-            this_file = open(engine.format_filename(stateslist[0][0] + "_" + table + ".CSV"), "rb")
+            this_file = open(engine.format_filename(stateslist[0][0] + "_" + table + ".csv"), "rb")
             col_names = this_file.readline()
             prep_file.write(col_names)
             column_names = [col.strip('"') for col in col_names.split(',')]
@@ -69,7 +69,7 @@ def download(self, engine=None, debug=False):
             this_file.close()
 
             for state, year in stateslist:
-                this_file = open(engine.format_filename(state + "_" + table + ".CSV"), "rb")
+                this_file = open(engine.format_filename(state + "_" + table + ".csv"), "rb")
                 this_file.readline()
                 for line in this_file:
                     values = line.split(',')


Test patch:
Repository: weecology/retriever

Issue:
check if file is already downloaded
When using the download engine and installing the data somewhere other than the working directory,
the retriever didn't check to see if the file was already there. This change checks the destination and
if the file is already there it does not download/move the file and reports that it already exists.

This also required changing the download engines regression tests so that they were still testing
the main behavior of the code.

References commit  #0912e5f7



Discussion:

Resolving patch:
diff --git a/scripts/bbs.py b/scripts/bbs.py
--- a/scripts/bbs.py
+++ b/scripts/bbs.py
@@ -82,8 +82,8 @@ def download(self, engine=None, debug=False):
             # Weather table
             if not os.path.isfile(engine.format_filename("weather_new.csv")):
                 engine.download_files_from_archive(self.urls["weather"],
-                                                   ["weather.csv"])
-                read = open(engine.format_filename("weather.csv"), "rb")
+                                                   ["Weather.csv"])
+                read = open(engine.format_filename("Weather.csv"), "rb")
                 write = open(engine.format_filename("weather_new.csv"), "wb")
                 print "Cleaning weather data..."
                 for line in read:
diff --git a/scripts/bbs50stop.py b/scripts/bbs50stop.py
--- a/scripts/bbs50stop.py
+++ b/scripts/bbs50stop.py
@@ -81,8 +81,8 @@ def download(self, engine=None, debug=False):
             # Weather table
             if not os.path.isfile(engine.format_filename("weather_new.csv")):
                 engine.download_files_from_archive(self.urls["weather"],
-                                                   ["weather.csv"])
-                read = open(engine.format_filename("weather.csv"), "rb")
+                                                   ["Weather.csv"])
+                read = open(engine.format_filename("Weather.csv"), "rb")
                 write = open(engine.format_filename("weather_new.csv"), "wb")
                 print "Cleaning weather data..."
                 for line in read:


Test patch:
Repository: weecology/retriever

Issue:
Can't install AmnioteLH dataset to MySQL
Here's the output:

```
retriever install mysql Amniote -u travis -d testdb
=> Installing Amniote life History database
Creating database testdb...
Creating table testdb.main...
INSERT INTO testdb.main (class, ordered, family, genus, species, subspecies, common_name, trait, trait_value) VALUES ('Aves', 'Accipitriformes', 'Accipitridae', 'Accipiter', 'albogularis', null, 'Pied Goshawk', 'litter_or_clutch_size_n', null);
(1406, u"Data too long for column 'trait' at row 1")
Done!
```

The table `main` for this script is also empty in MySQL:

```
mysql> use testdb
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables
    -> ;
+------------------+
| Tables_in_testdb |
+------------------+
| main             |
+------------------+
1 row in set (0.00 sec)

mysql> select * from main;
Empty set (0.00 sec)
```



Discussion:
@goelakash, I have installed the current copy of master. 
seems like it is working well on my system.

I get the same result as @goelakash. Ubuntu 16.04, MySQL 5.7.12.

Let me reset and test again

I have tried both unix and windows and they are working fine. If the error is persistent when using MYSQL try to play around with the setting of MYSQL.
you could start here "Error Code: 1406. Data too long for column - MySQL"

Weird. I'm seeing a similar issue for postgres, which would seem to suggest that we have a value that is longer than we're indicating somewhere:

```
ethan@stice:~$ retriever install postgres Amniote -u ethan -p XXXXXXXXX
=> Installing Amniote life History database
Creating database AmnioteDB...
Creating table AmnioteDB.main...
INSERT INTO AmnioteDB.main (class, ordered, family, genus, species, subspecies, common_name, trait, trait_value) VALUES ('Aves', 'Accipitriformes', 'Accipitridae', 'Accipiter', 'albogularis', null, 'Pied Goshawk', 'litter_or_clutch_size_n', null);
value too long for type character varying(20)

Done!
```

This is probably related to @goelakash's early question about the length of `ct_column` in chat.


Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -177,7 +177,7 @@ def auto_create_table(self, table, url=None, filename=None, pk=None):
             self.auto_get_datatypes(pk, lines, columns, column_values)
 
         if self.table.columns[-1][1][0][:3] == "ct-" and hasattr(self.table, "ct_names") and not self.table.ct_column in [c[0] for c in self.table.columns]:
-            self.table.columns = self.table.columns[:-1] + [(self.table.ct_column, ("char", 20))] + [self.table.columns[-1]]
+            self.table.columns = self.table.columns[:-1] + [(self.table.ct_column, ("char", 50))] + [self.table.columns[-1]]
 
         self.create_table()
 


Test patch:
Repository: weecology/retriever

Issue:
download_only w/path fails to use path argument when checking for file
When `download_only` checks to see if the file already exists before copying it, it ignores the path argument. This means that:

```
retriever download MoM2003 -p testdata
```

will keep overwriting the file in `testdata` if it exists, and it will not copy the file to `testdata` if the file exists in `.`.

Fixes this is probably just a little logic improvement in the `final_cleanup` function of `download_only`.



Discussion:

Resolving patch:
diff --git a/engines/download_only.py b/engines/download_only.py
--- a/engines/download_only.py
+++ b/engines/download_only.py
@@ -66,7 +66,9 @@ def final_cleanup(self):
                 file_path, file_name_nopath = os.path.split(file_name)
                 subdir = os.path.split(file_path)[1] if self.opts['subdir'] else ''
                 dest_path = os.path.join(self.opts['path'], subdir)
-                if os.path.abspath(file_path) == os.path.abspath(os.path.join(DATA_DIR, subdir)):
+                if os.path.isfile(os.path.join(dest_path, file_name_nopath)):
+                    print ("File already exists at specified location")
+                elif os.path.abspath(file_path) == os.path.abspath(os.path.join(DATA_DIR, subdir)):
                     print ("%s is already in the working directory" %
                            file_name_nopath)
                     print("Keeping existing copy.")


Test patch:
Repository: weecology/retriever

Issue:
Gentry url update
The url given for the gentry dataset at 

https://github.com/weecology/retriever/blob/master/scripts/gentry.py#L33

Should be changed from http://www.wlbcenter.org/gentry_data.htm (which no longer seems to exist) to http://www.mobot.org/mobot/research/gentry/welcome.shtml



Discussion:

Resolving patch:
diff --git a/scripts/gentry.py b/scripts/gentry.py
--- a/scripts/gentry.py
+++ b/scripts/gentry.py
@@ -30,7 +30,7 @@ def __init__(self, **kwargs):
                      "counts": ""}
         self.tags = ["Taxon > Plants", "Spatial Scale > Global",
                      "Data Type > Observational"]
-        self.ref = "http://www.wlbcenter.org/gentry_data.htm"
+        self.ref = "http://www.mobot.org/mobot/research/gentry/welcome.shtml"
         self.addendum = """Researchers who make use of the data in publications are requested to acknowledge Alwyn H. Gentry, the Missouri Botanical Garden, and collectors who assisted Gentry or contributed data for specific sites. It is also requested that a reprint of any publication making use of the Gentry Forest Transect Data be sent to:
 
 Bruce E. Ponman


Test patch:
Repository: weecology/retriever

Issue:
Remove extra newline characters from dataset before writing to tables
A few datasets, like `SonoranPerennials` have extra `\n` in one of their raw files (Line 67 in `Species.csv` for this one as an example), which results in the xml errrors from #551 .  Need to process these before adding to tables in `engine.py`.



Discussion:

Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -95,11 +95,26 @@ def source_gen():
             real_line_length = sum(1 for _ in len_source)
 
         total = self.table.record_id + real_line_length
+        pos = 0
         for line in real_lines:
             if not self.table.fixed_width:
                 line = line.strip()
             if line:
                 self.table.record_id += 1
+
+                # Check for single row distributed over multiple lines
+                val_list = self.table.split_on_delimiter(line)
+                while len(val_list) < len(self.table.get_column_datatypes()):
+                    line = line.rstrip('\n')
+                    if type(real_lines) != 'list':
+                        line += next(real_lines)
+                    else:
+                        line += real_lines[pos+1]
+                        real_lines.pop(pos+1)
+                    val_list = (self.table.split_on_delimiter(line))
+                pos += 1
+                assert(len(val_list) == len(self.table.get_column_datatypes()))
+
                 linevalues = self.table.values_from_line(line)
 
                 types = self.table.get_column_datatypes()


Test patch:
diff --git a/test/test_integration.py b/test/test_integration.py
--- a/test/test_integration.py
+++ b/test/test_integration.py
@@ -19,7 +19,12 @@
             'script': "shortname: crosstab\ntable: crosstab, http://example.com/crosstab.txt\n*column: record_id, pk-auto\n*column: a, int\n*column: b, int\n*ct_column: c\n*column: val, ct-double\n*ct_names: c1,c2",
             'expect_out': '"record_id","a","b","c","val"\n1,1,1,"c1",1.1\n2,1,1,"c2",1.2\n3,1,2,"c1",2.1\n4,1,2,"c2",2.2'}
 
-tests = [simple_csv, crosstab]
+extra_newline = {'name': 'extra_newline',
+              'raw_data': 'col1,col2,col3\n1,2\n,3\n',
+              'script': "shortname: extra_newline\ntable: extra_newline, http://example.com/extra_newline.txt",
+              'expect_out': '"col1","col2","col3"\n1,2,3'}
+
+tests = [simple_csv, crosstab, extra_newline]
 
 
 def setup_module():
@@ -65,3 +70,10 @@ def test_crosstab_from_csv():
     crosstab_module.SCRIPT.engine.disconnect()
     obs_out = file_2string("crosstab_crosstab.txt")
     assert obs_out == crosstab['expect_out']
+
+def test_extra_newline():
+    extra_newline_module = get_script_module('extra_newline')
+    extra_newline_module.SCRIPT.download(csv_engine)
+    extra_newline_module.SCRIPT.engine.disconnect()
+    obs_out = file_2string("extra_newline_extra_newline.txt")
+    assert obs_out == extra_newline['expect_out']