internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: weecology/retriever

Issue:
urllib.request.urlretrieve fails to download Gentry data
[Gentry site data](https://github.com/weecology/retriever/blob/master/scripts/gentry.py#L28) is failing and it is because of `urllib.request.urlretrieve` in [Engine](https://github.com/weecology/retriever/blob/master/lib/engine.py) `def download_file(self, url, filename):` 

I have attached a small script.

```
url = 'http://www.ecologicaldata.org/sites/default/files/gentry_sites_data.txt'

urllib.request.urlretrieve(url, "fileout")
```

`AttributeError: '_fileobject' object has no attribute 'readinto'`

ref: #502



Discussion:

Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -14,9 +14,12 @@
 import zipfile
 import gzip
 import tarfile
-import urllib.request, urllib.parse, urllib.error
 import csv
 import io
+if sys.version_info[0] >= 3:
+    from urllib.request import urlretrieve
+else:
+    from urllib import urlretrieve
 from retriever import DATA_SEARCH_PATHS, DATA_WRITE_PATH
 from retriever.lib.cleanup import no_cleanup
 from retriever.lib.warning import Warning
@@ -388,7 +391,7 @@ def download_file(self, url, filename):
             path = self.format_filename(filename)
             self.create_raw_data_dir()
             print("Downloading " + filename + "...")
-            response = urllib.request.urlretrieve(url, path)
+            urlretrieve(url, path)
 
 
     def download_files_from_archive(self, url, filenames, filetype="zip",


Test patch:
Repository: weecology/retriever

Issue:
urllib.request.urlretrieve fails to download Gentry data
[Gentry site data](https://github.com/weecology/retriever/blob/master/scripts/gentry.py#L28) is failing and it is because of `urllib.request.urlretrieve` in [Engine](https://github.com/weecology/retriever/blob/master/lib/engine.py) `def download_file(self, url, filename):` 

I have attached a small script.

```
url = 'http://www.ecologicaldata.org/sites/default/files/gentry_sites_data.txt'

urllib.request.urlretrieve(url, "fileout")
```

`AttributeError: '_fileobject' object has no attribute 'readinto'`

ref: #502



Discussion:
This is the same as the missing table part of #528. We had created this table from a pdf and posted it to the Ecological Data Wiki. When we upgraded the wiki a month or two ago we didn't move this table. I'll try to put in a PR to fix this soon.


Resolving patch:
diff --git a/scripts/gentry.py b/scripts/gentry.py
--- a/scripts/gentry.py
+++ b/scripts/gentry.py
@@ -25,7 +25,7 @@ def __init__(self, **kwargs):
         self.name = "Alwyn H. Gentry Forest Transect Dataset"
         self.shortname = "Gentry"
         self.urls = {"stems": "http://www.mobot.org/mobot/gentry/123/all_Excel.zip",
-                     "sites": "http://www.ecologicaldata.org/sites/default/files/gentry_sites_data.txt",
+                     "sites": "https://ndownloader.figshare.com/files/5515373",
                      "species": "",
                      "counts": ""}
         self.tags = ["Taxon > Plants", "Spatial Scale > Global",
@@ -42,7 +42,7 @@ def __init__(self, **kwargs):
     def download(self, engine=None, debug=False):
         Script.download(self, engine, debug)
 
-        self.engine.auto_create_table(Table("sites"), url=self.urls["sites"])
+        self.engine.auto_create_table(Table("sites"), url=self.urls["sites"], filename='gentry_sites.csv')
         self.engine.insert_data_from_url(self.urls["sites"])
 
         self.engine.download_file(self.urls["stems"], "all_Excel.zip")


Test patch:
Repository: weecology/retriever

Issue:
Checking for row over multiple lines creates install error for FrayJorge
```
ethan@stice:~$ retriever install --debug csv FrayJorge
=> Installing Fray Jorge community database
'list' object is not an iteratornuals.csv: 17900 / 1914258
Traceback (most recent call last):
  File "/home/ethan/anaconda/bin/retriever", line 9, in <module>
    load_entry_point('retriever===1.9.0-de', 'console_scripts', 'retriever')()
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/__main__.py", line 135, in main
    dataset.download(engine, debug=debug)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/templates.py", line 99, in download
    self.engine.insert_data_from_url(value)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/engine.py", line 637, in insert_data_from_url
    self.insert_data_from_file(find)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/engine.py", line 629, in insert_data_from_file
    self.add_to_table(data_source)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/engine.py", line 116, in add_to_table
    line += next(real_lines)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/future/builtins/newnext.py", line 62, in newnext
    iterator.__class__.__name__))
TypeError: 'list' object is not an iterator
```

This is triggered by 9f05ea314d36cce2672103313769d2e54ce1e5b6, which has logic to protect against the issue but it fails to activate due to checking type against `'list'` instead of `list`. Fix is inbound.



Discussion:

Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -112,7 +112,7 @@ def source_gen():
                 val_list = self.table.split_on_delimiter(line)
                 while len(val_list) < len(self.table.get_column_datatypes()):
                     line = line.rstrip('\n')
-                    if type(real_lines) != 'list':
+                    if type(real_lines) != list:
                         line += next(real_lines)
                     else:
                         line += real_lines[pos+1]


Test patch:
diff --git a/test/test_integration.py b/test/test_integration.py
--- a/test/test_integration.py
+++ b/test/test_integration.py
@@ -36,8 +36,12 @@
               'script': "shortname: skip_csv\ntable: skip_csv, http://example.com/skip_csv.txt\n*do_not_bulk_insert: True\n*column: a, skip\n*column: b, int\n*column: c, int",
               'expect_out': 'b,c\n2,3\n5,6\n'}
 
+extra_newline = {'name': 'extra_newline',
+                'raw_data': 'col1,col2,col3\n1,2\n,3\n',
+                'script': "shortname: extra_newline\ntable: extra_newline, http://example.com/extra_newline.txt",
+                'expect_out': 'col1,col2,col3\n1,2,3\n'}
 
-tests = [simple_csv, autopk_csv, crosstab, autopk_crosstab, skip_csv]
+tests = [simple_csv, autopk_csv, crosstab, autopk_crosstab, skip_csv, extra_newline]
 
 # Create a tuple of all test scripts and expected values
 # (simple_csv, '"a","b","c"\n1,2,3\n4,5,6')
Repository: weecology/retriever

Issue:
Checking for row over multiple lines breaks fixed-width tables
The `val_list = self.table.split_on_delimiter(line)` line added in 9f05ea314d36cce2 runs `split_on_delimiter` without checking if the table has a delimiter. When the table is fixed width this results in an error:

```
ethan@stice:~$ retriever install sqlite bbs
=> Installing USGS North American Breeding Bird Survey
Creating table BBS_species...
"delimiter" must be an 1-character string
Done!
```



Discussion:

Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -109,7 +109,7 @@ def source_gen():
                 self.table.record_id += 1
 
                 # Check for single row distributed over multiple lines
-                val_list = self.table.split_on_delimiter(line)
+                val_list = self.table.extract_values(line)
                 while len(val_list) < len(self.table.get_column_datatypes()):
                     line = line.rstrip('\n')
                     if type(real_lines) != 'list':
@@ -117,7 +117,7 @@ def source_gen():
                     else:
                         line += real_lines[pos+1]
                         real_lines.pop(pos+1)
-                    val_list = (self.table.split_on_delimiter(line))
+                    val_list = (self.table.extract_values(line))
                 pos += 1
 
                 linevalues = self.table.values_from_line(line)


Test patch:
Repository: weecology/retriever

Issue:
Add a code of conduct
I keep planning on doing this and not getting around to it so I'm opening an issue. As usual we can probably just copy @ctb: https://github.com/dib-lab/khmer/blob/master/CODE_OF_CONDUCT.rst



Discussion:
This is the latest version, http://contributor-covenant.org/version/1/4/  for comparison

Let's go with the latest version then.


Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -1,3 +1,13 @@
+from __future__ import print_function
+from __future__ import absolute_import
+from imp import reload
+import sys
+# sys removes the setdefaultencoding method at startup; reload to get it back
+reload(sys)
+if hasattr(sys, 'setdefaultencoding'):
+    # set default encoding to latin-1 to decode source text
+    sys.setdefaultencoding('latin-1')
+
 from builtins import str
 from retriever import VERSION,COPYRIGHT
 from retriever.lib.repository import check_for_updates
diff --git a/try_install_all.py b/try_install_all.py
--- a/try_install_all.py
+++ b/try_install_all.py
@@ -11,7 +11,7 @@
 from __future__ import absolute_import
 import os
 import sys
-from importlib import reload
+from imp import reload
 from retriever.lib.tools import choose_engine
 from retriever import MODULE_LIST, ENGINE_LIST, SCRIPT_LIST
 


Test patch:
Repository: weecology/retriever

Issue:
Refactor insert_statement in lib/engine
current status:

```
    def insert_statement(self, values):
        """Returns a SQL statement to insert a set of values."""
        columns = self.table.get_insert_columns()
        types = self.table.get_column_datatypes()
        columncount = len(self.table.get_insert_columns(join=False, create=False))
        insert_stmt = "INSERT INTO {} ({}) VALUES ".format(self.table_name(), columns)
        for rows in values:
            vals = rows
            insert_stmt2 = " ("
            for i in range(0, columncount):
                insert_stmt2 += "%s, "
            insert_stmt2 = insert_stmt2.rstrip(", ") + "),"
            n = 0
            while len(vals) < insert_stmt.count("%s"):
                vals.append(self.format_insert_value(None, types[n]))
                n += 1
            insert_stmt2 %= tuple([str(value) for value in vals])
            insert_stmt += insert_stmt2
        insert_stmt = insert_stmt.rstrip(", ") + ";"
        if self.debug:
            print(insert_stmt)
        return insert_stmt
```

This function could be refactor to improve readability 



Discussion:
This function is so confusing!

> This function is so confusing!

Yeah, definitely some serious technical debt in there.


Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -670,18 +670,13 @@ def insert_statement(self, values):
         types = self.table.get_column_datatypes()
         columncount = len(self.table.get_insert_columns(join=False, create=False))
         insert_stmt = "INSERT INTO {} ({}) VALUES ".format(self.table_name(), columns)
-        for rows in values:
-            vals = rows
-            insert_stmt2 = " ("
-            for i in range(0, columncount):
-                insert_stmt2 += "%s, "
-            insert_stmt2 = insert_stmt2.rstrip(", ") + "),"
-            n = 0
-            while len(vals) < insert_stmt.count("%s"):
-                vals.append(self.format_insert_value(None, types[n]))
-                n += 1
-            insert_stmt2 %= tuple([str(value) for value in vals])
-            insert_stmt += insert_stmt2
+        for row in values:
+            row_length = len(row)
+            # Add None with appropriate value type for empty cells
+            for i in range(columncount - row_length):
+                row.append(self.format_insert_value(None, types[row_length + i]))
+
+            insert_stmt += " (" + ", ".join([str(val) for val in row]) +"), "
         insert_stmt = insert_stmt.rstrip(", ") + ";"
         if self.debug:
             print(insert_stmt)


Test patch:
Repository: weecology/retriever

Issue:
Download only fails for `gwdd`
```
ethan@gandalf:~$ retriever download gwdd
=> Installing Zanne et al. Global wood density database.
Creating database GWDD...
Couldn't create database ('NoneType' object has no attribute 'execute'). Trying to continue anyway.
'table_name'
```

And the file doesn't appear.

From `try_install_all.py`:

```
('download', 'gwdd', KeyError('table_name',))
```



Discussion:

Resolving patch:
diff --git a/scripts/gwdd.py b/scripts/gwdd.py
--- a/scripts/gwdd.py
+++ b/scripts/gwdd.py
@@ -117,7 +117,8 @@ def format_value(s):
         self.engine.table = table
         self.engine.create_table()
         self.engine.add_to_table(data)
-        
+        self.engine.find_file("GlobalWoodDensityDatabase.xls")
+
         return self.engine
 
 SCRIPT = main()


Test patch:
Repository: weecology/retriever

Issue:
Remove unnecessary warning when using `download`
When running `retriever download` an unnecessary warning displays

```
ethan@gandalf:~$ retriever download Clark2006
Creating database Clark2006...
Couldn't create database ('NoneType' object has no attribute 'execute').
...
```

The command executes correctly, but this warning isn't really correct (there is no database to create since we're just downloading the raw data files) and could be distracting to users.



Discussion:
Could not replicate this error. 
If anyone else could try this and let me know. Thanks

To expand my report: I'm seeing this on Linux (Ubuntu 16.04) on both Python 2 and Python 3.


Resolving patch:
diff --git a/engines/download_only.py b/engines/download_only.py
--- a/engines/download_only.py
+++ b/engines/download_only.py
@@ -138,7 +138,7 @@ def dummy_method(self, *args, **kwargs):
                 'auto_create_table',
                 'insert_data_from_url',
                 }
-remove_methods = ['insert_data_from_file']
+remove_methods = ['insert_data_from_file', 'create_db']
 for name, method in methods:
     if (name not in keep_methods and
             'download' not in name and


Test patch:
Repository: weecology/retriever

Issue:
Add Python 3 to setup.py
We need to note in the setup.py that Python 3 is supported.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -72,7 +72,8 @@ def clean_version(v):
       classifiers=['Intended Audience :: Science/Research',
                    'License :: OSI Approved :: MIT License',
                    'Programming Language :: Python',
-                   'Programming Language :: Python :: 2', ],
+                   'Programming Language :: Python :: 2',
+                   'Programming Language :: Python :: 3',],
       packages=packages,
       package_dir={
           'retriever': ''


Test patch:
Repository: weecology/retriever

Issue:
Gentry site CEDRAL has data entry error
On line 130 of the CEDRAL data there is an extra `'` in the `voucher1` column. This is related to a parsing error in the raw Excel file that makes all following columns off by one, which results in the `Counts` column containing non-integer values (because the first size ends up in the counts column) and the stems table will be missing a single, 2.5, value.



Discussion:
@ethanwhite does this still hold? #416 
Yes. It happens for the following line:

```
count_id,line,species_id,site_code...
18681,4,4224,CEDRAL...
```

I'll submit a PR to fix this shortly.

Resolving patch:
diff --git a/scripts/gentry.py b/scripts/gentry.py
--- a/scripts/gentry.py
+++ b/scripts/gentry.py
@@ -25,7 +25,7 @@ def __init__(self, **kwargs):
         self.name = "Alwyn H. Gentry Forest Transect Dataset"
         self.shortname = "Gentry"
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '1.1'
+        self.version = '1.2'
         self.urls = {"stems": "http://www.mobot.org/mobot/gentry/123/all_Excel.zip",
                      "sites": "https://ndownloader.figshare.com/files/5515373",
                      "species": "",
@@ -122,6 +122,13 @@ def download(self, engine=None, debug=False):
                                               if not Excel.empty_cell(row[c])]
                         this_line["site"] = filename[0:-4]
 
+                        # Manually correct CEDRAL data, which has a single line
+                        # that is shifted by one to the left starting at Liana
+                        if this_line["site"] == "CEDRAL" and type(this_line["liana"]) == float:
+                            this_line["liana"] = ""
+                            this_line["count"] = 3
+                            this_line["stems"] = [2.5, 2.5, 30, 18, 25]
+
                         lines.append(this_line)
 
                         # Check how far the species is identified


Test patch:
Repository: weecology/retriever

Issue:
Reinstate Fia script (Forest Inventory and Analysis)
Forest Inventory and Analysis changed the file locations.
New species table url: https://apps.fs.usda.gov/fiadb-downloads/CSV/REF_SPECIES.csv
new main:

The new changes needed for FIA

        self.urls = {"main": "https://apps.fs.usda.gov/fiadb-downloads/CSV/CSV/",
                     'species': 'https://apps.fs.usda.gov/fiadb-downloads/CSV/REF_SPECIES.csv'}


Discussion:
@ethanwhite, should we make the changes in a separate PR, or include them in the current PR under review?
Separate PR.
Am waiting for `FIA` server to go up again, 
https://apps.fs.usda.gov/fiadb-downloads/CSV/CSV/AL_TREE.zip
Else we may switch to downloading  `.csv` files instead of `.zip`

Resolving patch:
diff --git a/scripts/forest-inventory-analysis.py b/scripts/forest-inventory-analysis.py
--- a/scripts/forest-inventory-analysis.py
+++ b/scripts/forest-inventory-analysis.py
@@ -7,12 +7,10 @@
 standard_library.install_aliases()
 
 import os
-import urllib.request, urllib.parse, urllib.error
-import zipfile
-from decimal import Decimal
+
 from retriever.lib.templates import Script
-from retriever.lib.models import Table, Cleanup, no_cleanup
-from retriever import HOME_DIR, open_fr, open_fw
+from retriever.lib.models import Table
+from retriever import open_fr, open_fw
 
 
 class main(Script):
@@ -21,10 +19,10 @@ def __init__(self, **kwargs):
         self.name = "Forest Inventory and Analysis"
         self.shortname = "forest-inventory-analysis"
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '1.2.0'
+        self.version = '1.3.0'
         self.ref = "http://fia.fs.fed.us/"
-        self.urls = {"main": "http://apps.fs.fed.us/fiadb-downloads/CSV/",
-                     'species': 'http://apps.fs.fed.us/fiadb-downloads/CSV/REF_SPECIES.csv'}
+        self.urls = {"main": "https://apps.fs.usda.gov/fiadb-downloads/CSV/",
+                     'species': 'https://apps.fs.usda.gov/fiadb-downloads/CSV/REF_SPECIES.csv'}
         self.tags = ["plants", "continental-scale", "observational"]
         self.citation = "DATEOFDOWNLOAD. Forest Inventory and Analysis Database, St. Paul, MN: U.S. Department of Agriculture, Forest Service, Northern Research Station. [Available only on internet: http://apps.fs.fed.us/fiadb-downloads/datamart.html]"
         self.description = """WARNING: This dataset requires downloading many large files and will probably take several hours to finish installing."""


Test patch:
Repository: weecology/retriever

Issue:
Check MySQL and Postgres credential files
In addition to allowing users to directly provide their MySQL and PostgreSQL credentials, it should also be possible for them to store these credentials in the usual places.

We should check information given by the user to the retriever first, and then fall back on the configuration files for usernames and passwords if they are not provided.

For PostgreSQL this is `~/.pgpass` with the format:

```
hostname:port:database:username:password 
```

See: https://wiki.postgresql.org/wiki/Pgpass. `*`s can be used in place of any of the `:` separated values.

For MySQL this is `~/.my.cnf` with the format:

```
[client]
user = root
password = yourpassword
```

See: https://dev.mysql.com/doc/refman/5.5/en/option-files.html. `.my.cnf` can contain a lot of additional configuration information so we'll need to look explicitly for `user =` and `password =`.



Discussion:
@ethanwhite @henrykironde I would like to work on this issue if it is still open
Go for it, I  will PR you a branch that i was working on, you can start from there
@jainamritanshu, the system with the branch is not with me but in case you need any clarification let me know
Sure I will start working on the issue right away

Resolving patch:
diff --git a/retriever/engines/mysql.py b/retriever/engines/mysql.py
--- a/retriever/engines/mysql.py
+++ b/retriever/engines/mysql.py
@@ -116,4 +116,4 @@ def get_connection(self):
         import pymysql.constants.CLIENT as client
         args['client_flag'] = client.LOCAL_FILES
         self.get_input()
-        return dbapi.connect(**args)
+        return dbapi.connect(read_default_file='~/.my.cnf', **args)


Test patch:
Repository: weecology/retriever

Issue:
install to msaccess error
When I try to install a dataset to msaccess I get the error:
Couldn't create table (no module named 'pypyodbc')

I've tried doing "pip install pypyodbc --upgrade" which seems to work, but doesn't help with the retriever error.

I'm on a windows 10 machine


Discussion:
Is this install done using the Windows installer (the .exe)? If so then I think we packaged the wrong ODBC package with it.
Yes I used the .exe
Thanks. @henrykironde will rebuild that package and have you test again.
after installing with the new .exe, I now get the error "Access Driver is not found"
@emchristensen - I can't remember, did we ever get this figured out?
I can't remember either.  Can you remind me how to test this, and I'll try it again?
I just tried it again and got the same error message -- Henry is going to check it out

Resolving patch:
diff --git a/retriever/engines/msaccess.py b/retriever/engines/msaccess.py
--- a/retriever/engines/msaccess.py
+++ b/retriever/engines/msaccess.py
@@ -27,6 +27,7 @@ class engine(Engine):
                       "Format of table name",
                       "[{db} {table}]"),
                      ]
+    placeholder = "?"
 
     def convert_data_type(self, datatype):
         """MS Access can't handle complex Decimal types"""
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -7,6 +7,8 @@
 
 current_platform = platform.system().lower()
 extra_includes = []
+if current_platform == "windows":
+    extra_includes = ["pypyodbc"]
 
 if os.path.exists(".git/hooks"):  # check if we are in git repo
     os.system("cp hooks/pre-commit .git/hooks/pre-commit")


Test patch: