Repository: weecology/retriever

Issue:
[Bug] Data transformations with install_csv
I noticed that data was importing slightly different on linux than on my Mac laptop for a particular data file and it seems to be an issue with how the CSV file is being generated on my Mac.

In all cases, I'm using retriever v2.3.0, and running from within Python3.
1. I use retriever to download dataset "veg-plots-sdl":
```python
import retriever
retriever.install_csv("veg-plots-sdl")
```

2. Then I look at the "Photo_info" file. On my mac, line 9 of the data (line 10 of the file) shows:
```
9,375,11/14/1988,Turner,9:30,Graphlex XL,,58,,280,up,20,1.45,2,0,0,,0
```
and on linux, it shows:
```
9,375,11/14/1988,Turner,9:30,Graphlex XL,,58,,280,up,2_0,1.45,2,0,0,,0
```
**notice the entry in column 12, which shows `20` vs. `2_0`**

3. Looking at the `veg_plots_sdl.json` script, I see the entry for Photo_info as:
```json
            "name": "Photo_info",
            "schema": {},
            "url": "https://ndownloader.figshare.com/files/5625354"
```

When I use that link to download the file directly, it seems to match up with the version of the file when downloading on Linux:
```
9,375,11/14/1988,Turner,9:30,Graphlex XL,NA,58,,280,up,2_0,1.45,2,0,0,NA,0
```

**Here, the NAs are explicitly present, but are not present when examining the CSV files downloaded by retriever.**

This suggests to me that there is some auto-conversion of CSVs operating under the hood of retriever, and that it does things slightly differently depending on OS. Perhaps, the missing values in the first rows of data cause that column to be set differently as a numeric or character column by default, and so the underscore gets dropped in one OS, but not another.


Discussion:
Thank you @ha0ye, this needs some redesign.
```Python
Python 3.6.5
>>> 2_0
20
>>> 
Python 2.7.15 
>>> 2_0
  File "<stdin>", line 1
    2_0
      ^
```
I guess we shall have to change the way we detect the data type in https://github.com/weecology/retriever/blob/master/retriever/lib/engine.py#L265

Resolving patch:
diff --git a/retriever/lib/engine.py b/retriever/lib/engine.py
--- a/retriever/lib/engine.py
+++ b/retriever/lib/engine.py
@@ -737,7 +737,6 @@ def get_input(self):
                 if not os.path.exists(self.opts['data_dir']):
                     os.makedirs(self.opts['data_dir'])
 
-
     def insert_data_from_archive(self, url, filenames):
         """Insert data from files located in an online archive. This function
         extracts the file, inserts the data, and deletes the file if raw data


Test patch:
diff --git a/test/test_integration.py b/test/test_integration.py
--- a/test/test_integration.py
+++ b/test/test_integration.py
@@ -401,6 +401,47 @@
     'expect_out': ['aa,bb,c_c', '1,2,3', '4,5,6']
 }
 
+underscore_csv = {
+    'name': 'underscore_csv',
+    'raw_data': ['a,b,c',
+                 '1,2,y',
+                 '4,5,2_0'],
+    'script': {"name": "underscore_csv",
+               "resources": [
+                   {"dialect": {"do_not_bulk_insert": "True"},
+                    "name": "underscore_csv",
+                    "schema": {
+                        "fields": [
+                            {
+                                "name": "record_id",
+                                "type": "pk-auto"
+                            },
+                            {
+                                "name": "a",
+                                "type": "int"
+                            },
+                            {
+                                "name": "b",
+                                "type": "int"
+                            },
+                            {
+                                "name": "c",
+                                "type": "char"
+                            }
+                        ]
+                    },
+                    "url": "http://example.com/underscore_csv.txt"
+                    }
+               ],
+               "retriever": "True",
+               "retriever_minimum_version": "2.0.dev",
+               "version": "1.0.0",
+               "urls": {"underscore_csv": "http://example.com/underscore_csv.txt"}
+               },
+    'expect_out': ['record_id,a,b,c', '1,1,2,y', '2,4,5,2_0']
+}
+
+
 tests = [
     simple_csv,
     comma_delimiter,
@@ -412,7 +453,9 @@
     autopk_crosstab,
     skip_csv,
     extra_newline,
-    change_header_values]
+    change_header_values,
+    underscore_csv
+]
 
 # Create a tuple of all test scripts with their expected values
 test_parameters = [(test, test['expect_out']) for test in tests]