Repository: weecology/retriever

Issue:
Dataset install issues with SQL for some databases
E.g., Adler2007 installs fine into csv (and other text file engines):

```
ethan@stice:~$ retriever install csv Adler2007
=> Installing Kansas plant quadrats (Ecological Archives 2007)
Downloading species_list.csv...ain.csv: 163400 / 16340000 41
Done!ting rows to ./Adler2007_species.csv: 151 / 151
```

But when install into the SQL engines consistently fails with `invalid literal for int() with base 10: '30,20'`:

```
ethan@stice:~$ retriever install sqlite Adler2007
=> Installing Kansas plant quadrats (Ecological Archives 2007)
Creating table Adler2007_quadrat_inventory...
Creating table Adler2007_monthly_temp...ntory: 41 / 41
Creating table Adler2007_quadrat_info...: 110 / 110
Creating table Adler2007_monthly_ppt...o: 51 / 51
Creating table Adler2007_main...hly_ppt: 110 / 110
invalid literal for int() with base 10: '30,20'
Done!
```

```
ethan@stice:~$ retriever install postgres Adler2007
=> Installing Kansas plant quadrats (Ecological Archives 2007)
Creating database Adler2007...
Couldn't create database (schema "adler2007" already exists
). Trying to continue anyway.
Creating table Adler2007.quadrat_inventory...
Creating table Adler2007.monthly_temp...ntory: 41 / 41
Creating table Adler2007.quadrat_info...: 110 / 110
Creating table Adler2007.monthly_ppt...o: 51 / 51
Creating table Adler2007.main...hly_ppt: 110 / 110
invalid literal for int() with base 10: '30,20'
Done!
```

```
ethan@stice:~$ retriever install mysql Adler2007 -u root -p bats1e
=> Installing Kansas plant quadrats (Ecological Archives 2007)
Creating database Adler2007...
Creating table Adler2007.quadrat_inventory...
Creating table Adler2007.monthly_temp...ntory: 41 / 41
Creating table Adler2007.quadrat_info...: 110 / 110
Creating table Adler2007.monthly_ppt...o: 51 / 51
Creating table Adler2007.main...hly_ppt: 110 / 110
invalid literal for int() with base 10: '30,20'
Done!
```

The same issue is also present for `ArticBreedBird`, `BAAD`, and probably others.



Discussion:
This is being caused by the addition of a couple of `int()` wraps in `lib/engine` added in 6b52de0ff2f61b04200412c0eb1a7a1415328ea4. Specifically the two sets of

`if len(datatype) > 1 and int(datatype[1]) > 0:`

Given the way `datatype` was being treated here this change made reasonable sense, but there are currently values of `datatype[1]` that aren't integers. For the `Adler2007` `main` table the values of `datatype` at this point in the code are:

```
('char', 9)
('int',)
('char', 26)
('decimal', '30,20')
('decimal', '30,20')
('decimal', '30,20')
```

These are decimal values are indicating values for the left and right side of the decimal point, but they are either not being parsed properly or are valid and therefore we shouldn't be doing a conversion to `int()`.

`'30,20' > 0` evaluates to `True`.

Ah  `'30,20' > 0` evaluates to `True` in Python 2 and errors in Python 3 (as does `'30' >0`), hence the addition of the `int()`s.

I think we could remove the ints and make 0 -> "0" to get the old
functionality back.
On Jun 25, 2016 4:28 AM, "Ethan White" notifications@github.com wrote:

> Ah '30,20' > 0 evaluates to True in Python 2 and errors in Python 3 (as
> does '30' >0), hence the addition of the int()s.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/weecology/retriever/issues/537#issuecomment-228483348,
> or mute the thread
> https://github.com/notifications/unsubscribe/AGplhAPhLUIienhDVC5QkbreejT8cSIjks5qPGEDgaJpZM4I9_Cw
> .


Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -277,12 +277,12 @@ def convert_data_type(self, datatype):
             thistype = self.datatypes[thistype]
 
             if isinstance(thistype, tuple):
-                if len(datatype) > 1 and int(datatype[1]) > 0:
+                if len(datatype) > 1:
                     thistype = thistype[1] + "(" + str(datatype[1]) + ")"
                 else:
                     thistype = thistype[0]
             else:
-                if len(datatype) > 1 and int(datatype[1]) > 0:
+                if len(datatype) > 1:
                     thistype += "(" + str(datatype[1]) + ")"
         else:
             thistype = ""


Test patch: