Repository: weecology/retriever

Issue:
Database connectivity problems with MySQL some databases
On Ubuntu 13.04 (MySQL 5.5.31) and Windows 7 (MySQL 5.6) databases with multiple tables are failing to import. For BBS on Windows the Retriever just hangs saying "Connecting to Database".

Error messages on Linux are of the form:

```
Failed bulk insert ((2013, 'Lost connection to MySQL server during query')), inserting manually
INSERT INTO Gentry.sites (site_id, site_name, country, state_province, continent, abbreviation, lat, lon, min_elev, max_elev, precip) VALUES (1, 'Kitlope 1', 'Canada', 'British Columbia', 'North America', 'KITLOPE1', 53.07, -127.83, 20, 20, 2100);
(2013, 'Lost connection to MySQL server during query')
```

On Windows if the Retriever doesn't hang an error message like this occurs:

`error 10053 An established connection was aborted by the software in your host machine.`



Discussion:
This seems to be an issue with pymysql closing the connection before creating the second table for some reason. Doesn't happen if MySQLdb is swapped in.

Are there any serious challenges for building or distribution that would be caused by switching back to MYSQLdb? 

The swap was two years ago: https://github.com/weecology/retriever/commit/2931a2d - I only vaguely remember MySQLdb having installation issues on Windows and Mac. Maybe they've been solved in the past two years? I have no idea. 

PyMySQL is pure python, so it doesn't require any compilation and we're less likely to have installation/packaging/segfault issues from a pure python library. Rather than swapping, another possibility would be submitting this as an issue to PyMySQL (https://github.com/PyMySQL/PyMySQL) as good open source citizens. If their connections aren't working the same way as other libraries that implement the DB API, it's a bug. I'm sure they'd appreciate it, and they seem to be pretty active still. In the meantime, I can try to find a temporary workaround.

Thoughts?

Perhaps not surprisingly a basic test using a simple script that creates a database, creates a new table, adds some data to it, creates another new table, and adds some data to it, works just fine:
https://gist.github.com/ethanwhite/69118587296c985d27ca

So, this is probably not a PyMySQL bug, at least not a simple one. As we talked about we should check to see if the multithreading is causing the issue or something else.

I also noticed that Gentry fails on the first table while BBS fails on the second:

``` bash
ethan@kaylee:~$ retriever install mysql gentry
=> Installing Alwyn H. Gentry Forest Transect Dataset
Creating database Gentry...
Creating table Gentry.sites...
Inserting data from gentry_sites_data.txt...
Failed bulk insert (an integer is required), inserting manually
INSERT INTO Gentry.sites (site_id, site_name, country, state_province, continent, abbreviation, lat, lon, min_elev, max_elev, precip) VALUES (1, 'Kitlope 1', 'Canada', 'British Columbia', 'North America', 'KITLOPE1', 53.07, -127.83, 20, 20, 2100);
(1156, u'Got packets out of order')
Done!
```

``` bash
ethan@kaylee:~$ retriever install mysql BBS
=> Installing USGS North American Breeding Bird Survey
Creating database BBS...
Creating table BBS.species...
Inserting rows to BBS.species: 1464 / 1464
Creating table BBS.routes...
Inserting data from routes_new.csv...
Failed bulk insert (an integer is required), inserting manually
INSERT INTO BBS.routes (countrynum, statenum, route, active, lati, loni, stratum, bcr, landtypeid, routetypeid, routetypedetailid) VALUES (840, 2, 1, 1, 34.8686880, -87.6041417, 14, 27, 0, 1, 1);
(1156, u'Got packets out of order')
```

FYI, the error switched to 'Got packets out of order' with the newest PyMySQL, but it appears to be the same basic problem. 

If it's happening from the command line, it's not due to multithreading; if I recall correctly, multithreading is only used by the GUI.

OK, it turns out that this isn't a multi-table thing per se. Woods2009 installs just fine.

DelMoral2010 fails:

``` bash
ethan@natasha:~$ retriever install mysql DelMoral2010 -u root
=> Installing Vegetation plots - del Moral, 2010
Creating database DelMoral2010...
Creating table DelMoral2010.species_plot_year...
Inserting rows to DelMoral2010.species_plot_year: 148155 / 148155
Creating table DelMoral2010.species...
Inserting data from MSH_SPECIES_DESCRIPTORS.csv...
Failed bulk insert (an integer is required), inserting manually
INSERT INTO DelMoral2010.species (family, species_and_authority, common_name, synonymy_and_comment, raw_code, nlspn_code, succession_status, higher_taxon, duration, growth_form, life_form, clonal, dispersal, origin, wet, location_and_disturbance) VALUES ('Pinaceae', 'Abies lasiocarpa (Hook.) Nutt.', 'Subalpine fir', '', 'Abilas', 'ALBL', 'Seral; climax', 'Conifer', 'PL', 'ET', 'Phan-scap', 'O', 'Wind; rodent', 'Native', 'FACU', 'BUCA Tephra;BUCA Lahar;BUCA Scour;PICR Scour;ABPL Blast;PUPL Blast;STRD Blast');
(1156, u'Got packets out of order')
```

The commonality here seems to be that a bulk insert fails and the attempt to fall back results in an error. Still investigating.


Resolving patch:
diff --git a/engines/mysql.py b/engines/mysql.py
--- a/engines/mysql.py
+++ b/engines/mysql.py
@@ -66,6 +66,7 @@ def insert_data_from_file(self, filename):
                 self.cursor.execute(statement)
             except Exception as e:
                 print "Failed bulk insert (%s), inserting manually" % e
+                self.disconnect() # If the execute fails the database connection can get hung up
                 return Engine.insert_data_from_file(self, filename)
         else:
             return Engine.insert_data_from_file(self, filename)


Test patch:
diff --git a/test/test_regression_cli.py b/test/test_regression_cli.py
--- a/test/test_regression_cli.py
+++ b/test/test_regression_cli.py
@@ -11,6 +11,7 @@
                       'DelMoral2010' : '606f97c3ddbfd6d63b474bc76d01646a',
                       'MoM2003' : 'ef0a31c132cfe1c6594739c872f70f54'},
               'mysql': {'AvianBodySize' : 'f60ac93d9be4671dbef77da9d10676b8',
+                        'DelMoral2010' : 'f241fd296130512d4e1029376b58a4ea',
                         'MoM2003' : '9728728d72af4c21a2a6e29fec3edb48'},
               'postgres': {'AvianBodySize' : '60c252af74d914e3c15fa9af43edefca',
                          'DelMoral2010' : '1e8de8fa3ddbd4ca3a7cab921926e70e',