Repository: weecology/retriever

Issue:
Installs into MySQL using GUI fail sporadically
Often reports:

```
Error: (1146, u"Table 'bbs50.region_codes' doesn't exist")
```

The particular table reported varies, but it is always a table from a different dataset.

Also reports:

```
Error: Result length not requested length: Expected=8. Actual=4. Position: 1. Data Length: 5
```

The installation works properly using the CLI and the GUI problem occurs on both OSX and Linux.

On Linux, if you install via the CLI then the GUI will reinstall properly, but on OSX the GUI install fails even after a CLI install.

Occurs across datasets (tested with DelMoral, Zachman, MCDB, and MammalLH) but varies in frequency (DelMoral most of the time, Zachman half of the time, MCDB ~20%, MammalLH ~10%).

@bendmorris any ideas on this one?



Discussion:
One other clue I forgot. These installations work fine using the Mac .app package built from the current source.

The "result length not requested" part is a PyMySQL error, but not a very helpful one. At a low level, it's apparently trying to read 8 bytes from a packet when there are only 4 left. Without more detail I'm not sure what the cause could be.

More clues:

On Linux with the Retriever built from source using the GUI I first checked to see if Del Moral installed using other engines (and it succeeded on all of them, and then tried to install it 10 times into MySQL. Of these 10 attempted installs it installed correctly 5 times, errored 4 times (2 "BBS50.counts doesn't exist", 1 "Result length not requested length", 1 "Broken pipe"), and crashed once.

I then removed `pymysql`, rebuilt the Retriever, and tried in the installations again. All attempts to install resulted in:

```
Error: (2006, 'MySQL server has gone away')
```

or hanging while trying to connect to the database. All other datasets I tried worked fine, except for McGlinn2010 which displays the same behavior as DelMoral2010.

I just updated the issue description and title because this is a more general problem than it first appeared. @bendmorris I'm not really sure where to start on this one. Some sort of race condition with the multi-threading?

I'm not able to reproduce this right now, but I have a hunch that the "table_exists" function being called from the GUI is causing trouble by using the same cursor as the download thread. Can you try cherry-picking https://github.com/bendmorris/retriever/commit/8bd4625086d0a69a838c79826d144aad7860eafd and see if the issue is still there?

If this was the issue, I'm surprised that it only happened when using pymysql.

Cherry-picking that commit didn't solve the problem unfortunately.

When Del Moral fails it appears to always fail (3/3 failures reporting the 'Table XXX doesn't exist' error) is failing in the middle of importing the species table. Failing at different lines, but always in the same table. 


Resolving patch:
diff --git a/engines/mysql.py b/engines/mysql.py
--- a/engines/mysql.py
+++ b/engines/mysql.py
@@ -72,14 +72,12 @@ def insert_data_from_file(self, filename):
             return Engine.insert_data_from_file(self, filename)
         
     def table_exists(self, dbname, tablename):
-        try:
-            tablename = self.table_name(name=tablename, dbname=dbname)
-            test_statement = "SELECT * FROM " + tablename + " LIMIT 1"
-            self.cursor.execute(test_statement)
-            l = len(cursor.fetchall())
-            return l > 0
-        except:
-            return False
+        if not hasattr(self, 'existing_table_names'):
+            self.cursor.execute("SELECT table_schema, table_name FROM information_schema.tables WHERE table_schema NOT IN ('mysql', 'information_schema', 'performance_schema');")
+            self.existing_table_names = set()
+            for schema, table in self.cursor:
+                self.existing_table_names.add((schema.lower(), table.lower()))
+        return (dbname.lower(), tablename.lower()) in self.existing_table_names
         
     def get_connection(self):
         """Gets the db connection."""


Test patch: