Repository: weecology/retriever

Issue:
Mac App MySQL installations failing based on "Incorrect string value"
MCDB:

```
INSERT INTO MCDB.trapping (site_id, initial_year, final_year, n_sampling_months, trap_nights, months_of_sampling, pitfall_traps, small_traps, large_traps, snap_traps, notes) VALUES (1206, '1995', '1995', '1', '580', 'November', '1', '0', '0', '0', 'each pitfall trapline 100-500 m?; unclear how many or length');
(1366, "Incorrect string value: '\\xB2; unc...' for column 'notes' at row 1")
```

MarineSize (Barnes 2008):

```
INSERT INTO MarineSize.main (record_number, in_ref_id, individual_id, predator, predator_common_name, predator_taxon, predator_lifestage, type_of_feeding_interaction, predator_length, predator_length_unit, predator_dimension_measured, predator_standard_length, predator_fork_length, predator_total_length, predator_tl_fl_sl_conversion_reference, standardised_predator_length, predator_measurement_type, predator_length_mass_conversion_method, predator_length_mass_conversion_reference, predator_quality_of_length_mass_conversion, predator_mass, predator_mass_unit, predator_mass_check, predator_mass_check_diff, predator_ratio_mass_mass, si_predator_mass, diet_coverage, prey, prey_common_name, prey_taxon, prey_length, prey_length_unit, prey_conversion_to_length_method, prey_quality_of_conversion_to_length, prey_conversion_to_length_reference, si_prey_length, prey_dimension_measured, prey_width, prey_width_unit, prey_measurement_type, prey_mass, prey_mass_unit, prey_mass_check, prey_mass_check_diff, prey_ratio_mass_mass, si_prey_mass, prey_conversion_to_mass_method, prey_conversion_to_mass_reference, prey_quality_of_conversion_to_mass, geographic_location, latitude, lonitude, depth, mean_annual_temp, sd_annual_temp, mean_pp, sd_pp, reference, specific_habitat, notes_assumptions) VALUES (1, 'ATSH063', 1, 'Rhizoprionodon terraenovae', 'Atlantic sharpnose shark', 'ectotherm vertebrate', 'adult', 'predacious/piscivorous', 7.8000E+02, 'mm', 'fork length', 7.5433E+02, 7.8000E+02, 9.3990E+02, 'Fishbase (species)', 9.3990E+01, 'individual', 'M=0.0056SL^2.897', 'Bonfil et al. (1990)', 1, 1.5399E+03, 'g', 4.3453E+04, 4.1913E+04, 2.8218E+01, 1.5399E+03, 'all', 'teleosts/molluscs/crustaceans', 'teleosts/molluscs/crustaceans', 'mixed', 1.1259E+02, 'mm', null, 0, null, 1.1259E+01, 'length', null, null, 'individual', 1.4274E+01, 'g', 7.4699E+01, 6.0425E+01, 5.2333E+00, 1.4274E+01, 'M=0.01L^3', 'Generalised', 5, 'Apalachicola Bay, Florida', '29?40\'N', '85?10\'W', 30, 24.1, 4.2, 866, 214, 'Bethea et al (2004)', 'Coastal Bay', null);
(1366, "Incorrect string value: '\\xBA40'N' for column 'latitude' at row 1")
```

McGlinn2010:

```
INSERT INTO McGlinn2010.species (spnum, spcode, family, genus, species, variety, subspecies, spname, binomia_auth, trinomial_auth) VALUES (257, 'seneplat', 'Asteraceae', 'Packera', 'plattensis', '', '', 'Packera plattensis', '(Nutt.) W.A. Weber & A. L?ve', '');
(1366, "Incorrect string value: '\\xF6ve' for column 'binomia_auth' at row 1")
```

All of these datasets install fine using the source installation. When using the .app build from the CLI they also all report:

```
Couldn't create database (unsupported operand type(s) for +: 'NoneType' and 'str'). Trying to continue anyway.
```

but this appears to be reported by all of the datasets, including those that are successfully installed.



Discussion:
These errors are caused by unicode characters. Now we need to understand why these only cause issues in the App but not when the Retriever is built from source.

Perhaps related:
http://stackoverflow.com/questions/2108824/mysql-incorrect-string-value-error-when-save-unicode-string-in-django

The difference is caused by #145. Uninstalling pymysql and running the source build produces the same error. pymysql is handling the unicode inserts properly, but MySQLdb is failing on the unicode characters. Since we default to pymysql this is only an issue if pymysql isn't installed (for source installations) or isn't working for some reason (i.e., in our App).


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -97,6 +97,7 @@ def clean_version(v):
       setup_requires=['py2app'] if 'darwin' in p else [],
 
       # options
+      # optimize is set to 1 of py2app to avoid errors with pymysql
       options = {'py2exe': {'bundle_files': 1,
                             'compressed': 2,
                             'optimize': 2,
@@ -108,7 +109,7 @@ def clean_version(v):
                             'includes': includes,
                             'site_packages': True,
                             'resources': [],
-                            'optimize': 2,
+                            'optimize': 1,
                             'argv_emulation': True,
                             'no_chdir': True,
                             },


Test patch: