mozilla/datazilla — pull request #30 by jeads (2012-06-04 02:12:41+00:00)
Title: Fix transfer data

There was a bug in the new manage.py method transfer_data. When making calls to DatazillaModel while stepping through an iterator from datasource, where two DatazillaModels are being used to communicate to two different databases, the MySQL-python execute function blocks between the two databases even when commit is called after each transaction. I'm not sure why this happens, I could be misinterpreting the symptoms but I think I found a reasonable fix. I used the data_iter.sqlChunks attribute which holds a list of all of the sql statements to execute in a chunking operation. Data is retrieved from the source, disconnect is called, and then data is loaded into the target. This solved the blocking problem. I also modified the execute function in datasource so it takes a new option called chunk_total, which allows the caller to specify a total number of rows to return in a chunking transaction. This is cleaner than having callers deal with it.

I fixed a bug in datasource that prevents debug_show from being set to False explicitly. You can now actually turn off debugging through local.py by setting DEBUG to False.

I added a new option to the execute function in datasource, executemany. When set to True, it uses MySQL-python's executemany for multi-row inserts. You can pass a list of lists of mixed types (strings, ints, floats) that will be passed to executemany, this is way more efficient than doing individual row inserts. I updated DatazillaModel._setTestAuxData() and DatazillaModel._setTestValues() which were the worst row level insert offenders. This gives a decent performance boost and reduces inefficient database operation code. Now we just need to cache the reference data and loadTestData will be a reasonable method!

I fixed populate_test_collections.py so it will now work with the new version of models.py.

Also, I was transferring data to mariadb over the weekend and the data transfer job died when I had to restart memcached. This was caused by the active_status column being removed from the database. I updated DataSource and DataSourceManager in models.py so they don't reference active_status. This is a critical update that needs to be pushed to s4n4 so the web service will not break if we restart memcached.