mozilla/datazilla — pull request #26 by jeads (2012-05-25 21:50:14+00:00)
Title: Data transfer

Added a new manage.py command called transfer_data. You can take a look at the options with: manage.py transfer_data --help

The following command:

python manage.py transfer_data --source talos --target test --records 10 --start 20000 --debug

Will write out 10 json structures to stdout starting at id 20000. The json data is pulled from the talos project's test_data table. I used this command to transfer 10000 records to the test database. When the --debug flag is removed the data transfer is actually carried out.

This command required adding a new option to datasource, "chunk_min", that allows a user to specify a minimum id to start chunking from. The updates in /vendor were from running bin/generate-vendor-lib.py with the new datasource commit hash.

This data transfer into the test database validates the foreign keys on the test database. There was one UNIQUE key on the operating_system.name column that needed to be removed, there is already a UNIQUE key on (name, version) other than that, there were no data integrity errors found in the 10,000 records transferred.

Jeads