mozilla/datazilla — pull request #42 by jeads (2012-06-25 23:21:53+00:00)
Title: Rollout objectstore

The following series of operations will need to be performed to roll out the new objectstore to the development environment.

1.) Execute the following alter statements on the perftest databases:

`ALTER TABLE test_value ADD COLUMN status tinyint NOT NULL DEFAULT 1;`
`ALTER TABLE test_value ADD KEY status_key (status);`

`ALTER TABLE test_aux_data ADD COLUMN status tinyint NOT NULL DEFAULT 1;`
`ALTER TABLE test_aux_data ADD KEY status_key (status);`

`ALTER TABLE test_option_values ADD COLUMN status tinyint NOT NULL DEFAULT 1;`
`ALTER TABLE test_option_values ADD KEY status_key (status);`

These statements add a `status` column to the core data tables. This will allow us to ignore data using sql without having to perform physical deletions.

2.) Install httpd, memcached, wsgi, and datazilla on s4n1 (completed)

3.) The url that talos is using for data submission looks like this:
`http://10.8.73.29/views/api/load_test`

The `views` string in this url, is used as the project identifier in the current version of datazilla. I added a conditional to `views.py` that forces the the project to be `talos` if `views` is used as the project name. This will give us compatibility with the old url structure talos is using. Once the new url format is rolled out this conditional should be removed.

4.) Change existing talos project name to 'talosv1' in datazilla.datasource and drop the unused talos objectstore database. This will point at the graphs_exp database, we can eventually drop this database entirely but should transfer data from the last month to the new talos database beforehand so we have data to develop with. This might not be necessary, depends on how fast we accumulate data.

NOTE: As of this weekend we ran out of disk space on s4n4. This was from the talos data in graphs_exp exceeding the VM's capacity. I physically deleted all test_run data with an id <= 350,000. This should account for ~1/2 of the data. This requires rebuilding the index to get MySQL to free up the disk space. I mysqldumped graphs_exp after the physical deletion and will manually load it after this upgrade is complete. I deleted all data tables in graphs_exp after the mysqldump was complete so that all other projects are functional. We might get away with just loading the contents of the test_data table in graphs_exp and then use this to transfer objects from, this would save a lot of space in our development environment.

5.) Create a new project called 'talos'.

6.) Start httpd on s4n1. Confirm data ingestion in the talos objectstore.

7.) Install cron to run process_objects for all dev projects on s4n4: talos, b2g, camd, carljm, jeads, jetperf, xperf, peptest, secreview, stoneridge.

8.) update datazilla repo on s4n3 and update the memcache.

10.) Remove the test_data table from all development databases.

11.) POST json data to each project and confirm that the object is transferred to the corresponding project's `perftest` database to confirm success.