mozilla/datazilla — pull request #43 by jeads (2012-06-26 17:15:19+00:00)
Title: Handle objectstore errors

Once the objectstore was rolled out this morning and talos objects were being stored several bugs were found. The web service method that stores the objects was not storing any captured errors from json deserialization, it was also always returning a success message even when errors were generated. This caused objects with malformed json to be stored. When the process_objects command was run it breaks on the malformed json and leaves the unprocessed objects in a loading state. When a connection id is recycled process_objects continues to break leaving well formed json objects permanently in the loading state.

To fix this, I modified views.set_test_data() to explicitly capture/store the error flag and message. The object retrieval/marking sql statements used by process_objects were modified to only operate on objects with an error_flag of `N` so that json generating an error is never processed by process_objects.

Another change made was converting the timestamp column to an int to match perftest. This is the time format that talos and most of the other Mozilla database applications I've seen use. I would like to stay consistent with our handling of time across all content types in datazilla. This change will require rolling out a new object store table, to do this I will run the following SQL statement on all objectstore databases:

DROP TABLE IF EXISTS objectstore;

CREATE TABLE objectstore (
id int(11) NOT NULL AUTO_INCREMENT,
test_run_id int(11) NULL,
date_loaded int(11) NOT NULL,
processed_flag enum('ready','loading','complete') DEFAULT 'ready',
error_flag enum('N','Y') DEFAULT 'N',
error_msg mediumtext,
json_blob blob,
worker_id int(11),
PRIMARY KEY (id),
KEY test_run_id_key (test_run_id),
KEY processed_flag_key (processed_flag),
KEY error_flag_key (error_flag),
KEY worker_id_key (worker_id)
) ENGINE=Aria DEFAULT CHARSET=utf8;