guardian/deploy — pull request #54 by sihil (2012-12-06 18:01:47+00:00)
Title: Decouple internal and database representations

This request decouples the internal representation from the database representation. This will allow useful changes to be made out of step with database migrations.

This has required fundamental changes to the way in which logs are generated in riff-raff. It turned out that it was impossible to reliably build trees with the information provided. As a result it is impossible to migrate old logs over to the new format.

Most notably each log message now has an emission time generated UUID associated with it. Also, the messages that signal the end of a node's children (FinishContext/FailContext) are now issued as a child of the node, rather than as a sibling to the node. This makes it much easier to build a tree.

The mongo representation has been split into two collections. One for deploy summaries and one containing the log lines. The classes serialised using salat are entirely in a new representation file. the mapping logic is also bundled separately - able to convert to and from the DB representation. To summarise a deploy you just have to delete the associated log entries.

Thanks to the new format, many database queries are substantially cheaper (e.g. the history page no longer needs to read in the whole record, just a summary).

Also, the internal DeployRecord cache is now cleaned up once a deploy completes which should remove the known resource leak.

Testing pages have been added to facilitate migration and troubleshooting.

There is clean-up work to be done after this is pushed out. The internal representation has been changed in order to accommodate the new message IDs. This meant cloning the DeployRecord class to DeployV2Record so as not to break the serialisation of the previous format. Once the migration is complete this can be removed along with various bits of then redundant mapping code. It should be feasible to remove the V2 monikers at that stage. Also, the concept of a message stack can be removed as parent IDs are all that is needed from the message class to provide context.