gem/oq-engine — pull request #887 by larsbutler (2012-10-01 16:02:04+00:00)
Title: GMF storage optimization

This branch is related to https://bugs.launchpad.net/openquake/+bug/1046720.

We identified that the storage of GMFs was highly inefficient, and for a "reasonably-size" calculation it would take about 1TB of space to store just the GMFs.

These changes offer a 55% optimization with respect to the space requirements for GMFs. (Actually, the `complete logic tree GMF` was being stored as simply a giant collection of copies of the ground motion field data. This copying is no longer being done and instead, the complete LT GMF is being generated on export. So if you take that into account, that's a 75% optimization.)

Here are some notes I took and some stats I gathered. The test case I ran was a fairly small calculation, with 500 logic tree samples: http://pastebin.com/ZDGN6i7s

Before the optimization, the `gmf_node` table grew to 785MB. With indices and so on, the total database size was 1.36GB.

After the optimization, the `gmf` table (`gmf_node` was deleted and its data was restructured to fit into the `gmf` table) grew to only 335MB. Total database size was only 0.63GB.

This pull request requires https://github.com/gem/oq-engine/pull/886.