gem/oq-engine — pull request #1595 by micheles (2014-11-14 14:47:13+00:00)
Title: Fast scenario export

The export time of the GMFs depends very much on how much are packed the arrays in the table gmf_data.
For the scenario calculator we can safely assume that the generated GMFs fit in memory and that the amount of data is so small to not kill celery/rabbitmq (both assumptions are not safe for the event based calculator). Given such assumptions, we can return the GMFs to the controller node and we can regroup them so that they are stored in an optimal way in the database. For instance in the case of demos/risk/ScenarioRisk there are 1000 realizations and we can store arrays of 1000 elements instead of array of 1 element. With this trick the runtime of the demo (which is totally dominated by the export) goes down from 46 minutes to 4 minutes.

The tests are green: https://ci.openquake.org/job/zdevel_oq-engine/837