openstate/open-cultuur-data — pull request #77 by bartdegoede (2014-12-09 11:39:45+00:00)
Title: Collection refresh mechanism

Collection "refresh" mechanism
=======================
Most sources don't support options to synchronize external copies of their data (such as the ones made available through the OCD API) with their "production" data (by returning only data that is added, modified or removed since a provided date(time), for instance).

In order to keep the collections in the OCD API up-to-date, **without** users experiencing downtime or inconsistent state of the index, this pull request contains code for a "refresh" mechanism. When adding or updating a collection, the data is no longer indexed in the existing index, but in a new one. Once the import has finished, the index alias used by the OCD REST API will be pointed to these newly created index. This way, users won't experience downtime, and updates can be "atomic".

Implementation details
------------------------------
In order to achieve this behaviour, we need to know when a particular import has finished. To do this in Celery's asynchronous environment, we make use of some of [Celery's primitives](http://celery.readthedocs.org/en/latest/userguide/canvas.html#the-primitives): every item is essentially turned into a `chain`ed task (transformer --> loader), which are executed one after another. These tasks are then `group`ed, so that each item can be processed in parallel. These `group`s are finally wrapped in a `chord`, which will execute a callback task (`ocd_backend.tasks.UpdateAlias`). This task is responsible for cleaning up (removing task results from the result backend; required for the workflow structure), as well as switching the alias to the new index, and removing the old one.

This functionality replaces the setup of the old pipeline transparently, and introduces a bit more bookkeeping on the Celery side.