elasticsearch/elasticsearch-river-rabbitmq — pull request #30 by rore (2013-07-10 09:07:13+00:00)
Title: Custom commands

We use the RabbitMQ river to synchronize 2 ES clusters (having one online cluster and another backup cluster that is up-to-date). All indexing commands go through RabbitMQ to update the backup cluster.

From time to time we also need to update the mapping for types (when defining fields or new types). This is done programmatically in the application and not as a special management process. We need those mappings updates to also synchronize to the backup cluster (otherwise indexing operations will not have the desired results on that cluster), and to happen through the regular channel (the RabbitMQ update) so it does not require special configuration.

So, the solution is supporting the put mapping command in the RabbitMQ river itself.

In order to not affect the current batch-api functionality, this support is based on adding a custom header to the rabbitMQ message - X-ES-Command = mapping. This allows us to handle only this case, with no need to parse the batch requests (they are handled as before).

If the mapping command is found in the special header the payload is parsed. The first line should contain a meta data JSON (similar to the batch API format, only without the containing object as we already got the command in the header).
The next line is the mapping source.

If the index is not found it will create it with the requested mapping.

This addition also opens a way to support other custom commands in the RabbitMQ river (by adding handling to other values in the command header).