NationalGenomicsInfrastructure/ngi_pipeline — pull request #63 by guillermo-carrasco (2014-08-22 17:23:33+00:00)
Title: Celery server

So, this PR fixes the issues that we had before.

Also... I have decided to change the backend for the messaging queue, moving from RabbitMQ to Redis. Here the why:
* Redis is *way* easier to configure... it took me 5 min to compile and run, while RabbitMQ... well... was more difficult.
* Yes, RabbitMQ is more supported than Redis and has a longer life, but Redis is [powerful enough](https://twitter.com/nick_craver/status/492661975416315904) for what we want and its community is growing. Also, we will use Redis as well for Logstash.

The big disatvantage is that Redis does not ensure that if the server crashes we will keep the messages[1]:

> Redis is quick to start, lightweight & fast broker but does Not support reliable delivery. Hence can be chosen for applications where in case system terminates, losing the information about tasks for a few minutes is Not critical.

Is that a problem for us? I don't think, in the case that the server broke, we just re-send the tasks and that's it. Is not that we are loosing message of the style "Pay to X person Y$", but more like "trigger the analysis of this run".

As you can see, the code has reduced a loot and it is actually easy to understand.

Vad tror du?

[1]: http://blog.langoor.mobi/django-celery-redis-vs-rabbitmq-message-broker/