celery/celery — pull request #2414 by javrasya (2014-12-11 15:00:01+00:00)
Title: High available celery beat

This is related with the issue #1495 and PR #1888 . In PR #1888 requested by @magmax is still opened.

I see It is not accepted because it does not support other than `AMQP`. Actually the solution can not be implemented for other type of brokers easily.

In my solution, there is a method used like in broker backends. There will be many implementation as they are developed. There is implementation only for AMQP at the moment. I added Redis strategy to be implemented as soon. The solution in @magmax 's PR is helped me a lot to implement it for AMQP.

I saw, tasks are applying in schedulers. So I added a scheduler which is `HAScheduler`(High Available Scheduler). This one is checking whether current node is master or not according to chosen strategy. If current node is master defined by the chosen failover strategy, normal way is used. There is no change for that case. But, if the node is slave defined by the chosen failover strategy, beat will think, task is applied, but it won't actually be applied.

To decide what strategy is used, broker backend is used by parsing `BROKER_URL` as default. This can also be configured with a celery config which is `CELERYBEAT_ FAILOVER_BROKER_URL`. So you can use different broker for your celery cluster and beat failover strategy. If you don't give, it'll be using BROKER_URL.

Now, you can run multiple celery beat as you want. One of them will be master. If master goes down, one among still up an running slave beats, will be new master by taking the throne with the way defined in the chosen failover strategy.

I thought, beat code is not structured well. Because of failover strategy and its factory class, I just moved **celery/beat.py** into **celery/beat/__init__.py** and I implemented all these codes under **celery/beat** module. It can still be accessed by `from celery import beat`or `from celery.beat import *`. I hope that doesn't cause another problem. If it does, I can change beat module name which I created.