Package: celery-amqp-events
Celery-AMQP-Eventsis a library that implements voluntary events handling
on top ofCelery.AMQP-based robustness of event handlingCelery tasks interfaceAnti-flood tacticsInstallationpipinstallcelery-amqp-eventsConfigurationPass a unique "service name" toCelery()instance for each service that
has event handlers (seeamqp_events.celery:events_app).Tell celery withimportssetting where to find event handlers.Configure broker connection and other celery settings.Leave result backend empty - each event may have multiple consumers,
event result is meaningless in this case.You absolutely need to set separate name for each service that consumes
events, because without that each fired event will be handled only by
single randomly chosen service, because your services will share same queue
for this event.fromamqp_events.celeryimportevents_appapp=events_app("service_name",# important in multi-service environmentimports=['demo.tasks'],# modules where to find event handlersbroker_url='amqp://guest:guest@rabbitmq:5672/',)Adding events and handlersfromdemo.celeryimportapp@app.event('service_name.object_name.event_name')defeven_number_generated(number:int):# You may add some validation logic in event function body;ifnumber%2!=0:raiseValueError("number is not even")@app.handler('service_name.object_name.event_name')defon_even_number_generated(number:int):# Handle event somehowprint(f"even number{number}generated")RunningStart ordinary celery worker for your consumer serviceNote thatmingle,gossipandheartbeatshould be disabled if not used.
These algorithms use broadcast events, which means that you'll haveN^2messages in RabbitMQ forNworkers without any purpose.celeryworker-Ayour_service.celery\--without-mingle--without-heartbeat--without-gossipSending eventsimportrandomfromdemo.eventsimportnumber_is_eventry:number_is_even(random.randint(0,100))exceptValueError:print("oops, number was odd")RobustnessIf event fails with unhandled error, it is retried to separate queue with
exponential backoff.Backoff is used to prevent resources exhausting (like free http connections)If no retry attempts left, unhandled event is moved to "archive" queueArchive is used to catch messages which always produce an error in consumer;
these messages can be manually retried when fix is released.Archive is limited both by message TTL and message count limit, so alerts
should exist.Retry is done via separate queue because of multiple reasons:usingcountdownforces consumer to keep "unacknowledged" events in
memory, which is bad for load balancing and resource usage.retrying to same queue will slow down event processing if retry
probability is high enoughtwo faulty consumers retrying same event withsame routing keywill
cause exponential growth of message count in RabbitMQ because message is
split to multiple messages when published from same exchange to multiple
queues.By default, some fault-tolerance celery settings are enabled:task_acks_latewill delay task acknowledgement till end of processingtask_reject_on_worker_lostwill preventackif worker was killedTask.autoretry_foris set to retry on anyExceptionautomaticallydisablingtask_acks_on_failure_or_timeoutforces Celery to reject
failed messages ifautoretry_forfailed to handle this.confirm_publishinbroker_transport_optionswill block producer till
broker will confirm that it received incoming message.Delay on broker sideCelery`s default retry mechanics is:construct a copy of currently handling messagechange arguments or options for the new message fromretryparamspublish the copy to the same exchange and routing_key found in messagedelivery_infoacknowledge current messageafter receiving the new message, celery puts it to an in-memory queue to wait
until message ETA, and then pass this message to a first free worker.All this is becauseAMQPprotocol doesn't define any delay-related properties.
Currently,RabbitMQhas a plugin for delayed messages and some tricky schemes
that allow to delay message delivery on the broker side. We use one of these
schemes.Retry causesA Celery task could be retried in different cases:manual retry from task implementationretry on an exception listed inTask.autoretry_foralso a task may be rejected ifretry()failed, if task task failed or if
task has timeoutedRetry routingTo implement broker-side delaysEventsCelerydeclares a set of retry exchanges
and queues:each exchange and queue is prefixes with "service name"each queue is declared withx-message-ttlthat is equal to2^n, wherenisAMQP_EVENTS_MAX_RETRIESenv variable andTask.max_retriessettings
for every task handler.each queue also definesx-dead-letter-exchangeargument, which points to
"recover" exchangethis "recover" exchange hastopictype and is bound to same queues as
default "events" exchange"retry" exchange has typefanout, so any message published to this exchange,
regardless it's routing key, will be routed to corresponding queuewith "message ttl" set up, retried message will expire in2^nseconds,
which is basically exponential backoff implementationbecause of "dead letter exchange", expired message is moved torecoverexchange, from which it is routed again to initial event queue via initial
message routing key.Reject handlingIt's important to retry rejected messages when something goes wrong.Each celery task queue is declared withx-dead-letter-exchangeargument,
which points to first "retry" exchange.As described in "retry routing" section, rejected message is moved by RabbitMQ
to a retry queue; after a second it expires and is moved again to "recover"
exchange, and after that to initial event queue.ArchivingBy default, Celery just drops a task whenMaxRetriesExceededErrorhappens,
but we want to archive such events for some time:EventsCelerydeclares afanout"archive" exchange and corresponding queue"archive" exchange and queue are also prefixed with "service name""archive" queue is declared withx-message-ttlandx-max-lengtharguments,
and message archive storage is limited both by time and message count.WhenEventsCeleryfinds that task retries count is exceeded, it retries
this task to "archive" exchange.CaveeatsMessage reordering. What if broker-side delay will shuffle normal message
ordering? This may break message ETA handling of change order of events.By default, Celery receives retried and delayed message and keeps it in
memory till message ETA. Before ETA it handles other incoming messages,
and if these new messages have later ETA, they will be scheduled at
corresponding time.This is how things work when there are less thanprefetch_multipliermessages in an incoming queue. If there are lot's of incoming messages,
retried messages will arrive behind ETA time and of cource this changes
event order.With broker-side delay implementation taskmessage-ttlis guaranteed to
be less or equal to task ETA, so for celery worker the situation will
look as a very long amqpbasic.publishcall. If there are no messages,
celery-side ETA logic will apply and the task will start in time.Queue count.RabbitMQis sensitive to a total queue count. What if we
declare too much queues?Total queue count is linear to a max retries count and to events count.Each retry queue is available to any Celery task and implements a delay
equal to a degree of 2. 20 queues provide max delay more than 23 days,
but if somebody needs milliseconds delays, he will need 10 more queues.By default each event is routed to it's own queue, but withtopicexchange you can manually set up a set of queues which handle groups of
event types. You can even manually setup a single queue for each handled
event.Worst case needsN + M + 1queue, where N ismax_retrieslimit andMis number of handled events.Best case needs2queues, if max retries is set to 1 and all events are
routed to single queue. "Archive" queue may be also disabled.Celery internals. What ifCelerywill change and all this stuff will break?To trigger auxiluary queues declaration we use celery signals, which we
consider as public interface.To declare queues we useCelery.broker_connection()default channel and
this could be changed to use separate amqp connection.To bind event queues we extendCelery.conf.task_queueswithkombuentities, which provide public interface for queue arguments and
additional bindings.To perform retries, we changeTask.retryarguments, this is also public
interface.Other things set up to provide more robust events handling are described
inConfiguration and defaultssection of Celery documentation.At last, we have plans to run integration tests with real celery workers,
not only unit-tests for mocked celery internals.Broker-side ProsObservability. There is no way to distinct delayed task and task in a
deadlock state except celery events mechanics. RabbitMQ API provides clear
way to monitor a number of retried and archived messages.Lesser memory usage. Default Celery implementation may keep thousands of
delayed messages in worker memory. Also it affects AMQP channelqosparameter, which could not be more that 1000 - this messes things up.
Broker-side delayed messages are stored on disk inRabbitMQdb.Autoretry for failed tasks. By default, each rejected message is lost forever.
With our setup it will be retried, which is good for delivery guarantees but
is also bad is message handling breaks constantly (i.e. in case of
persistent celery failure). This should be handled carefully.Robustness. Because of asynchronous nature of events handling pattern, we
do all efforts to ensure that every single-time fired event is not lost
somewhere between network failures and unhandled exceptions in business
logic. It's worth mentioning thatRabbitMQis considered as durable system
and we don't touch upon broker failures.ConsCelery internals. Despite public celery interface usage, we can't guarantee
that changes in default celery behavior won't affect broker-side retry model
as Celery authors decide to move fast and break things in 5.x and later
versions.Complex logic. Debugging such complex algorithms requires deep RabbitMQ
architecture and celery-amqp-events code knowledge. It's not trivial to
find "the man who can".Broker memory usage. Broker-side retry system may require a lot of memory and
disk space at RabbitMQ, and it's whell-known that RabbitMQ performs much
better without any disk usage. Retries should not be used as "normal" event
handling flow.Related projectsCelery-message-consumerRobustness tactics is inspired bycelery-message-consumerproject which aims to handle events published to AMQP broker from
non-celery-based projects (maybe from other languages).
The main difference is thatCelery-AMQP-Eventsuses Celery tasks instead ofincludingadditionalconsumer stepnearby the celery worker.