Package: celery-message-consumer
Tool for using thebin/celeryworker to consume vanilla AMQP
messages (i.e. not Celery tasks)Whilewriting a simple consumer
scriptusing Kombu can be quite easy, the Celery worker provides many features
around process pools, queue/routing connections etc as well as being
known to run reliably over long term.It seems safer to re-use this battle-tested consumer than try to write
our own and have to learn from scratch all the ways that such a thing
can fail.Usagepipinstallcelery-message-consumerHandlersIn your code, you can define a message handler by decorating a python
function, in much the same way as you would a Celery task:fromevent_consumerimportmessage_handler@message_handler('my.routing.key')defprocess_message(body):# `body` has been deserialized for us by the Celery workerprint(body)@message_handler(['my.routing.key1','my.routing.key2'])defprocess_messages(body):# you can register handler for multiple routing keys@message_handler('my.routing.*')defprocess_all_messages(body):# or wildcard routing keys, if using a 'topic' exchangeLike a Celery task, the module it is defined in must actually get
imported at some point for the handler to be registered.A queue (in fact, three queues - see below) will be created to receive
messages matching the routing key.CeleryElsewhere in your code you will need to instantiate a Celery app and
apply our custom ‘ConsumerStep’ which hooks our message handlers into
the worker. If you are already using Celeryas Celeryin your project
then you probably want separate Celery apps for tasks and for the
message consumer.fromceleryimportCeleryfromevent_consumer.handlersimportAMQPRetryConsumerStepmain_app=Celery()consumer_app=Celery()consumer_app.steps['consumer'].add(AMQPRetryConsumerStep)You likely will want separate config for each app. SeeCelery docs.In the config for your message consumer app, add the modules containing
your decorated message handler functions toCELERY_IMPORTS, exactly
as you would for Celery tasks - this ensures they get imported and
registered when the worker starts up.Then from the command-line, run the Celery worker just like you usually
would, attaching to the consumer app:bin/celeryworker-Amyproject.mymodule:consumer_appConfigurationSettings are intended to be configured primarily via a python file, such
as your existing Djangosettings.pyor Celeryceleryconfig.py.
To bootstrap this, there are a couple of env vars to control how config
is loaded:EVENT_CONSUMER_APP_CONFIGshould be an import path to a python module, for example:EVENT_CONSUMER_APP_CONFIG=django.conf.settingsEVENT_CONSUMER_CONFIG_NAMESPACESets the prefix used for loading further config values from env and
config file. Defaults toEVENT_CONSUMER.See source ofevent_consumer/conf/for more details.Some useful config keys (all of which are prefixed withEVENT_CONSUMER_by default):SERIALIZERthis is the name of a Celery serializer name, e.g.'json'. The consumer will only accept messages serialized in this
format.QUEUE_NAME_PREFIXif using default queue name (routing-key) then
this prefix will be added to the queue name. If you supply a custom
queue name in the handler decorator the prefix will not be applied.MAX_RETRIESdefaults to4(i.e. 1 attempt + 4 retries = 5
strikes)BACKOFF_FUNCtakes a function(int)->floatwhich returns
the retry delay (in seconds) based on current retry counter for the
message.ARCHIVE_EXPIRYtime in milliseconds to keep messages in the
“archive” queue, after which the exchange will delete them. Defaults
to 24 days.USE_DJANGOset toTrueif your message handler uses the
Django db connection, so that the worker is able to cope with the
dreaded“current transaction is aborted”error and continue.EXCHANGESif you need your message handlers to connect their
queues to specific exchanges then you can provide a dict like:EXCHANGES={# a reference name for this config, used when attaching handlers'default':{'name':'data',# actual name of exchange in RabbitMQ'type':'topic',# an AMQP exchange type},'other':{...},...}The'default'config will be used… by default. You can attach
handler to a specific exchange when decorating:@message_handler('my.routing.key',exchange='other')defprocess_message(body):passQueue layoutWhile all of the broker, exchange and queue naming is configurable (see
source code) this project implements avery specific queue pattern.Briefly: for each routing key it listens to, the consumer sets upthreequeues and a ‘dead-letter exchange’ (DLX).The “main” message queueIf any unhandled exceptions occur, and we have retried less thansettings.MAX_RETRIES, the message will be put on the “retry”
queue with a TTL. After the TTL expires, the DLX will put the message
back on the main queue.If all retries are exhausted (orPermanentFailureis raised) then
the consumer will put the message on the “archive” queue. This gives
opportunity for someone to manually retry the archived messages,
perhaps after a code fix has been deployed.You will of course note that this istotally different and separatefrom Celery’s owntask.retrymechanism.Pros:matches pattern we were already using for non-Celery,
non-Python apps, “archive” queue provides an extra safety net.Cons:Relies on RabbitMQ-specific feature, more queues (more
complicated).CompatibilityPython 2.7 and 3.6-3.8 are both supported.OnlyRabbitMQ transport is supported.We depend on Celery and Kombu. Their versioning seems to be loosely in
step so that Celery 3.x goes with Kombu 3.x and Celery 4.x goes with
Kombu 4.x. We test against both v3 and v4.Django is not required, but when used we have some extra integration
which is needed if your event handlers use the Django db connection.
This must be enabled if required via thesettings.USE_DJANGOflag.This project is tested against:xDjango 1.11Django 2.2Celery/Kombu 3.xCelery/Kombu 4.xPython 2.7Python 3.6Python 3.7Python 3.8Running the testsCircleCIThe easiest way to test the full version matrix is to install the
CircleCI command line app:https://circleci.com/docs/2.0/local-jobs/(requires Docker)The cli does not support ‘workflows’ at the moment so you have to run
the two Python version jobs separately:circlecibuild--jobpython-2.7circlecibuild--jobpython-3.6py.test (single combination of dependency versions)It’s also possible to run the tests locally, allowing for debugging of
errors that occur.We rely on some RabbitMQ features for our retry queues so we need a
rabbit instance to test against. Adocker-compose.ymlfile is
provided.docker-composeup-dexportBROKER_HOST=$(docker-machineipdefault)(adjust the last line to suit your local Docker installation)Therabbitmqadminweb UI is available to aid in debugging queue issues:http://{BROKER_HOST}:15672/Now decide which version combination from the matrix you’re going to
test and set up your virtualenv accordingly:pyenvvirtualenv3.6.2celery-message-consumerYou will need to editrequirements.txtandrequirements-test.txtfor the specific versions of dependencies you want to test against. Then
you can install everything via:pipinstall-rrequirements-test.txtSet an env to point to the target Django version’s settings in the test
app (for Django-dependent tests) and for general app settings:exportDJANGO_SETTINGS_MODULE=test_app.dj111.settingsexportEVENT_CONSUMER_APP_CONFIG=test_app.settingsNow we can run the tests:PYTHONPATH=.py.test-v-s--pdbtests/tox (all version combinations for current Python)You’ll notice in the CircleCI config we run tests against the matrix
dependency versions usingtox.There aresome wartsaround usingtoxwithpyenv-virtualenvso if you created a Python 3.6
virtualenv using the instructions above the best thing to do is delete it and
recreate it like this:pyenvvirtualenv-ppython3.6myenvpipinstalltox(it’s actually easier not to use a virtualenv at all - tox creates its
own virtualenvs anyway, but that does mean you’d have to install tox
globally)You need the Docker container running:docker-composeup-dexportBROKER_HOST=$(docker-machineipdefault)You can now run tests for any versions compatible with your virtualenv
python version, e.g.tox-epy36-dj111-cel4To run the full version matrix you need to have both Python 2.7 and 3.6. The
easiest way is viapyenv. You will also need to make both Python versions
‘global’ (or ‘local’) via pyenv, and then install and runtoxoutside of
any virtualenv.sourcedeactivatepyenvglobal2.7.143.6.2pipinstalltoxtox