Package: aio-celery
aio-celeryWhat is aio-celery?This project is an alternative independent asyncio implementation ofCelery.Quoting Celerydocumentation:Celery is written in Python, but the protocol can be implemented in any language.And aio-celery does exactly this, it (re)implementsCelery Message Protocol(in Python) in order to unlock access to asyncio tasks and workers.The most notable feature of aio-celery is that it does not depend on Celery codebase.
It is written completely from scratch as a thin wrapper aroundaio-pika(which is an asyncronous RabbitMQ python driver)
and it has no other dependencies (except forredis-pyfor result backend support, but this dependency is optional).There have been attempts to create asyncio Celery Pools before, andcelery-pool-asynciois one such example, but its implementation, due to convoluted structure
of the original Celery codebase, is (by necessity) full of monkeypatching and other
fragile techniques. This fragility was apparently thereasonwhy this library became incompatible with Celery version 5.Celery project itself clearlystruggleswith implementing Asyncio Coroutine support,
constantly delaying this feature due to apparent architectural difficulties.This project was created in an attempt to solve the same problem but using the opposite approach.
It implements only a limited (but still usable — that is the whole point) subset of Celery functionality
without relying on Celery code at all — the goal is to mimic the basic
wire protocol and to support a subset of Celery API minimally required for running and manipulating
tasks.FeaturesWhat is supported:Basic tasks API:@app.taskdecorator,delayandapply_asynctask methods,AsyncResultclass etc.Everything is asyncio-friendly and awaitableAsyncronous Celery worker that is started from the command lineRouting and publishing options such ascountdown,eta,queue,priority, etc.Task retriesOnly RabbitMQ as a message brokerOnly Redis as a result backendImportant design decisions for aio-celery:Complete feature parity with upstream Celery project is not the goalThe parts that are implemented mimic original Celery API as close as possible, down to
class and attribute namesThe codebase of this project is kept as simple and as concise, it strives to be easy to understand and reason aboutThe codebase is maintained to be as small as possible – the less code, the fewer bugsExternal dependencies are kept to a minimum for the same purposeThis project must not at any point have celery as its external dependencyInstallationInstall usingpip:pipinstallaio-celeryIf you intend to use Redis result backend for storing task results, run this command:pipinstallaio-celery[redis]UsageDefineCeleryapplication instance and register a task:# hello.pyimportasynciofromaio_celeryimportCeleryapp=Celery()@app.task(name="add-two-numbers")asyncdefadd(a,b):awaitasyncio.sleep(5)returna+bThen run worker:$aio_celeryworkerhello:appQueue some tasks:# publish.pyimportasynciofromhelloimportadd,appasyncdefpublish():asyncwithapp.setup():tasks=[add.delay(n,n)forninrange(50000)]awaitasyncio.gather(*tasks)asyncio.run(publish())$python3publish.pyThe last script concurrently publishes 50000 messages to RabbitMQ. It takes about 8 seconds to finish,
with gives average publishing rate of about 6000 messages per second.Using Redis Result Backendimportasynciofromaio_celeryimportCeleryfromaio_celery.exceptionsimportTimeoutErrorapp=Celery()app.conf.update(result_backend="redis://localhost:6379",)@app.task(name="do-something")asyncdeffoo(x,y,z):awaitasyncio.sleep(5)returnx+y-zasyncdefmain():asyncwithapp.setup():result=awaitfoo.delay(1,2,3)try:value=awaitresult.get(timeout=10)exceptTimeoutError:print("Result is not ready after 10 seconds")else:print("Result is",value)if__name__=="__main__":asyncio.run(main())Adding contextimportcontextlibimportasyncpgfromaio_celeryimportCeleryapp=Celery()@app.define_app_context@contextlib.asynccontextmanagerasyncdefsetup_context():asyncwithasyncpg.create_pool("postgresql://localhost:5432",max_size=10)aspool:yield{"pool":pool}@app.taskasyncdefget_postgres_version():asyncwithapp.context["pool"].acquire()asconn:version=awaitconn.fetchval("SELECT version()")returnversionRetriesimportrandomfromaio_celeryimportCeleryapp=Celery()@app.task(name="add-two-numbers",bind=True,max_retries=3)asyncdefadd(self,a,b):ifrandom.random()>0.25:# Sends task to queue and raises `aio_celery.exception.Retry` exception.awaitself.retry(countdown=2)Priorities and QueuesSupport forRabbitMQ Message Priorities:importasynciofromaio_celeryimportCeleryapp=Celery()app.conf.update(task_default_priority=5,# global default for all taskstask_default_queue="queue-a",# global default for all taskstask_queue_max_priority=10,# sets `x-max-priority` argument for RabbitMQ Queue)@app.task(name="add-two-numbers",priority=6,# per task default (overrides global default)queue="queue-b",# per task default (overrider global default))asyncdefadd(a,b):awaitasyncio.sleep(3)returna+basyncdefmain():asyncwithapp.setup():awaitadd.apply_async(args=(2,3),priority=7,# overrides all defaultsqueue="queue-c",# overrides all defaults)if__name__=="__main__":asyncio.run(main())See alsoRabbitMQ documentationon priorities.Send unregistered task by nameimportasynciofromaio_celeryimportCeleryapp=Celery()app.conf.update(result_backend="redis://localhost:6379",)asyncdefmain():asyncwithapp.setup():result=awaitapp.send_task("add-two-numbers",args=(3,4),queue="high-priority",countdown=30,)print(awaitresult.get(timeout=5))if__name__=="__main__":asyncio.run(main())Register tasks using@shared_taskdecoratorAnalogous to original Celeryfeature,
the@shared_taskdecorator lets you create tasks without having any concrete app instance:fromaio_celeryimportCelery,shared_task@shared_taskasyncdefadd(a,b):returna+bapp=Celery()# `add` task is already registered on `app` instanceReferencesSimilar Projectshttps://github.com/cr0hn/aiotaskshttps://github.com/the-wondersmith/celery-aio-poolhttps://github.com/kai3341/celery-pool-asyncioInspirationhttps://github.com/taskiq-python/taskiqRelevant Discussionshttps://github.com/celery/celery/issues/3884https://github.com/celery/celery/issues/7874https://github.com/anomaly/lab-python-server/issues/21https://github.com/anomaly/lab-python-server/issues/32