Package: celery-pool-asyncio
Celery Pool AsyncIOFree software: Apache Software License 2.0Featuresimport asyncio
from celery import Celery

# celery_pool_asyncio importing is optional
# It imports when you run worker or beat if you define pool or scheduler
# but it does not imports when you open REPL or when you run web application.
# If you want to apply monkey patches anyway to make identical environment
# when you use REPL or run web application app it's good idea to import
# celery_pool_asyncio module
import celery_pool_asyncio  # noqa
# Sometimes noqa does not disable linter (Spyder IDE)
celery_pool_asyncio.__package__


app = Celery()


@app.task(
    bind=True,
    soft_time_limit=42,  # raises celery.exceptions.SoftTimeLimitExceeded inside the coroutine
    time_limit=300,  # breaks coroutine execution
)
async def my_task(self, *args, **kwargs):
    await asyncio.sleep(5)


@app.task
async def my_simple_task(*args, **kwargs):
    await asyncio.sleep(5)Then run celery:$ celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPoolMonkey patching: wtf and whyThere are many monkey patches should be applied to make application working, and
some of them should be applied as early as possible. You are able to switch off
any of them by setting environment variableCPA_MONKEY_DENY. Remember you
should have a great reason to do it.Except critical for work features it allows:# await data sending to broker
async_result = await my_simple_task.delay()

# await wainting for AsyncResult
result = await async_result.get()You can manually disable any of them by enumerating it comma separated:$ env CPA_MONKEY_DENY=CELERY.SEND_TASK,ALL_BACKENDS celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPoolDisabling is available for:SIGNAL.SENDCELERY.SEND_TASKWORKCONTROLLER.USE_EVENTLOOPWORKER.CPU_COUNTBASERESULTCONSUMER.WAIT_FOR_PENDINGBASERESULTCONSUMER.DRAIN_EVENTS_UNTILASYNCBACKENDMIXIN.WAIT_FOR_PENDINGALL_BACKENDSBEAT.SERVICE.STARTBEAT.SERVICE.STOPBUILD_TRACERKOMBU.UTILS.COMPATRPC.RESULTCONSUMER.DRAIN_EVENTSAMQPBACKEND.DRAIN_EVENTSAMQPBACKEND.GET_MANYAMQP_BACKENDRPC_BACKENDSchedulingDefault scheduler doesn't work.PersistentScheduleris subclass of default
celery scheduler.Running celery with scheduler:$ celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPool --scheduler celery_pool_asyncio:PersistentScheduler
$ celery beat -A hello_async_celery.app --scheduler celery_pool_asyncio:PersistentSchedulerEmbeding also supported:$ celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPool --scheduler celery_pool_asyncio:PersistentScheduler -BWARNING: embeded scheduler startup is not stable. It starts correctly in ~50%
of cases. It looks like race condition. But after correct startup it works well.
That's why it's good idea to run scheduler in separated process.Celery Signalsfrom celery.signals import worker_init, worker_shutting_down


@worker_init.connect
async def do_startup_async(sender, **kwargs):
    # Coroutine functions are available after pool initialized
    await MyClass.init_async()


@worker_init.connect
def do_startup(sender, **kwargs):
    # regular functions are available too
    pass


@worker_shutting_down.connect
async def do_shutdown(sender=None, **kwargs):
    await MyClass.shutdown()More examplesThere is an example project usescelery-pool-asyncio.Changelog[0.2.0]Allow to decorate corofunctions bycelery.signalsImplement pool capacity (-c, --concurency)[0.1.12]Finalize monkey patcher refactoring. Now you able to switch off applying of
any monkey patch. Remember with great power comes great responsibilityImplementsoft_time_limitImplementrevokeFix keywords[0.1.11]Total monkey patching refactor. Now it is enabled by default, but you can
manually disable some of features using environment variableCPA_MONKEY_DENY[0.1.10]Make Celery Beat workingAdd async Celery SchedulerMore monkey patchingMove loop and loop_runner to own moduleAvoid creating multiple loops and loop_runners per application[0.1.9]Large rework ofawait AsyncResult.get()Works much better than earlier, but it's crap stillAdded outnumber of monkey-patchesFixed race condition on first microseconds of pool shutdown[0.1.8]Cleanup tracer, use celery.app.trace namespase where it possible[0.1.7]Refactor monkey, split itMovepatch_send_taskto own functionAddpatch_result_gettoawait AsyncResult.get[0.1.6]Avoid building trace twiceAlso this small performance optimization fixedAsyncResult.get[0.1.5]Fix graceful shutdown[0.1.4]Fix monkey: another function must be patched[0.1.3]Add changelogAppend documentation[0.1.2]Add monkey patcher to make brocker IO operations nonblocking[0.1.1]Refactor codeFix found errors[0.1.0]Initial commit