Package: oxalis
OxalisDistributed async task/job queue, like Celery forasyncioworldFeatureRedis and AMQP(RabbitMQ etc.) supportTask timeout and concurrency limit supportDelayed task(Both Redis and RabbitMQ) supportCron task/job beaterBuilt-in coroutine pool with concurrency and time limitInstallpip install oxalisExample with Redis backendDefine task:fromredis.asyncio.clientimportRedisfromoxalis.redisimportOxalisoxalis=Oxalis(Redis(host=os.getenv("REDIS_HOST","redis")))@oxalis.register()asyncdefhello_task():print("Hello oxalis")Run worker(consumer):oxalis.run_worker_master()pythonex.py
INFO:oxalis:RegisteredTask:<Task(hello_task)>
INFO:oxalis:Runworker:<Oxalis(pid-101547)>...
INFO:oxalis:Runworker:<Oxalis(pid-101548)>...
INFO:oxalis:Runworker:<Oxalis(pid-101549)>...
INFO:oxalis:Runworker:<Oxalis(pid-101550)>...
INFO:oxalis:Runworker:<Oxalis(pid-101551)>...
INFO:oxalis:Runworker:<Oxalis(pid-101552)>...
INFO:oxalis:Runworker:<Oxalis(pid-101554)>...Run client(producer):importasyncioasyncio.get_event_loop().run_until_complete(oxalis.connect())foriinrange(10):asyncio.get_event_loop().run_until_complete(hello_task.delay())asyncio.get_event_loop().run_until_complete(hello_task.delay(_delay=1))# delay execution after 1sRun cron beater:fromoxalis.beaterimportBeaterbeater=Beater(oxalis)beater.register("*/1 * * * *",hello_task)beater.run()pythonexb.pyINFO:oxalis:Beattask:<Task(hello_task)>at<*/1****>...TaskCodecTheTaskCodecwill encode/decode task args, default codec will usejsonCustom task codec:fromoxalis.baseimportTaskCodecclassMyTaskCodec(TaskCodec):@classmethoddefencode(cls,task:Task,task_args:tp.Sequence[tp.Any],task_kwargs:tp.Dict[str,tp.Any],)->bytes:...@classmethoddefdecode(cls,content:bytes)->TaskCodec.MESSAGE_TYPE:...oxalis=Oxalis(Redis(host=os.getenv("REDIS_HOST","redis")),task_codec=MyTaskCodec())...Task poolOxalis use one coroutine pool with concurrency limit and timeout limit to run all taskCustom pool:fromredis.asyncio.clientimportRedisfromoxalis.redisimportOxalisfromoxalis.poolimportPooloxalis=Oxalis(Redis(host=os.getenv("REDIS_HOST","redis")),pool=Pool(concurrency=10,timeout=60))For Redis task, thequeuewill be blocked utilpoolis not fully loadedFor AMQP task, oxalis use AMQP's QOS to limit worker concurrency(pool's concurrency will be -1 which means the pool's concurrency will not be limited)asyncio.TimeoutErrorwill be raised if one task is timeoutEvery worker process has owned limited poolSpecified one task timeout limit:@oxalis.register(queue=custom_queue,timeout=10)defcustom_task():print("Hello oxalis")Custom hookOxalis defined some hook API for inherited subclass:classMyOxalis(Oxalis):defon_worker_init():# will be called before worker startedpassdefon_worker_close():# will be called after worker startedpassSome API can be rewritten or inherited for custom usage, eg:importsentry_sdkclassMyOxalis(Oxalis):asyncdefexec_task(self,task:Task,*task_args,**task_kwargs):"""capture exception to sentry"""try:awaitsuper().exec_task(task,*task_args,**task_kwargs)exceptExceptionase:sentry_sdk.capture_exception(e)Redis Backend DetailOxalis use redis'slistandpubsubstructure as a message queueQueueCustom queue:fromoxalis.redisimportQueue,PubsubQueuecustom_queue=Queue("custom")bus_queue=PubsubQueue("bus")Register task:@oxalis.register(queue=custom_queue)defcustom_task():print("Hello oxalis")@oxalis.register(queue=bus_queue)defbus_task():print("Hello oxalis")For task producer, the task will send to specified queue when calltask.delay()For task consumer, oxalis will listen those queues and receive task from themConcurrency limitOxalis using coroutine pool's concurrency limit way, we can set different concurrency limit with specified pool for one task:@oxalis.register(pool=Pool(concurrency=1))defcustom_task():print("Hello oxalis")Delayed taskSupport by rediszsetAMQP Backend DetailCustom Queue and ExchangeOxalis using AMQP's way to define Exchange, Queue and their bindingsimportasyncioimportloggingimporttimefromaio_pikaimportRobustConnectionfromoxalis.amqpimportExchange,ExchangeType,Oxalis,Pool,Queuee=Exchange("test")q=Queue("test",durable=False)e2=Exchange("testfanout",type=ExchangeType.FANOUT)q2=Queue("testfanout",durable=False)oxalis=Oxalis(RobustConnection("amqp://root:letmein@rabbitmq:5672/"))oxalis.register_binding(q,e,"test")oxalis.register_binding(q2,e2,"")oxalis.register_queues([q,q2])@oxalis.register(exchange=e,routing_key="test")asyncdeftask1():awaitasyncio.sleep(1)print("hello oxalis")@oxalis.register(exchange=e2)asyncdeftask2():awaitasyncio.sleep(10)print("hello oxalis")For producer, taskoxalis.registerdefined one task message will send to which exchange(by routing key)For consumer,register_queuesdefined which queues oxalis will listenedTask routing defined by bindingsConcurrency limitOxalis use AMQP's QOS to limit worker concurrency(Task'sack_latershould be true), so coroutine pool's concurrency should not be limited.Custom queue QOS:oxalis=Oxalis(RobustConnection("amqp://root:letmein@rabbitmq:5672/"),default_queue=Queue("custom",consumer_prefetch_count=10))...fanout_queue=Queue("testfanout",durable=False,consumer_prefetch_count=3)oxalis.register_queues([fanout_queue])...Custom task behaviorDefine task how to performackandreject# always ack even task failed(raise exception)@oxalis.register(ack_always=True,reject=False)asyncdeftask2():awaitasyncio.sleep(10)print("hello oxalis")#  reject with requeue when task failed@oxalis.register(reject_requeue=True)asyncdeftask2():awaitasyncio.sleep(10)print("hello oxalis")Delayed taskSupport by RabbitMq'splugin