Package: narq
Introductionnarq is a distributed task queue with asyncio and redis, which is built uponReArq(itself a rewrite ofarq)MotivationsThis project is an independent fork of ReArq because it is fundamentally different in its goals. Narq is intended as a
simple to reason about production-grade task queue.FeaturesAsyncIO support, easy integration withFastAPI.Delayed tasks, cron tasks and async task support.Full-featured built-in web interface.Built-in distributed task lock to ensure a given task is ran one at a time.Other powerful features to be discovered.Web InterfaceRequirementsRedis >= 5.0Quick StartTask Definition# main.pyfromnarqimportNarqnarq=Narq(db_url='mysql://root:123456@127.0.0.1:3306/narq')@narq.on_shutdownasyncdefon_shutdown():# you can do some clean up work here like close db and so on...print("shutdown")@narq.on_startupasyncdefon_startup():# you can do some initialization work hereprint("startup")@narq.task(queue="q1")asyncdefadd(self,a,b):returna+b@narq.task(cron="*/5 * * * * * *")# run task per 5 secondsasyncdeftimer(self):return"timer"Run narq worker>narqmain:narqworker-qq1-qq2# consume tasks from q1 and q2 as the same time2021-03-29 09:54:50.464 | INFO     | narq.worker:_main:95 - Started worker successfully on queue: narq:queue:default
2021-03-29 09:54:50.465 | INFO     | narq.worker:_main:96 - Registered tasks: add, sleep, timer_add
2021-03-29 09:54:50.465 | INFO     | narq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.43M clients_connected=5 db_keys=6Run narq timerIf you have timing task or delay task, you should run another command also:>narqmain:narqtimer2021-03-29 09:54:43.878 | INFO     | narq.worker:_main:275 - Start timer successfully
2021-03-29 09:54:43.887 | INFO     | narq.worker:_main:277 - Registered timer tasks: timer_add
2021-03-29 09:54:43.894 | INFO     | narq.worker:log_redis_info:86 - redis_version=6.2.1 mem_usage=1.25M clients_connected=2 db_keys=6Also, you can run timer with worker together bynarq main:narq worker -t.Integration with FastAPIfromfastapiimportFastAPIapp=FastAPI()@app.on_event("shutdown")asyncdefshutdown()->None:awaitnarq.close()# then run task in view@app.get("/test")asyncdeftest():job=awaitadd.delay(args=(1,2))# orjob=awaitadd.delay(kwargs={"a":1,"b":2})# orjob=awaitadd.delay(1,2)# orjob=awaitadd.delay(a=1,b=2)result=awaitjob.result(timeout=5)# wait result for 5 secondsprint(result.result)returnresultStart web interface>narqmain:narqserver
Usage:narqserver[OPTIONS]Startrestapiserver.

Options:--hostTEXTListenhost.[default:0.0.0.0]-p,--portINTEGERListenport.[default:8000]-h,--helpShowthismessageandexit..After starting the server, checkhttps://127.0.0.1:8000/docsto see all endpoints andhttps://127.0.0.1:8000to use the web interface.Other options will be passed intouvicorndirectly, such as--root-pathetc.narqmain:narqserver--host0.0.0.0--root-path/narqMount as FastAPI sub appIf you have an existing FastAPI service, to simplify your deployment you might want to mount the narq server as a FastAPI sub app.fromfastapiimportFastAPIfromexamples.tasksimportnarqfromnarq.server.appimportappasnarqapp=FastAPI()app.mount("/narq",narq_app)narq_app.set_narq(narq)Start worker inside appYou can also start worker inside your app.@app.on_event("startup")asyncdefstartup():awaitnarq.init()awaitnarq.start_worker(with_timer=True,block=False)ThanksToarq, Fast job queuing and RPC in python with asyncio and redis.ReArq, Improved arq rewrite with an API + web interfaceLicenseThis project is licensed under theApache-2.0License.