Package: sharded-queue
Sharded queueIntroductionImagine your job queue operates at very high rps and needs distribution over multiple workers. But you need to keep context-sensitive requests in same thread and manage thread request processing priority. In other words, sharded queue is a queue with sub-queues inside. Tasks are executed in FIFO order and you define how to route them correctly per handler basis.InstallationInstall using pippip install sharded-queueGetting startedThere are some roles that you need to understand:requesta simple message that should be delivered to a handlerhandlerrequest handler that performs the jobroutedefines internal queue that is used for request distributionthreada group of context-sensitive requestsprioritycan be used to sort requests inside the threadLet's start with a simple notification task that is shared by 3 threads and there are no priorities. Notice, that handler methods are written using batch approach to reduce io latency per each message.fromsharded_queueimportHandler,Queue,RouteclassNotifyRequest(NamedTuple):'''In this example we have simple notify request containing user identifierIn addition, the value is used to shard requests over worker threads'''user_id:intclassNotifyHandler(Handler):@classmethodasyncdefroute(cls,*requests:NotifyRequest)->list[Route]:'''Spread requests by 3 threads that can be concurrently processed'''return[Route(thread=request.user_id%3)forrequestinrequests]asyncdefperform(self,*requests:NotifyRequest)->None:'''Perform is called using configurable batch sizeThis allows you to reduce io per single request'''# users = await UserRepository.find_all([r.user_id for r in requests])# await mailer.send_all([construct_message(user) for user in users])defbatch_size(self)->Optional[int]:'''Optionally, you can ovveride global batch size per handler'''return128Usage exampleWhen a handler is described you can use queue and worker api to manage and process tasks. Let's describe runtime components:lockhelps worker bind to the queuequeueis used to register requestsstoragea database containing queue dataworkerperforms requests using handlerfromasyncioimportgatherfromnotificationsimportNotifyHandler,NotifyRequestfromsharded_queueimportQueue,Workerfromsharded_queue.driversimportRuntimeLock,RuntimeStorageasyncdefexample():'''let's register notification for first 9 users'''queue:Queue=Queue(RuntimeStorage())awaitqueue.register(NotifyHandler,*[NotifyRequest(n)forninrange(1,9)])'''now all requests are waiting for workers on 3 notify handler tubesthey were distributed using route handler methodfirst tube contains notify request for users 1, 4, 7second tube contains requests for 2, 5, 8 and other goes to third tube'''futures=[Worker(RuntimeLock(),queue).loop(3),Worker(RuntimeLock(),queue).loop(3),Worker(RuntimeLock(),queue).loop(3),]'''we've just run three coroutines that will process messagesworkers will bind to each thread and process all messages'''awaitgather(*futures)'''now all emails were send in 3 threads'''DriversThere are several implementations of components:RedisLockpersist locks in redis using setnx apiRedisStoragepersist msgs using lists and lrange/lpop/rpush apiRuntimeLockpersist locks in memory (process in-memory distribution)RuntimeStoragepersist msgs in dict (process in-memory distribution)Handler lifecycleAs you can notice, routing is made using static method, but perform is an instance method. When a worker start processing requests it can bootstrap and tear down the handler usingstartandstopmethodsclassParseEventRequest(NamedTuple):'''Event identifier should be enough to get it contents from storage'''event:intclassParseEventHandler(Handler):@classmethodasyncdefcreate(cls)->Self:'''define your own handler and dependency factory'''returncls()@classmethodasyncdefroute(cls,*requests:ParseEventRequest)->list[Route]:'''override default single thread tube'''return[Route(0,0)forrequestinrequests]asyncdefstart(self):'''run any code on worker is bind to the queue'''asyncdefperform(self,*requests:ParseEventRequest):'''the handler'''asyncdefhandle(self,*requests:ParseEventRequest)->None:'''process requests batch'''asyncdefstop(self):'''run any code when queue is empty and worker stops processing thread'''DeduplicationThere is an optional if_not_exists flag. If it is set, request will be registered only if not persent in a queueawaitqueue.register(SycBucket,Bucket(7756527),if_not_exists=True)awaitqueue.register(SycBucket,Bucket(7756527),if_not_exists=True)Deferred handlersYou can use built-in deferred request handler to defer callawaitqueue.register(Housekeep,Room(402),defer=5)# numeric means secondsawaitqueue.register(Housekeep,Room(324),defer=timedelta(minutes=15))Recurrent handlersYou can use built-in recurrent request handler for regular request registrationawaitqueue.register(BalanceCheck,Company('basis'),recurrent=timedelta(hours=1))PerformancePerformance dependends on many factors, we can only measure clean library overhead with in-memory storages. You can run performance on your hardware withpytest -s, with this option performance test will print result for different cases. Perfomance test on intel i5-4670K, Ubuntu 23.04 LTS using Python 3.11.4 gives us about200_000rps for batch request registration with sharding and about600_000requests for request handling in concurrent mode.Advanced queue configurationYou can configure sharded queue using envQUEUE_LOCK_PREFIX = 'lock_'Lock key prefixQUEUE_LOCK_TIMEOUT = 24 * 60 * 60Lock key ttlQUEUE_STORAGE_PREFIX = 'tube_'Default queue prefixQUEUE_WORKER_ACQUIRE_DELAY = 1Worker acquire delay in seconds on empty queuesQUEUE_WORKER_BATCH_SIZE = 128Worker batch processing sizeQUEUE_WORKER_DEFERRED_RETRY_DELAY = 1Deferred tasks retry delayQUEUE_WORKER_EMPTY_LIMIT = 16Worker empty queue attempt limit berfore queue rebindQUEUE_WORKER_EMPTY_PAUSE = 0.1Worker pause in seconds on empty queueQUEUE_WORKER_RECURRENT_CHECK_INTERVAL = 30Recurrent interval check in secondsQUEUE_WORKER_RECURRENT_TASKS_LIMIT = 1024Recurrent tasks limit countYou can change runtime settingsfromsharded_queueimportQueue,Workerfromsharded_queue.driversimportRuntimeLock,RuntimeStorageworker=Worker(RuntimeLock(),Queue(RuntimeStorage()))worker.lock.settings.timeout=5*60worker.settings.acquire_delay=5worker.settings.batch_size=64awaitworker.loop()