Package: beanie_batteries_queue
Task QueueTask Queue is an advanced queue system for Beanie (MongoDB), designed to efficiently manage and process tasks. It features task priorities, states, dependencies, and automatic expiration. Different task queues can be processed together using the Worker class. Multiple workers can be run in separate processes using the Runner class.Installationpipinstallbeanie[queue]Examplefrombeanie_batteries_queueimportTask,RunnerclassExampleTask(Task):data:strasyncdefrun(self):self.data=self.data.upper()awaitself.save()runner=Runner(task_classes=[ExampleTask])runner.start()TaskDeclare a task classfrombeanie_batteries_queueimportTaskclassSimpleTask(Task):s:strProcess a taskfrombeanie_batteries_queueimportState# Producertask=SimpleTask(s="test")awaittask.push()# ConsumerasyncfortaskinSimpleTask.queue():asserttask.s=="test"# Do some workawaittask.finish()break# Check that the task is finishedtask=awaitSimpleTask.find_one({"s":"test"})asserttask.state==State.FINISHEDAsync generatorSimpleTask.queue()will return all unfinished tasks in the order they were created or based on the
priority if it was specified. It is an infinite loop, so you can usebreakto stop it.You can also useSimpleTask.pop()to get the next task from the queue.frombeanie_batteries_queueimportState# Producertask=SimpleTask(s="test")awaittask.push()# Consumertask=awaitSimpleTask.pop()asserttask.s=="test"# Do some workawaittask.finish()Task priorityThere are three priority levels:LOW,MEDIUM, andHIGH. The default priority isMEDIUM.
Tasks are popped from the queue in the following order:HIGH,MEDIUM,LOW.frombeanie_batteries_queueimportPrioritytask1=SimpleTask(s="test1",priority=Priority.LOW)awaittask1.push()task2=SimpleTask(s="test2",priority=Priority.HIGH)awaittask2.push()asyncfortaskinSimpleTask.queue():asserttask.s=="test2"awaittask.finish()breakTask stateThere are four states:CREATED,RUNNING,FINISHED, andFAILED. The default state isPENDING.
When a task is pushed, it is in theCREATEDstate. When it gets popped from the queue, it is in theRUNNINGstate.FINISHEDandFAILEDstates should be set manually.Finished:frombeanie_batteries_queueimportStatetask=SimpleTask(s="test")awaittask.push()asyncfortaskinSimpleTask.queue():asserttask.state==State.RUNNINGawaittask.finish()breaktask=awaitSimpleTask.find_one({"s":"test"})asserttask.state==State.FINISHEDFailed:frombeanie_batteries_queueimportStatetask=SimpleTask(s="test")awaittask.push()asyncfortaskinSimpleTask.queue():asserttask.state==State.RUNNINGawaittask.fail()breaktask=awaitSimpleTask.find_one({"s":"test"})asserttask.state==State.FAILEDTask dependenciesYou can specify that a task depends on another task. In this case, the task will be popped from the queue only when all
its dependencies have finished.frombeanie_batteries_queueimportTask,DependencyTypefrombeanie_batteries_queueimportLinkfrompydanticimportFieldclassSimpleTask(Task):s:strclassTaskWithDirectDependency(Task):s:strdirect_dependency:Link[SimpleTask]=Field(dependency_type=DependencyType.DIRECT)frombeanie_batteries_queueimportStatetask1=SimpleTask(s="test1")awaittask1.push()task2=TaskWithDirectDependency(s="test2",direct_dependency=task1)awaittask2.push()task_from_queue=awaitTaskWithDirectDependency.pop()asserttask_from_queueisNone# task2 is not popped from the queue because task1 is not finished yetawaittask1.finish()task_from_queue=awaitTaskWithDirectDependency.pop()asserttask_from_queueisnotNone# task2 is popped from the queue because task1 is finishedTask dependencies with multiple linksYou can specify that a task depends on multiple tasks. In this case, the task will be popped from the queue when all or
any its dependencies are finished. It is controlled by thedependency_typeparameter.AllclassTaskWithMultipleDependencies(Task):s:strlist_of_dependencies:Link[SimpleTask]=Field(dependency_type=DependencyType.ALL_OF)AnyclassTaskWithMultipleDependencies(Task):s:strlist_of_dependencies:Link[SimpleTask]=Field(dependency_type=DependencyType.ANY_OF)Tasks can have multiple links with different dependency types.classTaskWithMultipleDependencies(Task):s:strlist_of_dependencies_all:Link[SimpleTask]=Field(dependency_type=DependencyType.ALL_OF)list_of_dependencies_any:Link[SimpleTask]=Field(dependency_type=DependencyType.ANY_OF)direct_dependency:Link[SimpleTask]=Field(dependency_type=DependencyType.DIRECT)Expire timeYou can specify the time after which the task will be removed from the queue, even if it is not finished or has failed.
This is controlled by theexpireAfterSecondsindex, which is set to 24 hours by default.frompymongoimportASCENDINGfrombeanie_batteries_queueimportTaskclassTaskWithExpireTime(Task):s:strclassSettings:indexes=[# Other indexes,# Expire after 5 minutes[("created_at",ASCENDING),("expireAfterSeconds",300)],]Finished or failed tasks are not immediately removed from the queue. They are removed after the expiration time. You can
manually delete them using thedelete()method.QueueQueues are designed to manage tasks. It will handle all the logic of creating, updating, and deleting tasks. Task logic
should be defined in therunmethod of the taskfrombeanie_batteries_queueimportTaskclassProcessTask(Task):data:strasyncdefrun(self):# Implement the logic for processing the taskprint(f"Processing task with data:{self.data}")self.data=self.data.upper()awaitself.save()Now we can start the queue and it will process all the tasks. Be aware - it will run infinite loop. If you want to have
another logic after starting the queue, you should run it withasyncio.create_task().queue=ProcessTask.queue()awaitqueue.start()Stop the queueYou can stop the queue by calling thestop()method.awaitqueue.stop()Queue settingsYou can specify how frequently the queue will check for new tasks. The default value is 1 second.queue=ProcessTask.queue(sleep_time=60)# 60 secondsawaitqueue.start()WorkerQueue can handle only one task model. To process multiple task models, you should use Worker. It will run multiple queuesfrombeanie_batteries_queueimportTask,WorkerclassProcessTask(Task):data:strasyncdefrun(self):self.data=self.data.upper()awaitself.save()classAnotherTask(Task):data:strasyncdefrun(self):self.data=self.data.upper()awaitself.save()worker=Worker(task_classes=[ProcessTask,AnotherTask])awaitworker.start()Be aware - it will run infinite loop. If you want to have another logic after starting the worker, you should run it withasyncio.create_task().Stop the workerYou can stop the worker by calling thestop()method.awaitworker.stop()Worker settingsYou can specify how frequently the worker will check for new tasks. The default value is 1 second.worker=Worker(task_classes=[ProcessTask,AnotherTask],sleep_time=60)# 60 secondsawaitworker.start()RunnerRunner is a class that allows you to run multiple workers in separate processes. It is useful when your tasks are CPU intensive and you want to use all the cores of your CPU.frombeanie_batteries_queueimportTask,RunnerclassProcessTask(Task):data:strasyncdefrun(self):self.data=self.data.upper()awaitself.save()classAnotherTask(Task):data:strasyncdefrun(self):self.data=self.data.upper()awaitself.save()runner=Runner(task_classes=[ProcessTask,AnotherTask])runner.start()Stop the runnerYou can stop the runner by calling thestop()method.runner.stop()Runner settingsYou can specify how many workers will be run. The default value is 1.runner=Runner(task_classes=[ProcessTask,AnotherTask],workers_count=4)runner.start()You can specify how frequently the worker will check for new tasks. The default value is 1 second.runner=Runner(task_classes=[ProcessTask,AnotherTask],sleep_time=60)# 60 secondsrunner.start()You can specify if the start method should run while the workers are alive or if it should return immediately. The default value is True.runner=Runner(task_classes=[ProcessTask,AnotherTask],run_indefinitely=False)runner.start()