Package: taskkit
taskkitpypi:https://pypi.org/project/taskkit/Overviewtaskkitis a distributed task runner.How to use1. Implement TaskHandlerThis is the core part.importjsonfromtypingimportAnyfromtaskkitimportTaskHandler,Task,DiscardTaskclassHandler(TaskHandler):defhandle(self,task:Task):# Use `tagk.group` and `task.name` to determine how to handle the taskiftask.group=='...':iftask.name=='foo':# decode the data which encoded by `self.encode_data` if neededdata=json.loads(task.data)# do something with the `data`...# return result for the taskreturn...eliftask.name=='bar':# do somethingreturn...# you should raise DiscardTask if you want to discard the taskraiseDiscardTaskdefget_retry_interval(self,task:Task,exception:Exception)->float|None:# This method will be called if the handle method raises exceptions. You# should return how long time should be wait to retry the task in seconds# as float. If you don't want to retry the task, you can return None to# make the task fail or raise DiscardTask to discard the task.returntask.retry_countiftask.retry_count<10elseNonedefencode_data(self,group:str,task_name:str,data:Any)->bytes:# encode data of tasks for serializing itreturnjson.dumps(data).encode()defencode_result(self,task:Task,result:Any)->bytes:# encode the result of the taskreturnjson.dumps(result).encode()defdecode_result(self,task:Task,encoded:bytes)->Any:# decode the result of the taskreturnjson.loads(encoded)2. Make KitUse redis implYou can use redis backend like this:fromredis.clientimportRedisfromtaskkit.impl.redisimportmake_kitREDIS_HOST='...'REDIS_PORT='...'redis=Redis(host=REDIS_HOST,port=REDIS_PORT)kit=make_kit(redis,Handler())Use django implAdd'taskkit.contrib.django'toINSTALLED_APPSin the settingsRunpython manage.py migrateMake akitinstance like below:fromtaskkit.impl.djangoimportmake_kitkit=make_kit(Handler())3. Run workersGROUP_NAME='Any task group name'# it starts busy loopkit.start(# number of processesnum_processes=3,# number of worker threads per processnum_worker_threads_per_group={GROUP_NAME:3})# you can use `start_processes` to avoid busy loopkit.start_processes(num_processes=3,num_worker_threads_per_group={GROUP_NAME:3},daemon=True)4. Initiate taskfromdatetimeimporttimedeltafromtaskkitimportResultGetTimedOutresult=kit.initiate_task(GROUP_NAME,# task name'your task name',# task data which can be encoded by `Handler.encode_data`dict(some_data=1),# run the task after 10 or more seconds.due=datetime.now()+timedelta(seconds=10))try:value=result.get(timeout=10)exceptResultGetTimedOut:...Scheduled Tasksfromdatetimeimporttimezone,timedeltafromtaskkitimportScheduleEntry,ScheduleEntryDict,RegularSchedule,ScheduleEntriesCompatMapping# define entries# key is a name for scheduler# value is a list of instances of ScheduleEntry#       or a list of dicts conforming to ScheduleEntryDict## ScheduleEntryCompat: TypeAlias = ScheduleEntry | ScheduleEntryDict# ScheduleEntriesCompat: TypeAlias = Sequence[ScheduleEntryCompat]# ScheduleEntriesCompatMapping: TypeAlias = Mapping[str, ScheduleEntriesCompat]#schedule_entries:ScheduleEntriesCompatMapping={'scheduler_name':[# You can use ScheduleEntry instance as follows. Note that the data# MUST be encoded by the same algorithm as `Handler.encode_data`.ScheduleEntry(# A key which can identify the schedule in the listkey='...',# group namegroup=GROUP_NAME,# task namename='test2',# The data MUST BE encoded by the same algorithm as# `Handler.encode_data` so it would looks like:data=Handler.encode_data(GROUP_NAME,'test2','SOME DATA'),# It means that the scheduler will initiate the task twice# an hour at **:00:00 and **:30:00.schedule=RegularSchedule(seconds={0},minutes={0,30},),),# You can use dict form of schedule entry (recommended).# Note that in dict form, the data MUST NOT be encoded because `Kit`# takes care of encoding for convenience. Other properties are same# as ScheduleEntry. Also you can use ScheduleEntryDict for annotation.{'key':'...','group':GROUP_NAME,'name':'test3',# IT MUST NOT BE ENCODED'data':2,'schedule':RegularSchedule(seconds={0},minutes={30}),}],# You can have multiple schedulers'another_scheduler':[# other entries ...],}# pass the entries with kit.startkit.start(num_processes=3,num_worker_threads_per_group={GROUP_NAME:3},schedule_entries=schedule_entries,tzinfo=timezone(timedelta(hours=9),'JST'))