Package: jasyncq
jasyncqAsynchronous task queue using mysqlYou should knowDispatcher'sfetch_scheduled_tasksandfetch_pending_tasksmethod takes scheduled job and concurrently update their status asWORK IN PROGRESSin same transactionMost of tasks that queued in jasyncq would run inexactly oncebyfetch_scheduled_tasksBUT, some cases job disappeared because of worker shutdown while working. It could be restored byfetch_pending_tasks(that can check how long worker tolerateWIP-ed but notCompleted(deleted row))How to use1. Create aiomysql connection poolimportasyncioimportloggingimportaiomysqlloop=asyncio.get_event_loop()pool=awaitaiomysql.create_pool(host='127.0.0.1',port=3306,user='root',db='test',loop=loop,autocommit=False,)2. Generate topic (table) with initialize and inject repository to dispatcherfromjasyncq.dispatcher.tasksimportTasksDispatcherfromjasyncq.repository.tasksimportTaskRepositoryrepository=TaskRepository(pool=pool,topic_name='test_topic')awaitrepository.initialize()dispatcher=TasksDispatcher(repository=repository)3. Enjoy queuePublish tasksawaitdispatcher.apply_tasks(tasks=[...listofjasyncq.dispatcher.model.task.TaskIn...],)Consume tasksscheduled_tasks=awaitdispatcher.fetch_scheduled_tasks(queue_name='QUEUE_TEST',limit=10)pending_tasks=awaitdispatcher.fetch_pending_tasks(queue_name='QUEUE_TEST',limit=10,check_term_seconds=60,)tasks=[*pending_tasks,*scheduled_tasks]# ...RUN JOBS WITH tasks4. Complete taskstask_ids=[str(task.uuid)fortaskintasks]awaitdispatcher.complete_tasks(task_ids=task_ids)Other featuresApply tasks with dependencygenesis=TaskIn(task={},queue_name=queue_name)dependent=TaskIn(task={},queue_name=queue_name,depend_on=task.uuid)# 'dependent' task might fetched after 'genesis' task is completedawaitdispatcher.apply_tasks(tasks=[genesis,dependent])Apply delayed task(scheduled task)scheduled_at=time.time()+60task=TaskIn(task={},queue_name=queue_name,scheduled_at=scheduled_at)# 'task' task might fetched after 60 seconds from nowawaitdispatcher.apply_tasks(tasks=[task])Apply urgent task (priority)normal=TaskIn(task={},queue_name=queue_name)urgent=TaskIn(task={},queue_name=queue_name,is_urgent=True)# 'urgent' task might fetched earlier than 'normal' task if queue was already fulledawaitdispatcher.apply_tasks(tasks=[normal,urgent])Fetching with ignoring dependencyscheduled_tasks=awaitdispatcher.fetch_scheduled_tasks(queue_name='QUEUE_TEST',limit=10,ignore_dependency=True,)pending_tasks=awaitdispatcher.fetch_pending_tasks(queue_name='QUEUE_TEST',limit=10,check_term_seconds=60,ignore_dependency=True,)tasks=[*pending_tasks,*scheduled_tasks]# ...RUN JOBS WITH tasksExampleConsumer: /example/consumer.pyProducer: /example/producer.pyRun example scripts$ docker run --name test_db -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=true -d mysql:8.0.17
$ docker exec -it test_db bash -c 'mysql -u root -e "create database test;"'
$ python3 -m example.producer
$ python3 -m example.consumerBuild$ python3 setup.py sdist
$ python3 -m pip install ./dist/jasyncq-*Deploy$ twine upload ./dist/jasyncq-{version}.tar.gzTest$ docker run --name test_db -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=true -d mysql:8.0.17
$ docker exec -it test_db bash -c 'mysql -u root -e "create database test;"'
$ python3 -m pip install pytest==6.2.3
$ pytest