Package: chihuo
chihuo: An Asynchronous Task Running Enginechihuo is an asynchronous task running engine. It runs asynchronous tasks concurrently
from a queue which can be at memory, Redis, Lodis or Some MQ.The LoopChihuoLoopis the asynchronous engine core which responses to connect the backend queue
and distributes every tasks to the global event loop.For a process, the chihuo create only one global event loop to run all asynchronous
tasks and the loop is running forever.ChihuoLoop.startmethodWhen aChihuoLoopstarts, it needs to add some primary tasks to the backend queue.
Thestartfunction is the place to fire these priorities.ChihuoLoop.add_taskmethodAt anywhere and anytime, we can use theadd_taskmethod to add a task to backend queue.
The task discription must be the tuple as(task_id: str, task_info: jsonifiable object_)e.g.add_task(("task1", [1, 1+10]), ("task2", [2, 2+10]))ChihuoLoop.make_taskmethodEach classes which have implemented theChihuoLoopclass must to implement themake_taskmethod for receiving the task's information that is added byChihuoLoop.add_taskto
the backend queue.ChihuoLoop.task_finishmethodWhen a task is finished and is unneeded at future, we can tag the task as thefinishedstate
using thetask_finishmethod. e.g.task_finish(task_id). The task that is atfinishedstate can't be add to backend queue again.ChihuoLoop.task_unfinishmethodWhen a task is at thefinishedstate and we need to add it to queue again, we can use
thetask_unfinishmethod to set the task state tounfinish. Then, we can useChihuoLoop.add_taskto add the task to queue again.ChihuoLoop.stoppropertyWhen the propertystopto be set as True, theChihuoLoopwill stop pull the tasks from
queue and exit.Demo# filename: abc.pyfromchihuoimportChihuoLoopfromchihuo.commonimportSERVER_DEFAULT_CONFIG_PATHimportchihuoclassMyTasksA(ChihuoLoop):NAME="my-tasksA"# The unique id for backend queue serverCONCURRENCY=10# The number of tasks running concurrentlySERVER_CONFIG=SERVER_DEFAULT_CONFIG_PATHdef__init__(self):super().__init__()asyncdefstart(self):foridinrange(10):awaitself.add_task((str(i),{i**2}))asyncdefmake_task(self,task_id,task):print(f"my-task-A:{task_id}")print(f"power of{i}is{task}")# Talk chihuo that the task is finishedawaitself.task_finish(i)classMyTasksB(ChihuoLoop):NAME="my-tasksB"# The unique id for the loopCONCURRENCY=10# The number of tasks running concurrentlySERVER_CONFIG=SERVER_DEFAULT_CONFIG_PATHdef__init__(self):super().__init__()asyncdefstart(self):foridinrange(10):awaitself.add_task((str(i),{i**2}))asyncdefmake_task(self,task_id,task):print(f"my-task-B:{task_id}")print(f"power of{i}is{task}")# Talk chihuo that the task is finishedawaitself.task_finish(i)if__name__=='__main__':chihuo.run(MyTasksA,MyTasksB)To Run the project, firstly we need to launch a backend queue server.
Here, we use a Lodis instance as default server.Start a Lodis server:LODIS_DB_PATH=/data/test-lodis LODIS_IP_PORT="127.0.0.1:8311" nohup /path/to/lodis &Then, we write a configure file as./server.json(theSERVER_DEFAULT_CONFIG_PATH)
to connect the backend for ourMyTasksAandMyTasksB.backend=lodis
ip=127.0.0.1
port=8311Now, we can run the script.python3 abc.pyShutdown the TasksDefaultly, the global loop is running forever and we need to shutdown the loop by sending
aINT (interrupt)signal to the process. After the process received TheINTsignal, it will
set the propertystopof instance which is implemented fromChihuoLoopto True and the
instance will stop all running tasks and send them back to queue, finally exit.