Package: palpable
PalpableIntroductionPalpable is a producer-consumer type of task server that uses multiprocessing to do parallel computing. If you need a
small asynchronous task server and would not like to go through the complication of setting up rabbitmq and Celery,
Palpable is the choice.Palpable uses Clients to submit Tasks to a Server which manages several Workers. A Task is constructed given a Procedure
which defines what should be done in a task. Each Task has a unique task_id for querying.InstallpipinstallpalpableATTENTION Before Start UsingPalpable do not offer on disk persistencePalpable Server's data are all in memory. Once the server is shutdown, all data are lost.Palpable run each task in its own processThis ensures isolated environment for each task and multi-cpu usages. However, it adds overhead. Therefore,if each task
is easy, running these tasks will be way slower than directly run them in one process.Python'sprintmultiprocessingdeadlockDo not useprintin the code send to the Palpable server. Doing so may cause palpable Workers to deadlock without
notice. Palpable is trying to catch theprintstatement in the code sent to the server, and change it tolog.info,
but this is not guaranteed to work well. If you are not sure whetherprintwill be sent to your Palpable server, usepython -uto start your Server, or setPYTHONUNBUFFERED=1as an environment variable, when starting your Server.Basic UsageSetup Customized Server and Client# configurationsSERVER_ADDRESS=("127.0.0.1",8089)SERVER_FAMILY="AF_INET"SECRET=b"29r8in389rhd"NUM_OF_WORKERS:int=8TASK_TIMEOUT_SECONDS:float=3600*3RESULT_RETENTION_CAPACITY:int=100000RESULT_RETENTION_SECONDS:float=600frompalpable.servants.serverimportServerfrompalpable.units.clientimportClientimporttempfile,shutilclassExampleServer(Server):def__init__(self):super(ExampleServer,self).__init__(logging_folder=tempfile.mkdtemp(),address=SERVER_ADDRESS,family=SERVER_FAMILY,authkey=SECRET,num_workers=NUM_OF_WORKERS,task_timeout_seconds=TASK_TIMEOUT_SECONDS,result_retention_capacity=RESULT_RETENTION_CAPACITY,result_retention_seconds=RESULT_RETENTION_SECONDS,)defclose(self):super(ExampleServer,self).close()shutil.rmtree(self._logging_folder)classExampleClient(Client):def__init__(self):super(ExampleClient,self).__init__(address=SERVER_ADDRESS,family=SERVER_FAMILY,authkey=SECRET)Define a FunctionSuppose the following code is in the fileutils.pyfromtimeimportsleepdefsquare(x):sleep(1)returnx*xUse Palpable to Map a Function or Call a Functionfromtimeimportsleepfromutilsimportsquareif__name__=="__main__":withExampleServer()asserver:client=ExampleClient()result=client.map(square,range(1000))# map function `square` with parameters [0, 1, ..., 999]print(result)result=client.run(square,4)# run function `square` with parameter `4`print(result)NOICE: it is important to put thesquarefunction in a different module, otherwise imported function, likesleep,
may not be pickled correctly.In this example, thewithclause is used to start and stop the server. To do it withoutwith:server=ExampleServer()# start the serverserver.start()# signal the server to stopserver.stop()# wait for all processes and threads to endserver.join()# close the serverserver.close()Advanced UsageDefine a Customized ProcedureSubclass the Procedure class and implement the run method. The run method will be called by the Workers to do the job.classProcedure(Immutable):defrun(self,messenger):"""This method will be called by the Worker to execute in a process.Override this method.Use __init__ to set any params needed for this callThe messenger parameter is a Messenger instanceUse messenger.debug/info/warning/error to send logsUse messenger.submit_tasks to submit sub tasks to the serverUse messenger.query_results to query for results of the submitted sub tasksIf you call predefined functions in this method, to catch possible `print` in the function, do:predefined_function.__globals__["print"] = messenger.print  # inject messenger.print as printSee the RunFunction procedure as an exampleATTENTION: do not use multiprocessing in this method.:param messenger: Messenger:return: The data if the task is successful. The data will be constructed to a successfulTaskResult by the TaskWorker.:raise raise TaskFailed exception with the failed data if the task is unsuccessful. e.g.raise TaskFailed("ID not found"). The "ID not found" will be constructed to a failed TaskResult.Other exceptions will be caught by the Worker and be constructed to a unsuccessful TaskResult usingthe Exception instance as data"""raiseNotImplementedErrorIn the run method, you can submit more tasks or run procedures using the messenger. You can also submit blocking tasks
(that means you wait for the results of these tasks before moving on) in the run method. Palpable will handle and run
the blocking tasks to get the results, even when all the workers are blocked and waiting for results.Example:frompalpable.units.taskimportTaskfrompalpable.procedures.procedureimportProceduredefdouble(x):print(f"processing{x}")return2*xclassCheckIfOdd(Procedure):def__init__(self,nums):"""Check if all the nums are odd numbers"""self.nums=numsdefrun(self,messenger):forninself.nums:ifn%2==0:returnFalsereturnTrueclassDoubleOddNumberProc(Procedure):def__init__(self,nums):"""Check if the nums are all odd, if so double the value of the nums:param nums: odd numbers"""self.nums=numsdefrun(self,messenger):double.__globals__["print"]=messenger.print# inject messenger.print as printmessenger.info("check if the numbers are all odd numbers")# submit new CheckIfOdd procedure and wait for resultscheck_if_odd_task_result=messenger.run_procedure(CheckIfOdd(self.nums))ifnotcheck_if_odd_task_result:raiseException("Error: the given numbers are not all odd")res=[double(x)forxinself.nums]returnresRun the Customized Procedure with parametersif__name__=="__main__":withExampleServer()asserver:client=ExampleClient()# this task will succeedtask_result=client.run_procedure(DoubleOddNumberProc(range(1,10,2)))print(task_result.is_successful,task_result.data)# this task will failtask_result=client.run_procedure(DoubleOddNumberProc(range(2,10,2)))print(task_result.is_successful,task_result.data)More UsageCheck the source codes, test codes, and examples for more usagehttps://github.com/XiaoMutt/palpableMechanismHow it worksHere is how it works at a high level:you start a Palpable Server with n workersyou submit a Procedure through Client to the Server for the workers to finishthe Procedure will be wrapped into a Task with a unique task id, and the Task is put into the TaskQueueyou receive a task ID for future referenceif any worker is available, it will run the task and put the TaskResult into the ResultCacheyou can query the result cache at any time, using a task ID, to asking for the result of a Taskif there is no such Task with the task ID, None will be returnedif the task is still running, then you will get a TaskResult whose is_successful attribute is Noneif the task finished and the result is ready, you will get a TaskResult that has the following attributes:task_id: the task IDis_successful: a boolean to indicate whether the task is successfuldata: the result data if the task is successful, else the error messagefollowup_task_ids: the task may initiate new tasks. This is a list of task_ids initiated by this task to
followup. Attention: this TaskResult will then be removed from the ResultCache.ArchitectureThere are three classes that do the heavy lifting: Server, Manager, and Worker. Each of them has a main thread loop that
do some jobs:Server: the main thread listens incoming commands and starts a new short-lived thread to handle every received
commandsManager: the main thread periodically prunes the ResultCacheWorker: the main thread periodically checks for available Tasks to run, if so start the Task in a different Process
and monitors itThe three classes have the following ownership Server --> Manager --> Worker (TaskQueue & Lock / ResultCache & Lock)The Server owns Manager & its thread lock: the threads spawned by the Server use the lock to access the Manager
concurrentlyThe Manager owns several Workers, TaskQueue & its thread lock, ResultCache & its thread lock. The Manager shares the
TaskQueue & its thread lock and ResultCache & its thread lock with the Workers. Workers and the Manager communicate
through the TaskQueue and ResultCache