Package: taskproc
taskprocA Python library for building/executing/managing task pipelines that isMinimal yet sufficient for general use.Focused on the composability of tasks for building large/complex pipelines effortlessly.FeaturesUser defines a long and complex computation by composing shorter and simpler units of work, i.e., tasks.taskprocautomatically executes them in a distributed way, supporting multithreading/multiprocessing and third-party job-controlling commands such asjbsubanddocker. It also creates/reuses/discards a cache per task automatically.Full type-hinting support.NonfeaturesPeriodic schedulingAutomatic retryExternal service integration (GCP, AWS, ...)Graphical user interfaceInstallationpip install taskprocDocumentationBasicsWe define a task by class.fromtaskprocimportTask,Const,GraphclassChoose(Task):""" Compute the binomial coefficient. """def__init__(self,n:int,k:int):# Declaration of the upstream tasks.# Any instance of `Task` registered as an attirbute is considered as an upstream task.if0<k<n:self.left=Choose(n-1,k-1)self.right=Choose(n-1,k)elifk==0ork==n:# We can also set dummy tasks with their value already computed.self.left=Const(0)self.right=Const(1)else:raiseValueError(f'{(n,k)}')defrun_task(self)->int:# The main computation of the task, which is delayed until necessary.# The return values of the prerequisite tasks are accessible via `.get_result()`.returnself.left.get_result()+self.right.get_result()# A task pipeline is constructed with instantiation, which should be done inside the `Graph` context.withGraph('./cache'):# Specifies the directory to store the cache.task=Choose(6,3)# Builds a pipeline to compute 6 chooses 3.# Use the `run_graph()` method to run the pipeline.ans,stats=task.run_graph()# `ans` should be 6 chooses 3, which is 20. `stats` is the execution statistics.Commandline Interfacetaskprochas a utility classmethod to run with commandline arguments, which is useful if all you need is to run a single task.
For example, if you have# taskfile.pyfromtaskprocimportTask,DefaultCliArguments# ...classMain(Task):def__init__(self):self.result=Choose(100,50)defrun_task(self):print(self.result.get_result())# Optionally you can configure default CLI arguments.DefaultCliArguments(# ...).populate()ThenMaintask can be run with CLI:taskproc/path/to/taskfile.py-o/path/to/cache/directoryorpython-mtaskproc/path/to/taskfile.py-o/path/to/cache/directoryBesides, if you have the entrypoint inside some module, you can run it with# taskfile.py...classMain(Task):...# Must call the entrypoint explicitly.if__name__=='__main__':Main.cli()andpython-mmodule.path.to.taskfile-o/path/to/cache/directorySee alsotaskproc /path/to/taskfile.py --helporpython -m module.path.to.taskfile --helpfor more details.Futures and Task CompositionTo be more precise, any attributes of a task implementing theFutureprotocol are considered as upstream tasks.
For example,Tasks andConsts areFutures.
One can pass a future into the initialization of another task to compose the computation.fromtaskprocimportFutureclassDownstreamTask(Task):def__init__(self,upstream:Future[int],other_args:Any):self.upstream=upstream# Register upstream task...classMain(Task):def__init__(self):self.composed=DownstreamTask(upstream=UpstreamTaskProducingInt(),...)FutureListandFutureDictcan be used to aggregate multiple futures into one, allowing us to register a batch of upstream futures.fromtaskprocimportFutureList,FutureDictclassSummarizeScores(Task):def__init__(self,task_dict:dict[str,Future[float]]):self.score_list=FutureList([ScoringTask(i)foriinrange(10)])self.score_dict=FutureDict(task_dict)defrun_task(self)->float:# `.get_result()` evaluates `FutureList[T]` and `FutureDict[K, T]` into# `list[T]` and `dict[K, T]`, respectively.returnsum(self.score_dict.get_result().values())/len(self.score_dict.get_result())If a future is wrapping a sequence or a mapping, one can directly access its element with the standard indexing operation.
The result is also aFuture.classMultiOutputTask(Task):defrun_task(self)->dict[str,int]:return{'foo':42,...}classDownstreamTask(Task):def__init__(self):self.dep=MultiOutputTask()['foo']# type of Future[int]Input and Output SpecificationsIn general, tasks can be initialized with any JSON serializable objects which may or may not contain futures.
Any non-jsonable objects can be also passed, as the output of a task.SomeTask(1,'foo',bar={'baz':TaskProducingNonJsonableObj(),'other':[1,2,3]})On the other hand, the output of a task, i.e., the return value of the.run_task()method, should be serializable withcloudpickle.Data DirectoriesUsetask.task_directoryto get a fresh path dedicated to each task.
The directory is automatically created and managed along with the cache.classTrainModel(Task):defrun_task(self)->str:...model_path=self.task_directory/'model.bin'model.save(model_path)returnmodel_pathTask Label for Computational Resource ControlEach task class can be tagged with multiple labels.
The task labels are useful to configure prefix commands and concurrency limits for controlling of computational resources.classTaskUsingGPU(Task):task_label='gpu'...classAnotherTaskUsingGPU(Task):task_label=['gpu','memory']...withGraph('./cache'):# Label-wise prefix/concurrency controlSomeDownstreamTask().run_graph(# The number of tasks labeled with "gpu" running simultaneously is at most 2 (resp. "memory" is at most 1).rate_limits={'gpu':2,'memory':1,TaskUsingGPU.task_name:5,# Each task is also labeld with `cls.task_name` by default.},prefixes={# Task labeled with "gpu" is run with the job-dispatching command "jbsub ...".# Left labels prevail in the prefix collision.'gpu':'jbsub -wait -queue x86_1h -cores 16+1 -mem 64g'})Advanced TopicsExecution Policy ConfigurationOne can control the policy of parallelism withconcurrent.futures.Executorclasses.fromconcurrent.futuresimportProcessPoolExecutor,ThreadPoolExecutorclassMyTask(Task):...withGraph('./cache'):# Limit the number of parallel workersMyTask().run_graph(executor=ProcessPoolExecutor(max_workers=2))# Thread-based parallelismMyTask().run_graph(executor=ThreadPoolExecutor())Selective Cache DeletionIt is possible to selectively discard cache:withGraph('./cache'):# Selectively discard the cache of a specific task.Choose(3,3).clear_task()# `ans` is recomputed tracing back to the computation of `Choose(3, 3)`.ans,_=Choose(6,3).run_graph()# Delete all the cache associated with `Choose`.Choose.clear_all_tasks()One can also manage caches directly from the disk location, i.e.,./cachein the above.Cache CompressionThe task output is compressed withgzip.
The compression level can be changed as follows (defaults to 9), trading the space efficiency with the time efficiency.classNoCompressionTask(Task):task_compress_level=0...Built-in properties/methodsBelow is the list of the built-in attributes/properties/methods ofTask. Do not override these attributes in the subclass.NameOwnerTypeDescriptionrun_taskinstancemethodRun the tasktask_nameclasspropertyString id of the task classtask_directoryinstancepropertyPath to the data directory of the taskrun_graphinstancemethodRun the task after necessary upstream tasks and save the results in the cachecliclassmethodrun_graphwith command line argumentsclear_taskinstancemethodClear the cache of the task instanceclear_all_tasksclassmethodClear the cache of the task classget_task_configclassmethodGet task config from the current graphtask_workerinstanceattributeTask worker of instancetask_configinstanceattributeTask config of instancetask_compress_levelinstanceattributeCompression level of instancetask_labelinstanceattributeLabel of instanceget_resultinstancemethodGet the result of the task (fails if the result is missing)to_jsoninstancemethodSerialize itself as a JSON dictionaryget_workersinstancemethodGet the dictionary of the workersBrowsing cachesShow the whole task dependency tree:tree-l/<path_to_cache_directory>/<task_name>/results/<root_task_id>Show finished tasks only:tree-l-Presult.pkl.gz--prune/<path_to_cache_directory>/<task_name>/results/<root_task_id>Show finished tasks + running tasks:tree-l-P*.txt--prune/<path_to_cache_directory>/<task_name>/results/<root_task_id>TODOKnown issueCurrent task argument serialization is not ideal since JSON is mapping two different values into the same text representation (e.g., tuple and list). Consider using consistency checkx == json.loads(json.dumps(x)), or redesign the format.Feature enhancementError handling policy (eager/lazy) as an argument.Task-state tracker.Simple task graph visualizer.Pydantic/dataclass support in task arguments (as an incompatible, but better-UX object with TypedDict).Dynamic prefix generation with prefix template (e.g., for specifying the log locations).