Package: taskgraph
TaskGraphis a library that was developed to help manage complicated
computational software pipelines consisting of long running individual tasks.
Many of these tasks could be executed in parallel, almost all of them wrote
results to disk, and many times results could be reused from part of the
pipeline. TaskGraph manages all of this for you. With it you can schedule
tasks with dependencies, avoid recomputing results that have already been
computed, and allot multiple CPU cores to execute tasks in parallel if
desired.TaskGraph DependenciesTask Graph is written in pure Python, but if thepsutilspackage is
installed the distributed multiprocessing processes will beniced.Example UseInstallTaskGraphwithpip install taskgraphThenimportosimportpickleimportloggingimporttaskgraphlogging.basicConfig(level=logging.DEBUG)def_create_list_on_disk(value,length,target_path):"""Create a numpy array on disk filled with value of `size`."""target_list=[value]*lengthpickle.dump(target_list,open(target_path,'wb'))def_sum_lists_from_disk(list_a_path,list_b_path,target_path):"""Read two lists, add them and save result."""list_a=pickle.load(open(list_a_path,'rb'))list_b=pickle.load(open(list_b_path,'rb'))target_list=[]fora,binzip(list_a,list_b):target_list.append(a+b)pickle.dump(target_list,open(target_path,'wb'))# create a taskgraph that uses 4 multiprocessing subprocesses when possibleif__name__=='__main__':workspace_dir='workspace'task_graph=taskgraph.TaskGraph(workspace_dir,4)target_a_path=os.path.join(workspace_dir,'a.dat')target_b_path=os.path.join(workspace_dir,'b.dat')result_path=os.path.join(workspace_dir,'result.dat')result_2_path=os.path.join(workspace_dir,'result2.dat')value_a=5value_b=10list_len=10task_a=task_graph.add_task(func=_create_list_on_disk,args=(value_a,list_len,target_a_path),target_path_list=[target_a_path])task_b=task_graph.add_task(func=_create_list_on_disk,args=(value_b,list_len,target_b_path),target_path_list=[target_b_path])sum_task=task_graph.add_task(func=_sum_lists_from_disk,args=(target_a_path,target_b_path,result_path),target_path_list=[result_path],dependent_task_list=[task_a,task_b])task_graph.close()task_graph.join()# expect that result is a list `list_len` long with `value_a+value_b` in itresult=pickle.load(open(result_path,'rb'))CaveatsTaskgraph’s default method of checking whether a file has changed
(hash_algorithm='sizetimestamp') uses the filesystem’s modification
timestamp, interpreted in integer nanoseconds.  This check is only as
accurate as the filesystem’s timestamp.  For example:FAT and FAT32 timestamps have a 2-second modification timestamp resolutionexFAT has a 10 millisecond timestamp resolutionNTFS has a 100 nanosecond timestamp resolutionHFS+ has a 1 second timestamp resolutionAPFS has a 1 nanosecond timestamp resolutionext3 has a 1 second timestamp resolutionext4 has a 1 nanosecond timestamp resolutionIf you suspect timestamp resolution to be an issue on your filesystem, you
may wish to store your files on a filesystem with more accurate timestamps or
else consider using a differenthash_algorithm.Running TestsTaskgraph includes atoxconfiguration for automating builds across
multiple python versions and whetherpsutilis installed.  To execute all
tests on all platforms, run:$ toxAlternatively, if you’re only trying to run tests on a single configuration
(say, python 3.7 withoutpsutil), you’d run:$ tox -e py37Or if you’d like to run the tests for the combination of Python 3.7 withpsutil, you’d run:$ tox -e py37-psutilIf you don’t have multiple python installations already available on your system,
an easy way to accomplish this is to usetox-conda(https://github.com/tox-dev/tox-conda) which will use conda environments to manage
the versions of python available:$ pip install tox-conda
$ tox