Package: paradag
paradagparadaga robust DAG package for easy parallel execution.paradagis implemented in pure python and totally independent of any
third party packages.Directed acyclic graph (DAG)is commonly used as a dependency graph. It could be used to describe the
dependencies of tasks.
The order of task executions must comply with the dependencies,
where tasks with direct or indirect path must run in sequence,
and tasks without any connection could run in parallel.Installation$pipinstallparadagCreate a DAGBefore running tasks, first create a DAG,
with each vertex representing a task.
The vertex of DAG instance could be anyhashable object,
like integer, string, tuple of hashable objects, instance of
user-defined class, etc.fromparadagimportDAGclassVtx(object):def__init__(self,v):self.__value=vvtx=Vtx(999)dag=DAG()dag.add_vertex(123,'abcde','xyz',('a','b',3),vtx)dag.add_edge(123,'abcde')# 123 -> 'abcde'dag.add_edge('abcde',('a','b',3),vtx)# 'abcde' -> ('a', 'b', 3), 'abcde' -> vtxadd_edgeaccepts one starting vertex and one or more ending vertices.
Please pay attention not to make a cycle withadd_edge,
which will raise aDAGCycleError.The common DAG properties are accessible:print(dag.vertex_size())print(dag.edge_size())print(dag.successors('abcde'))print(dag.predecessors(vtx))print(dag.all_starts())print(dag.all_terminals())Run tasks in sequenceWrite your executor and optionally a selector.
The executor handles the real execution for each vertex.fromparadagimportdag_runfromparadagimportSequentialProcessorclassCustomExecutor:defparam(self,vertex):returnvertexdefexecute(self,param):print('Executing:',param)print(dag_run(dag,processor=SequentialProcessor(),executor=CustomExecutor()))dag_runis the core function for task scheduling.Run tasks in parallelRun tasks in parallel is quite similar, while only change the processor
toMultiThreadProcessor.fromparadagimportMultiThreadProcessordag_run(dag,processor=MultiThreadProcessor(),executor=CustomExecutor())The default selectorFullSelectorwill try to find as many tasks
as possible which could run in parallel.
This could be adjusted with custom selector.
The following selector will only allow at most 4 tasks running in parallel.classCustomSelector(object):defselect(self,running,idle):task_number=max(0,4-len(running))returnlist(idle)[:task_number]dag_run(dag,processor=MultiThreadProcessor(),selector=CustomSelector(),executor=CustomExecutor())Once you are usingMultiThreadProcessor, great attentions must be
paid thatexecuteof executor could run in parallel. Try not to modify
any variables outside theexecutefunction, and all parameters should
be passed by theparamargument. Also make sure that the return values
generated fromparamfunction are independent.Get task running statusThe executor could also implement the optional methods which could get
the task running status.classCustomExecutor:defparam(self,vertex):returnvertexdefexecute(self,param):print('Executing:',param)defreport_start(self,vertices):print('Start to run:',vertices)defreport_running(self,vertices):print('Current running:',vertices)defreport_finish(self,vertices_result):forvertex,resultinvertices_result:print('Finished running{0}with result:{1}'.format(vertex,result))dag_run(dag,processor=MultiThreadProcessor(),executor=CustomExecutor())Deliver result to descendantsIn case the result for one task should be used for its descendants,delivermethod could be implemented in executor.classCustomExecutor:def__init__(self):self.__level={}defparam(self,vertex):returnself.__level.get(vertex,0)defexecute(self,param):returnparam+1defreport_finish(self,vertices_result):forvertex,resultinvertices_result:print('Vertex{0}finished, level:{1}'.format(vertex,result))defdeliver(self,vertex,result):self.__level[vertex]=resultThe result from parent will be delivered to the vertex before execution.Topological sortingTopological sortingcould also be done byparadag.dag_runfunction.
The return value ofdag_runcould be considered as
the result of topological sorting.A simple topological sorting without any execution:fromparadagimportSingleSelector,RandomSelector,ShuffleSelectordag=DAG()dag.add_vertex(1,2,3,4,5)dag.add_edge(1,4)dag.add_edge(4,2,5)print(dag_run(dag))print(dag_run(dag,selector=SingleSelector()))print(dag_run(dag,selector=RandomSelector()))print(dag_run(dag,selector=ShuffleSelector()))The solution for topological sorting is not necessarily unique,
and the final orders may vary with different selectors.