Patent ID: 8209703

Claim:
A non-transitory computer readable storage medium, comprising executable instructions to: assess system cache resources, inter-process communication requirements and staging requirements to divide an extract, transform, load (ETL) dataflow task into a plurality of sub-tasks; and execute the sub-tasks on distributed resources; wherein a directed acyclic graph (DAG) is generated that represents execution order dependencies among the plurality of sub-tasks and is used to divide the ETL task based on the accessed system cache resources, inter-process communication requirements and staging requirements, each node of the DAG representing a group of sub-data flows that can be executed together, each of the sub-data flows representing a sub-task process at run time, arrows between two nodes of the DAG indicating a dependency between two groups of sub-data flows; wherein: the nodes of the DAG are sorted to generate a list of prioritized group of sub-data flows; the sub-tasks are executed according to execution priorities specified by the list of prioritized groups of sub-data flows such that: sub-data flows having different priorities are run in serial; sub-data flows belonging to a group having a higher priority are executed prior to sub-data flows belonging to groups having a lower priority; and sub-data flows having a same priority are run in serial or in parallel.