Patent ID: 8869165

Claim:
A method for executing a batch of multiple workflows on multiple domains, comprising steps of: receiving a request for workflow execution of a batch of multiple workflows on multiple domains; sending at least one of one or more individual jobs in each of the multiple workflows and dependency information to a scheduler; assigning (i) the at least one of one or more individual jobs of the multiple workflows and (ii) one or more data activities of the multiple workflows to the multiple domains such that total makespan is minimized subject to multiple constraints, wherein said multiple constraints comprise: a feasibility constraint that ensures that each of one or more individual jobs finishes in a first given time interval at a first given domain; a compute constraint that ensures that a number of jobs that complete in a second given time interval at a second given domain does not exceed a specified threshold; a storage constraint that ensures that the size of any replicated data items is within a storage limit; a precedence constraint that ensures that a job does not begin execution before its preceding job finishes; and an affinity constraint that ensures that each of one or more affine jobs are co-located; computing, based on said assigning, one or more outputs, wherein the one or more outputs comprise (i) one or more resource mappings for one or more jobs and one or more data transfers in each of the multiple workflows and (ii) a resource specific ordered schedule of one or more jobs and one or more data transfers across the multiple workflows; and integrating execution of separate tasks of orchestration of the multiple workflows and scheduling of (i) the at least one of one or more jobs and (ii) one or more data transfers of the multiple workflows, wherein the integrating is used to execute the batch of multiple workflows based on the one or more outputs of the scheduler, static information, and run-time adaptations due to varying load, and wherein said integrating comprises: selecting a given job from the one or more jobs to move from a first queue based on (i) average queue length of the first queue, (ii) recency of a prior job move from the first queue, and (iii) one or more data characteristics of the given job; and targeting a second queue to move the given job based on minimizing (i) execution time of the given job, (ii) queue wait time, and (iii) data transfer time.