Patent ID: 8555265

Claim:
A method comprising: executing an application that includes a data parallel pipeline, wherein the data parallel pipeline specifies multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects; generating, based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline; applying one or more graph transformations to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations; and executing the deferred, combined parallel operations to produce materialized parallel data objects corresponding to the deferred parallel data objects, wherein the deferred, combined parallel data operations includes at least one generalized mapreduce operation, the generalized mapreduce operation including multiple, parallel map operations and multiple, parallel reduce operations and being translatable to a single mapreduce operation that includes a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations, and wherein each deferred parallel operation includes a pointer to a parallel data object that is an input to the deferred parallel operation and a pointer to a deferred parallel object that is an output of the deferred parallel operation.