Patent ID: 8209664

Claim:
A machine implemented method for distributed parallel processing, comprising: accessing an expression from a sequential program that is executing at a first machine, the expression invoking at least one extension for distributed parallel processing by a distributed execution engine, wherein the distributed execution engine includes a compute cluster having a plurality of nodes, the at least one extension is an operator specifying a user-defined function using data parallel processing, and wherein the user-defined function is in a high-level programming language and receives a first dataset; automatically generating an execution plan for parallel processing of the expression by the distributed execution engine using the at least one extension, wherein the automatically generating the execution plan includes: specifying a partitioning of the first dataset at a set of nodes from said plurality of nodes; determining whether the user-defined function includes an annotation that the user-defined function is operable on partitions of the first dataset; if the user-defined function includes the annotation, automatically generating code for executing the user-defined function in parallel at the set of nodes; and if the user-defined function does not include the annotation, automatically generating code for streaming partitions of the first dataset to a single node and code for executing the user-defined function at the single node; and providing the execution plan to the distributed execution engine for controlling parallel execution of the expression.