Patent ID: 8612510

Claim:
A system for large-scale processing of data in a distributed and parallel processing environment including a set of interconnected computing systems, comprising: a plurality of interconnected processors; an application-independent framework for processing data, including: a plurality of application-independent map modules that execute on at least a first subset of the plurality of interconnected processors; and, a plurality of application-independent reduce modules, distinct from the plurality of application-independent map modules, that execute on at least a second subset of the plurality of interconnected processors, wherein the application-independent map modules and application-independent reduce modules use application-independent operators to-automatically handle parallelization of computations across the distributed and parallel processing environment when performing user-specified data processing operations; wherein the plurality of application-independent map modules and the plurality of application-independent reduce modules are independent of a plurality of user-specified, application-specific operators and operations; and the plurality of user-specified, application-specific operators, for use with the application-independent framework, perform the user-specified data processing operations on a user-specified set of input files, the plurality of user-specified, application-specific operators including: a map operator that produces intermediate data values, wherein the map operator is applied by the application-independent map modules to input data in the user-specified set of input files to produce the intermediate data values; and a reduce operator that produces final output data for at least one of the user-specified data processing operations, wherein the reduce operator is applied by the application-independent reduce modules to process the intermediate data values to produce the final output data for the user-specified data processing operation.