Patent ID: 9536014
Date: 2017-01-03
CPC Classifications: G06F,H04L

Claim:
1. A computer-implemented method for parallel processing of data executed by one or more processors, the method comprising: receiving first key-value pairs corresponding to a first input data block and second key-value pairs corresponding to a second input data block different than the first input data block, wherein the first and second input data blocks are included in a data set being processed in a distributed mapreduce operation, and wherein the first key-value pairs and the second key-value pairs represent results of a map phase of the distributed mapreduce operation; identifying common key-value pairs from the received first key-value pairs and the received second key-value pairs, each common key-value pair including at least one key common to the received first key-value pairs and the received second key-value pairs, wherein at least one of the common key-value pairs corresponds to the first input data block and at least one of the common key-value pairs corresponds to the second input data block; combining the common key-value pairs into combined key-value pairs, each combined key-value pair including a key and a combined value including values from each of the common key-value pairs that include the key; determining that an output condition has occurred; and in response to determining that the output condition has occurred, outputting intermediate combined key-value pair data including at least a portion of the combined key-value pairs for processing in a reduce phase of the distributed mapreduce operation, wherein determining that the output condition has occurred includes at least one of: (i) determining that a memory usage threshold has been reached, (ii) determining that processing of the first input block and the second input block has been completed, or (iii) receiving a command indicating that the intermediate data should be output.