Patent ID: 9684513
Date: 2017-06-20
CPC Classifications: G06F

Claim:
1. A method to adaptively pipeline a map stage of a MapReduce job, the method comprising: receiving, by one or more processors, into a first memory buffer, one or more data records from a storage, wherein a size of the first memory buffer is adaptive to one or more utilizations of one or more resources in a processor, wherein the processor is included in the one or more processors; inserting, by the one or more processors, the first memory buffer into a first queue, wherein a size of the first queue is adaptive to one or more utilizations of one or more resources in the processor; generating, by the one or more processors, one or more output records from the first memory buffer in the first queue by applying a map function to the one or more data records in the first memory buffer; writing, by the one or more processors, the one or more output records into a second memory buffer in a second queue, wherein the size of the second memory buffer and the size of the second queue is adaptive to one or more utilizations of one or more resources in the processor; deleting, by the one or more processors, the first memory buffer from the first queue; generating, by the one or more processors, one or more sorted records in the second memory buffer by sorting the one or more output records in the second memory buffer; responsive to generation, of one or more sorted records in the second memory buffer that overflow the second memory buffer, executing, by the one or more processors, a merge and spill of the one or more sorted records to storage; writing, by the one or more processors, the second memory buffer into a third queue, wherein the size of the third queue is adaptive to one or more utilizations of one or more resources in the processor; deleting, by the one or more processors, the second memory buffer from the second queue; merging, by the one or more processors, one or more sorted records in the second memory buffer into an output data file that is stored in storage; and deleting, by the one or more processors, the second memory buffer from the third queue.