Patent ID: 9389995
Filing Date: 2016-07-12
CPC Classification: G06F

Claim Text:
1. A method for optimizing a MapReduce shuffle, the method comprising: receiving information associated with partition segments of map task outputs and a pipeline policy for a job running on a computing device; transmitting to an operating system of the computing device a request to lock partition segments of the map task outputs in a memory of the computing device; transmitting to the operating system of the computing device an advisement to keep or load partition segments of map task outputs in the memory of the computing device, based on a capacity of the memory of the computing device, wherein partition segments of the map task outputs requested to be locked in the memory of the computing device are different from the partition segments of map task outputs advised to keep or load in the memory of the computing device; creating a pipeline based on the pipeline policy, wherein the pipeline includes partition segments of map task outputs locked in the memory of the computing device and partition segments of map task outputs advised to keep or load in the memory; of the computing device for the job; and selecting the partition segments locked in the memory, followed by the partition segments advised to keep or load in the memory, as a preferential order of partition segments to shuffle.