Patent ID: 9645867
Date: 2017-05-09
CPC Classifications: G06F

Claim:
1. A computer implemented method for shuffle optimization in map-reduce processing, comprising: obtaining intermediate results from a plurality of mappers for an application on a computing device; combining the intermediate results from the plurality of mappers; partitioning the combined intermediate results into intermediate results for respective reducers for the application; storing the intermediate results for the respective reducers into a buffer storage; partitioning the buffer storage into buffer sections for the respective reducers to store their respective intermediate results, wherein sizes of the buffer sections for the respective reducers are determined based on a predicted ratio of amounts of the intermediate results for respective reducers; monitoring a buffer storage usage and a rate of increase in the intermediate results stored in the buffer storage; predicting, based on the rate of increase in the intermediate results, a time at which a data capture notification shall be sent; sending the data capture notification, to indicate that the intermediate results for respective reducers stored in the buffer storage can be transmitted to the respective reducers, based on a predicted time being reached; and transmitting the intermediate results for the respective reducers to the respective reducers, based on the monitored buffer storage usage.