Patent ID: 9747131
Date: 2017-08-29
CPC Classifications: G06F,H04L

Claim:
1. A computer-implemented method in a distributed data processing job comprising: executing a plurality of processes on a plurality of interconnected hardware processing devices, wherein a plurality of the processes are worker processes; executing a master process configured to control scheduling and data distribution tasks among the plurality of worker processes, wherein the master process is not one of the worker processes; assigning the plurality of worker processes to a group identified from the master process; identifying as a lead worker one of the plurality of worker processes that has been assigned to the group identified from the master process; associating a metric variable with the group identified from the master process, wherein the metric variable is an amount of a shared resource used by the worker process; by each of the worker processes of the plurality of worker processes assigned to the group identified from the master process: asynchronously aggregating, by the lead worker, the posted values for the metric variable into an aggregated value; asynchronously announcing, by the lead worker, the aggregated value to the worker processes; and identifying one or more of the plurality of worker processes, other than the lead worker, as a standby lead worker, each of which maintains a history of worker processes' posts, preserves the aggregated value, and is configured to take over as lead worker if the lead worker fails or leaves the group.