A computer-implemented method executed by one or more processors, the method comprising: determining a target completion time for a batch data processing job of an input data set, the batch data processing job performed by a plurality of tasks (102; 204), each of the plurality of tasks processing a different input shard (104; 206) that includes a different portion of the input data set; identifying that the batch data processing job has entered an endgame state where all shards associated with the input data set are being processed by a task and there is at least one idle task associated with the batch processing job and in response: identifying, from the plurality of tasks, a first task having an estimated completion time greater than the target completion time of the batch data processing job, the estimated completion time representing an estimated time at which the first task will complete processing the portion of the input data set at a first input shard being processed by the first task; splitting the first input shard into a first split input shard corresponding to the portion of the shard that the task expects to complete by the target completion time of the batch data processing job and a second split input shard corresponding to the portion that the task does not expect to complete; returning the second split input shard to a supervisor, wherein the at least one idle task claims the second split input shard for processing. determining a target completion time for a batch data processing job of an input data set, the batch data processing job performed by a plurality of tasks (102; 204), each of the plurality of tasks processing a different input shard (104; 206) that includes a different portion of the input data set; identifying that the batch data processing job has entered an endgame state where all shards associated with the input data set are being processed by a task and there is at least one idle task associated with the batch processing job and in response: identifying, from the plurality of tasks, a first task having an estimated completion time greater than the target completion time of the batch data processing job, the estimated completion time representing an estimated time at which the first task will complete processing the portion of the input data set at a first input shard being processed by the first task; splitting the first input shard into a first split input shard corresponding to the portion of the shard that the task expects to complete by the target completion time of the batch data processing job and a second split input shard corresponding to the portion that the task does not expect to complete; returning the second split input shard to a supervisor, wherein the at least one idle task claims the second split input shard for processing. identifying, from the plurality of tasks, a first task having an estimated completion time greater than the target completion time of the batch data processing job, the estimated completion time representing an estimated time at which the first task will complete processing the portion of the input data set at a first input shard being processed by the first task; splitting the first input shard into a first split input shard corresponding to the portion of the shard that the task expects to complete by the target completion time of the batch data processing job and a second split input shard corresponding to the portion that the task does not expect to complete; returning the second split input shard to a supervisor, wherein the at least one idle task claims the second split input shard for processing.