Title: Approximate Gradient Coding for Distributed Learning with Heterogeneous Stragglers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Distributed Learning

Straggler Mitigation

Gradient Coding

Optimal Structure
Abstract: In this paper, we propose an optimally structured gradient coding scheme to mitigate the straggler problem in distributed learning. Conventional gradient coding methods often assume homogeneous straggler models or rely on excessive data replication, limiting performance in real-world heterogeneous systems. To address these limitations, we formulate an optimization problem minimizing residual error while ensuring unbiased gradient estimation by explicitly considering individual straggler probabilities. We derive closed-form solutions for optimal encoding and decoding coefficients via Lagrangian duality and convex optimization, and propose data allocation strategies that reduce both redundancy and computational load. We also analyze convergence behavior for $\lambda$-strongly convex and $\mu$-smooth loss functions. Numerical results show that our approach significantly reduces the impact of stragglers and accelerates convergence compared to existing methods.


Peer reviews:
comments: SUMMARY:
This paper propose a gradient coding scheme for distributed machine learning, which minimizes residual error and maintaining an unbiased gradient estimator. Theoretical analysis and simulations are provided.

STRENGTHS AND WEAKNESSES:
**Strengths**:

1. This paper proposed one method that minimizes the residual error and ensures the gradient estimator's unbiasedness.
2. This paper is very well-written, well-organized.
3. This paper is self-contained and very clear. This is the first paper I read about gradient coding, however, it is very easy for me to follow the main idea of this paper.

**Weaknesses**:

After checking all the proofs (except the convergence analysis, i.e., Theorems 2 and 3) carefully, I have not found any serious weaknesses.

QUESTIONS:
**Questions**:

1. How does the paper get the conclusion that (P2) is non-convex and (P3) is convex quickly? Can you provide more details. Are there some rules to help you get these conclusions quickly. I am curious about this.

2. Lines 219-220:

   > This design reduces the overall computation load on individual worker nodes.

   Can you kindly explain this in detail?

**Suggestions**:

1. The font size of the appendix seems smaller than the font size of the main body. It may be better to use the same font size for both.
2. Equation (61): For me, Equation (61) is unnecessary. From Equation (60) to Equation (62) is straightforward by Equations (58) and (59).

I cannot guarantee a high score (for example, 5 and 6), as I am not sure whether this simple way is appreciated by the experts in gradient coding.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an optimally structured gradient coding scheme to mitigate the straggler problem in distributed learning. To tackle with the stragglers in the real-world heterogeneous systems, an optimization problem is formulated in this paper to minimize the residual error while ensuring unbiased gradient estimation by explicitly considering individual straggler probabilities. Theoretical analysis of convergence is provided for strongly convex and smooth functions. The experiments show that the proposed algorithm can significantly mitigate the impact of stragglers and accelerates convergence.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper proposes an optimally structured gradient coding scheme to mitigate the straggler problem in distributed learning. To tackle with the stragglers in the real-world heterogeneous systems, an optimization problem is formulated in this paper to minimize the residual error while ensuring unbiased gradient estimation by explicitly considering individual straggler probabilities.\n\n2. Theoretical analysis of convergence is provided for strongly convex and smooth functions.\n\n3. The experiments show that the proposed algorithm can significantly mitigate the impact of stragglers and accelerates convergence.\n\nWeaknesses:\n\n1. The theoretical analysis is limited to the strongly convex and smooth functions, which is less practical for real-world applications.\n\n2. For the experiments, the optimizers are limited to SGD, which is not widely used these days compared to Adam.\n\nQUESTIONS:\n1. Is it possible to extend the theoretical analysis to non-convex or non-smooth functions?\n\n2. Is the proposed algorithm also compatible to Adam optimizer?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations and potential negative societal impact of their work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the structured gradient coding scheme to mitigate the straggler issue in distributed learning without relying on homogeneous straggler models or data replication. The authors derive closed-form solutions for the optimal coding and decoding coefficients and propose data allocation strategies to reduce the computation load.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper has a good motivation of improving the training efficiency in distributed learning, and gradient coding is important technique in distributed computing. \n* The proposed method does not require too much additional data allocation.\n* The theoretical analysis in section 3.1 is clear and easy to follow. \n\nWeaknesses:\n* We usually use (mini-batch) SGD for training in practice, and this paper also mentions batch sampling. However, it seems that the gradient $g^{(t)}$ is defined as the aggregated gradient at iteration $t$. It is unclear why the later analysis is not affected by the use of mini-batch gradients.\n* The writing in section 3.2 could be further improved, I think the amount of data sharing and allocation is still a little bit confusing. \n* The optimal structured gradient requires the known probability $p_1$ to $p_n$, which would be hard to predict during real application.\n* The experimental setup is somewhat insufficient. The model used in the experiments is relatively small, especially in the context of the current LLM era. I would suggest that the authors include some scaling-up experiments to better demonstrate the practicality of the proposed method. \n* Moreover, is the proposed method limited to gradient coding with SGD/GD optimizer? If so, it should be a limitation of the method for practical usage as well.\n\nQUESTIONS:\nThe questions are primarily included in the weaknesses part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies distributed gradient descent with the presence of straggler nodes. Linear coding is used to combat stragglers, by allocating segment $i$ of the data to $d_i$ workers, and sharing a linear combination of the gradients with a central server. The gradients are then (approximately) decoded (linearly) from the set of non-straggler nodes. The main technical problem the authors solve is: design the encoding and decoding matrices such that the gradient variance is minimized under the constraint that the gradient estimate is unbiased.\nUnder bounded gradient assumption, the optimization problem (variance and unbiasedness conditioin) is rewritten as a function of the encoding/decoding coefficients and formulated as a convex optimization problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is well-written and easy to follow.\n2. The authors propose optimal (minimizing variance) encoding schemes that also have a low computation load (average number of replicates across partitions).\n3. The condition of unbiased gradient is reasonable as it guarantees convergence of GD.\n4. The authors additionally prove properties for the optimal encoding scheme.\n5. Simulation results show better GD convergence compared to baselines.\n\nWeakness:\n1. The authors look for an encoding scheme with low computation load. However, the computation load is defined as the average number of repetitions for each partition, and does not take into account how these repetitions are allocated across nodes. Another important quantity is the load per worker node; namely: the maximum number of partitions allocated to a single worker.\n2. While the original formulation of P2 is non-convex due to the multiplication between $w$'s and $a$'s. The products can be absorbed into a single variable to reach a convex optimization problem.\n\nQUESTIONS:\nCould the authors discuss the worst-case computation load across nodes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}