Title: Leveraging partial stragglers within gradient coding
Venue: NeurIPS 2024
Accepted: yes
Keywords: gradient coding

stragglers

communication-efficient
Abstract: Within distributed learning, workers typically compute gradients on their assigned dataset chunks and send them to the parameter server (PS), which aggregates them to compute either an exact or approximate version of $\nabla L$ (gradient of the loss function $L$). However, in large-scale clusters, many workers are slower than their promised speed or even failure-prone. A gradient coding solution introduces redundancy within the assignment of chunks to the workers and uses coding theoretic ideas to allow the PS to recover $\nabla L$ (exactly or approximately), even in the presence of stragglers. Unfortunately, most existing gradient coding protocols are inefficient from a computation perspective as they coarsely classify workers as operational or failed; the potentially valuable work performed by slow workers (partial stragglers) is ignored. In this work, we present novel gradient coding protocols that judiciously leverage the work performed by partial stragglers. Our protocols are efficient from a computation and communication perspective and numerically stable. For an important class of chunk assignments, we present efficient algorithms for optimizing the relative ordering of chunks within the workers; this ordering affects the overall execution time. For exact gradient reconstruction, our protocol is around $2\times$ faster than the original class of protocols and for approximate gradient reconstruction, the mean-squared-error of our reconstructed gradient is several orders of magnitude better.

Leveraging partial stragglers within gradient coding

Abstract

Within distributed learning, workers typically compute gradients on their assigned dataset chunks and send them to the parameter server (PS), which aggregates them to compute either an exact or approximate version of ∇ L (gradient of the loss function L ). However, in large-scale clusters, many workers are slower than their promised speed or even failure-prone. A gradient coding solution introduces redundancy within the assignment of chunks to the workers and uses coding theoretic ideas to allow the PS to recover ∇ L (exactly or approximately), even in the presence of stragglers. Unfortunately, most existing gradient coding protocols are inefficient from a computation perspective as they coarsely classify workers as operational or failed; the potentially valuable work performed by slow workers (partial stragglers) is ignored. In this work, we present novel gradient coding protocols that judiciously leverage the work performed by partial stragglers. Our protocols are efficient from a computation and communication perspective and numerically stable. For an important class of chunk assignments, we present efficient algorithms for optimizing the relative ordering of chunks within the workers; this ordering affects the overall execution time. For exact gradient reconstruction, our protocol is around 2 × faster than the original class of protocols and for approximate gradient reconstruction, the mean-squared-error of our reconstructed gradient is several orders of magnitude better.

1 Introduction

Large scale distributed learning is the workhorse of modern day machine learning (ML) algorithms. The sheer size of the data and the corresponding computation needs, necessitate the usage of huge clusters for the purpose of parameter fitting in most ML problems of practical interest: deep learning [1], low-rank matrix completion [2] etc. A typical scenario consists of a dataset D = { ( x i , y i ) } ˜ N i =1 of ˜ N data points, where x i 's and y i 's are the features and labels respectively. We wish to minimize a loss function L = 1 ˜ N ∑ ˜ N i =1 l ( x i , y i , w ) with respect to w ∈ R d ( w : parameter vector, l : prediction error). When D is large, we can perform the learning task in a distributed manner [3].

PS computes the desired gradient ∇ L = 1 ˜ N ∑ ˜ N i =1 ∇ l ( x i , y i , w t ) and the updated parameter w t +1

Background: We partition D into N equal-sized chunks denoted D i , i ∈ [ N ] ( [ n ] denotes the set { 1 , . . . , n } ), where a chunk is a subset of the data points and distinct chunks are disjoint. Within each chunk, the assignment of the data points to the workers is identical. Suppose that there are m workers W 1 , . . . , W m and a parameter server (PS). We distribute the chunks to the different workers and let them compute the gradients on the data points assigned to them. The PS coordinates the training by aggregating the (partial) gradients from the workers and transmitting an updated parameter vector to the workers at each iteration. In the 'baseline' scheme, N = m , W j is assigned D j and it computes ∑ i ∈D j ∇ l ( x i , y i , w t ) (a vector of lengthd ) and sends it to the PS. Using these, the

thereafter. Unfortunately, in many large scale clusters, workers are often slower than their promised speed or even prone to failure. This is especially true in cloud platforms, where the load often fluctuates depending on system load and spot instance pricing [4] explicitly builds in the possibility of job preemption. To address these issues, gradient coding (GC) introduced in [5] incorporates redundancy within the assignment of chunks to the workers. Once a given worker calculates the gradient on all its assigned chunks, it computes a specified linear combination of these gradients and sends it to the PS. An exact gradient coding solution allows the PS to exactly recover ∇ L even in the presence of limited node failures.

̸

Let A ∈ R N × m be a matrix such that A i,j = 0 if and only if D i is assigned to W j ; henceforth, we call this the assignment matrix. Let nnz ( v ) denote the number of non-zero entries in a vector v . Let γ i and δ j denote nnz ( A ( i, :) and nnz ( A (: , j )) (using MATLAB notation). These correspond respectively to the number of times D i appears in the cluster (replication factor) and the number of chunks assigned to W j (load factor). We assume that at most s workers out of m are stragglers. Let g D j = ∑ i ∈D j ∇ l ( x i , y i , w t ) , so that ∇ L = ∑ N j =1 g D j . At iteration t , worker W j calculates g D i for all i ∈ supp ( A (: , j )) (non-zero entries in A (: , j ) ) and linearly combines them to obtain g j = ∑ N i =1 A i,j g D i . It subsequently transmits g j to the PS. For decoding ∇ L , the PS picks a decoding vector r which is such that r j = 0 if W j has not transmitted g j (we say that W j is straggling in this case). It subsequently calculates

Related Work: Under the original GC model [5], for exact gradient coding , we want that Ar = 1 (the all-ones vector) under any choice of at most s stragglers. This means that the PS can perform a full gradient update. For exact GC, we need γ i ≥ s + 1 for all i ∈ [ N ] . This setting has been studied extensively [6, 7, 8, 9, 10, 11]. Approximate gradient coding [6] considers the scenario where the full gradient is either not required (e.g., SGD [12] works with an inexact gradient) or too costly to work with because of the high replication factor needed. In this setting, we want to design A such that || Ar -1 || 2 ( ℓ 2 -norm) is small over all possibilities for the straggling workers. Prior work demonstrates constructions from expander graphs [6], sparse random graphs [13] and [14], block designs [15, 16] and the like. Within distributed training, a significant time cost is associated with the transmission of the computed gradients (vectors of lengthd ) by the workers to the PS [17, 18], e.g., deep learning usually operates in the highly over-parameterized regime [19] ( d ≫ ˜ N ). For exact GC, if γ i ≥ s + ℓ for i ∈ [ N ] , then the dimension of the transmitted vectors from the workers can be lowered to d/ℓ [20, 21], thus saving on communication time. This is referred to as communication-efficient GC and allows us to trade-off communication for computation. However, both [20] and [21] use polynomial interpolation in their solution. This leads to significant numerical instability [22] to the extent that their solution is essentially unusable for systems with twenty or more workers. This point is also acknowledged within the papers: Section V of [21] and Section II of [20]. Some work considering these issues appears in [23] under restrictive parameter assumptions.

The usage of the partial work performed by stragglers has been considered [10, 24, 25, 26, 27, 28] only within exact GC (i.e., approximate GC has not been considered); some of these apply in the communication-efficient setting. However, these approaches use multiple messages from the workers to the PS and thus incur a higher communication cost per iteration.

Motivation: Consider Fig. 1a where the edge labels indicate the encoded gradients. Note that under ideal operating conditions when each worker operates at the same speed, the overall gradient can be computed as long as each W i processes its first chunk D i for i = 1 , . . . , 3 . Thus, it is quite wasteful for the workers to continue processing their second assigned chunk as specified in the original GC scheme; it would double the computation time. In addition, the original GC formulation ignores partial work by slow but not failed workers; in the sequel, we refer to these as 'partial stragglers'. For instance, in Fig. 1a, we consider a scenario, where W 1 is slow and W 3 is failed. The state of the computation is such that there is enough information for the PS to obtain the full gradient. However, under the original GC model, W 1 will either wait until it processes D 2 before generating the encoded gradient to be sent to the PS, or the PS will treat W 1 as failed and will have to settle for an approximate gradient.

D

1

D

2

D

3

D

2

D

3

D

1

PS

g

D

1

+2

g

D

2

g

D

2

-

g

D

3

g

D

1

+2

g

D

3

W

1

W

2

W

3

(a)

D

1

D

1

D

1

D

2

D

1

D

2

D

3

D

4

D

5

D

2

D

3

D

3

D

4

D

5

D

4

D

5

PS

W

1

W

2

W

3

W

4

W

5

(b)

Figure 5: Figure 5

Gradient coding can be viewed as an application of coding-theoretic techniques [29] to distributed learning; it allows the PS to be resilient to failures with very little coordination in the cluster. We note here that within classical coding theory [29] most constructions of erasure codes do not consider feedback from the receiver to the sender since such feedback may be expensive or noisy when the sender and receiver are at remote locations or not even necessary ([30], Chap. 7). However, feedback is quite easy to implement in the distributed learning setup.

3 Main Contributions: We present a new GC protocol that exploits a small amount of additional interactive communication to/from the PS and the workers. It greatly improves the computationefficiency and communication-efficiency of GC while continuing to allow for efficient coordination within the cluster. Specifically, our protocol efficiently leverages the chunks processed by partial stragglers. Prior work that potentially applies in our setting suffers from the serious problem of numerical instability. In contrast, our protocol is provably numerically stable.

To our best knowledge, despite the importance of communication-efficiency, there are hardly any schemes for communication-efficient approximate GC. Our protocol provides an elegant way to address this problem, which in addition allows the PS to obtain an accurate estimate of the meansquare-error of the reconstructed gradient at any given point in the computation.

Prior work in the GC area ignores the relative ordering of the chunks within the workers. As our protocol leverages partial work performed by the workers, the relative ordering of the chunks within the workers is an important factor in the performance of the algorithm. For a large class of assignment matrices, we present an efficient polynomial-time algorithm that returns the optimal ordering of the chunks within workers.

2 Gradient Coding for partial stragglers

Our GC protocol operates under the following assumptions: (i) the workers know the assignment matrix and the ordering of the chunks within all the workers, (ii) at regular intervals the workers keep communicating to the PS, the number of chunks they have processed, and (iii) the PS wants the workers to communicate vectors of length d/ℓ for integer ℓ ≥ 1 .

We now provide a top-level description of our protocol; a formal statement appears in Algorithm 1. At the beginning of the training process, the PS generates a random ℓ × m matrix R with i.i.d. entries from a standard normal distribution, N (0 , 1) and shares it with all the workers. The workers keep reporting the number of chunks that they have processed in an iteration. The PS keeps monitoring these counts, and at a certain point, it broadcasts to all the workers an 'encode-and-transmit' signal and an integer vector ψ of lengthm that specifies the number of chunks that have been processed by each node. This shares the global system state amongst the workers. For exact GC, the PS sends the encode-and-transmit signal when at least ℓ copies of each D i have been processed across the cluster. For approximate GC, the PS can send the signal even earlier.

Algorithm 1 Find-Encoding-Coeff

Input: ℓ × m matrix R , m -length state vector ψ , δ i the number of chunks processed by W i . Output: Encoding coefficients for the i -th worker ε i .

1: W i forms the indeterminate matrix m × Nℓ matrix B based on ψ .

2: W i extracts the columns of B that correspond to its processed chunks. This matrix is called ˜ B i ( cf. (4)).

3: Worker W i solves the following minimumℓ 2 -norm least-squares problem, where the variables are the indeterminates in ˜ B i .

4: Set ε i = B ( i, :) .

Following this, the workers need to decide their own encoding coefficients for the gradients that they have computed. Each gradient g D i is block-partitioned into ℓ disjoint parts g D i [ k ] , k = 0 , . . . , ℓ -1 . Each worker forms a matrix B of dimension m × Nℓ that consists of indeterminates that specify the encoding coefficients of all the workers (see example in the upcoming Section 2.2). Matrix B consists of N block-columns, where each block-column itself consists of ℓ columns of lengthm , i.e., B = [ B (1) | B (2) | . . . | B ( N ) ] . The j -th block column, B ( j ) is associated with g D j [ k ] , k = 0 , . . . , ℓ -1 . Based on the global state vector ψ , all workers know whether a chunk D j , j ∈ [ N ] has been processed by worker W i , i ∈ [ m ] . If D j has been processed by W i , then the i -th row of B ( j ) is populated with indeterminates, otherwise these entries are set to zero. Once the indeterminates are found (see Algorithm 1 for a description), worker W β encodes its gradients as

Let ⃗ e i denote the i -th canonical basis vector of lengthℓ , i.e., it contains a 1 at location i ∈ { 0 , . . . , ℓ -1 } and zeros elsewhere. We denote where ⊗ denotes the Kronecker product and 1 N denotes the all-ones vector of length N . Recall that in the exact GC scenario, the PS wants to obtain ∑ N i =1 g D i [ k ] for k = 0 , . . . , ℓ -1 . This is equivalent to requiring that

Our protocol is such that each W i can independently calculate their encoding coefficients, such that collectively all the workers agree on the same matrix B with the same values assigned to all the indeterminates. Towards this end, let ˜ B j denote the submatrix of B that is relevant to W j , i.e., the block-columns in which the processed chunks of W j participate. For instance, suppose that W j has processed α j ≤ δ j chunks D i 1 , . . . , D i α j where 1 ≤ i 1 < i 2 < · · · < i α j ≤ N . Then,

W j then solves a minimumℓ 2 -norm least-squares solution to determine its encoding coefficients (see (5) in Algorithm 1). This ensures the solution is unique [31] even if the corresponding problem is under-determined. Thus, the workers automatically agree on the same solution.

Remark 1. If ℓ = 1 , then it is possible to arrive at a protocol whereby the workers agree to transmit appropriately weighted partial sums of their gradients, so that the PS can exactly/approximately recover the sum. However, in the communication-efficient setting when ℓ > 1 , the encoding is no longer straightforward. Thus, ℓ > 1 is the main scenario we consider in what follows.

Remark 2. We discussed that each worker can form the m × Nℓ matrix of indeterminates B for the sake of ease of explanation. In fact, W j only works with ˜ B j , which is of size at most m × δℓ .

Remark 3. The additional communication assumed in our protocol is only O ( m ) as against the parameter vector of lengthd and O ( m ) ≪ d . Thus, the additional communication cost of our algorithm is minimal.

2.1 Analysis of Algorithm 1

It is evident from our description and Algorithm 1 that upon receiving the encode-and-transmit signal and the vector ψ , each worker can independently create the matrix of indeterminates B .

Exact GC analysis: Suppose that each D i , i ∈ [ N ] has been processed at least ℓ times across the cluster, and suppose that W i has processed D j . This means that there is a block-column B ( j ) such that each column within B ( j ) has at least ∆ ≥ ℓ indeterminates that need to be assigned values. For one column within B ( j ) , W i will solve a system of equations that is specified by ℓ × ∆ submatrix of R denoted X (an example appears in Section 2.2). Note that the columns of X will be linearly independent with high probability owing to the choice of R and since ∆ ≥ ℓ , a solution is guaranteed to exist. Let κ 2 ( M ) denote the condition number of matrix M . For a random ℓ × ∆ matrix with i.i.d. N (0 , 1) entries, it is known that E (log κ 2 ( X )) ≤ O (log ∆) [32]. Thus, each such system of equations is well-conditioned with very high probability. In the under-determined case when ∆ > ℓ , each worker will still agree on the same values of the corresponding indeterminates since we enforce that we work with the (unique) minimum ℓ 2 -norm solution; no additional communication between the workers is required.

Approximate GC analysis: It is possible that the PS sends the encode-and-transmit vector when some D i has been processed ∆ ≤ ℓ -1 times. In this case, the corresponding step for B ( i ) in (5) will be an over-determined least-squares procedure, which implies that there will be a non-zero error associated with it. Let X be the relevant ℓ × ∆ submatrix of R where we have ∆ < ℓ now, and recall that all entries of X are i.i.d. N (0 , 1) random variables. The squared error corresponding to B ( i ) can be expressed as ∑ ℓ -1 i =0 || XX † ⃗ e i -⃗ e i || 2 2 ( X † denotes the pseudo-inverse [31]). Let X = USV T denote the SVD of X , where U and V are orthogonal matrices of dimension ℓ × ℓ and ∆ × ∆ respectively and S = [ D | 0] T where D is a ∆ × ∆ matrix with positive entries on the diagonal, and 0 represents a ∆ × ( ℓ -∆) matrix of zeros. Then, X † = V [ D -1 | 0] U T . It is well known ([33], Remark 5.2.8) that for a matrix with i.i.d. N (0 , 1) entries, the singular vectors are uniformly distributed on the unit-sphere. Therefore, the expected squared error becomes

The last step above follows since each u j is uniformly distributed over the sphere of dimension ℓ . Therefore, we have the E [ u 2 j, 0 ] = 1 /ℓ since || u j || 2 2 = 1 and each u j,k , k = 0 , . . . , ℓ -1 is identically distributed. We conclude that if D i appears ∆ i ≤ ℓ -1 times, then its error contribution is ℓ -∆ i and the overall error is therefore ∑ N i =1 max(0 , ℓ -∆ i ) .

Complexity analysis: The time complexity of each least-squares problem is O (∆ 2 ℓ ) [34] and the i -th worker solves at most ℓδ i of them independently and in parallel. The marginal cost of this calculation as against the calculation of the actual gradients will be very small in most practical settings.

2.2 Illustrative Example

Consider Fig. 1b (from [21]) where the dataset consists of chunks D 1 , . . . , D 5 and are assigned in a non-uniform fashion to workers W 1 , . . . , W 5 . Suppose that the PS wants the exact gradient with ℓ = 2 . As two copies of each chunk have been processed, the PS broadcasts the encode-and-transmit signal and the vector ψ = [5 2 0 2 3] to all the workers which indicates, e.g., that W 1 has processed all its chunks, W 3 is failed etc. The matrix B of indeterminates for this example and the corresponding B ( i ) 's, turn out to be

In this example, W 5 has processed D 1 , D 4 , D 5 so that ˜ B 5 = [ B (1) B (4) B (5) ] . Note that the PS requires the vectors [ ⃗ e T 0 ⃗ e T 0 ⃗ e T 0 ⃗ e T 0 ⃗ e T 0 ] and [ ⃗ e T 1 ⃗ e T 1 ⃗ e T 1 ⃗ e T 1 ⃗ e T 1 ] to lie in the row-space of B so that it

D

1

D

2

D

3

D

4

D

4

D

5

D

1

D

6

D

2

D

5

D

3

D

6

W

1

W

2

W

3

W

4

Q

5

= 10

(a)

D

1

D

2

D

3

D

4

D

5

D

4

D

6

D

1

D

2

D

5

D

3

D

6

W

1

W

2

W

3

W

4

(b)

Figure 7: Figure 7

Figure 6: Figure 6

3 can recover ∑ 5 i =1 g D i [ k ] , k = 0 , 1 from the encoded gradients. Towards this end, e.g., W 5 solves (5) in Algorithm 1 with the matrix ˜ B 5 , i.e.,

D 1 D 2 D 3 D 4 D 4 D 1 D 2 D 3 Q where the decision variables are a 1 , a 2 , a 7 , a 8 , a 9 , a 10 , b 1 , b 2 and d 1 , d 2 , d 7 , d 8 , d 9 , d 10 . For instance, corresponding to the first column of B (1) , W 5 's problem becomes determining the minimum ℓ 2 -norm solution of the under-determined least-squares problem

We emphasize that the same minimization will be independently performed at workers W 1 and W 2 as well. After each W j determines its encoding coefficients and transmits g j for j = 1 , . . . , 5 , the PS can easily recover ∑ 5 j =1 g D j [ k ] = ∑ 5 i =1 r kj g j for k = 0 , 1 .

3 Algorithm 1 works as is, even in the case when the PS is interested in an approximate gradient. For instance, suppose that the PS sends the encode-and-transmit signal when the vector ψ = [4 2 0 2 3] , so that only one copy of D 5 has been processed in the cluster. The corresponding encoding coefficients can still be computed using Algorithm 1. The only difference will be that the relevant indeterminate matrix becomes

This means, e.g., when W 5 is trying to find d ′ 9 , then it will be solving an over-determined least-squares problem: ∣ ∣ ∣ ∣ ∣ ∣ ∣ ∣ [ r 05 r ′ 05 ] d ′ 9 -[ 1 0 ]∣ ∣ ∣ ∣ ∣ ∣ ∣ ∣ 2 . It is evident, that all the workers can still agree on the same solution.

3 Chunk Ordering Algorithm

Note that the assignment matrix only specifies the assignment of chunks to workers but says nothing about the relative order of chunks within a worker. When taking into account partial stragglers, the relative ordering of the D i 's within a worker is crucial. Therefore, an important question when leveraging partial stragglers within GC is one of how to determine this chunk ordering within workers for a given assignment matrix. This will in general depend on models of worker speeds. However, it is a difficult task to obtain accurate models on the worker speeds within a cloud cluster as conditions can change dynamically.

We work instead with a combinatorial metric that depends on the total number of chunks that the cluster has to process in the worst case, such that at least a single copy of each D i is processed; this was also used in [35, 36] in a coded matrix computation context. This metric minimizes the worst case number of chunks that the cluster needs to process in the case when ℓ = 1 . In particular, for a given D i , let Q i denote the maximum number of chunks that can be processed by the cluster such that none of the copies of D i are processed (see Figs. 2a and 2b) and let Q max = max i =1 ,...,N Q i . Thus, 1 + Q max is a metric that quantifies the worst-case number of chunks that need to be processed

Q

5

5

= 9

= 10

Algorithm 2 Chunk-Ordering

Input: Assignment matrix A such that N = m and γ i = δ i = δ for all i ∈ [ m ] Output: An optimal ordering matrix O .

̸

2: for i ranges from 1 to δ do

4: Let ˜ A ( i +1) := ˜ A ( i ) -P i and δ ( i +1) = δ ( i ) -1 .

3: Apply Claim 1 with ˜ A ( i ) and δ ( i ) and solve the max-bipartite matching problem in Claim 1. Let the permutation matrix be P i .

5: end for

6: Set O = ∑ i ∈ [ δ ] iP i .

before at least a single copy of every D i is guaranteed to be processed. Here, the worst-case is over the speeds of the different workers. We say that an assignment is optimal if it achieves the lowest possible Q max for a given assignment matrix. Indeed, Figs. 2a and 2b show two different orderings for the same assignment matrix with different values of Q max .

̸

From the point of view of leveraging partial stragglers, for exact GC it can be shown that the ordering imposed by the cyclic assignment scheme (in [5]) is optimal (( cf. Remark 1 in work [35])) for this Q max metric. However, for approximate GC, when the number of stragglers can be much higher, other assignments, e.g., those based on regular graphs perform better [6, 37]. In particular, in these cases, the assignment matrix A is chosen as the adjacency matrix of the relevant graphs and is such that N = m and γ i = δ i = δ for i ∈ [ m ] . For such assignment matrices, Algorithm 2 presents an optimal algorithm for determining the ordering of the chunks within each worker in Q max -metric. Corresponding to the assignment matrix A , we can associate an ordering matrix O which is such that O i,j = 0 if A i,j = 0 and O i,j ∈ [ δ ] otherwise. Thus, if O i,j = α = 0 , it means that D i is the α -th chunk in W j , e.g., α = 1 implies that the chunk is at the top and α = δ means that it is at the bottom. Let Q i ( O ) denote the maximum number of chunks that can be processed across the cluster, such that no copy of D i has been processed. Then, upon inspection, we can see that

̸

̸

where the notation 1 Y denotes the indicator of Y . Next, let Q max ( O ) = max i ∈ [ m ] Q i ( O ) . Thus, given an assignment matrix A , our goal is to find an ordering matrix, such that Q max ( O ) is minimized. As ( m -δ -1) δ is a constant, this optimization is equivalent to the following min-max problem.

Each column of O has exactly δ non-zero entries, with each value in [ δ ] appearing exactly once. Counting the sum of the entries in O two different ways yields

It turns out that this bound is in fact achievable. Let ˜ A = ˜ A (1) (defined in Algorithm 2). Define G ( ˜ A ) = ( V, E ) as a bipartite graph on vertex set V = X ∪ Y such that | X | = | Y | = m,X ∩ Y = ∅ and deg ( v ) = δ for all v ∈ X ∪ Y . For u ∈ X,v ∈ Y , we have that ( u, v ) ∈ E if and only if ˜ A u,v = 1 . Thus, ˜ A is in one-to-one correspondence with G ( ˜ A ) . Algorithm 2 decomposes G ( ˜ A ) into a collection of disjoint perfect matchings [38] which are then assigned values in [ δ ] . This gives us the required ordering.

Claim 1. Let ˜ A ∈ { 0 , 1 } m × m be such that both i -th row sum and j -th column sum of ˜ A are δ for all i, j ∈ [ m ] . Then there exists a permutation matrix P ∈ { 0 , 1 } m × m such that ˜ A -P ∈ { 0 , 1 } m × m and both i -th row sum and j -th column sum of ˜ A -P are δ -1 for i, j ∈ [ m ] .

Proof. We claim that G ( ˜ A ) has a X -perfect matching. Let S ⊆ X be arbitrary and N ( S ) ⊆ Y denote its neighborhood. Let κ denote the average degree of the vertices in N ( S ) in the subgraph

(a)

(b)

100

Graph Assign.: Orig. GC

14

Graph Assign.: Partial GC

10-2

Cyclic Assign.: Orig.GC

Cyclic Assign.: Partial GC

10

-4

Partial GC Est. (f=1)

Time

12

Partial GC Est. (f=2)

E

10-6

王

PartialGCEst.(f=3)

Orig.GC

Completion T

10

10-8

Partial GC (f =1)

8

Partial GC(f=2)

10-10

Partial GC(l=3)

6

10-12

工

10-14

5

10

15

20

25

1

2

3

T

induced by S ∪ N ( S ) . Then, we have

The first inequality above follows because the degree of each node in N ( S ) in G ( ˜ A ) is δ . Thus, Hall's condition [39] holds and the claim follows. Since | X | = | Y | , this is actually a perfect matching M . Then, this perfect matching M gives the desired P : P u,v = 1 if ( u, v ) ∈ M and P u,v = 0 if ( u, v ) / ∈ M . Removing the matching M from G ( ˜ A ) results in a bi-regular bipartite graph with degree δ -1 which corresponds to ˜ A -P .

Remark 4. The proposed algorithm above finds the optimal ordering when considering the case of ℓ = 1 with N = m (number of chunks equal to number of workers); it only applies when N = m . While this algorithm is expected to have better performance than a randomly chosen ordering in the case of higher ℓ , we do not have an optimal construction in this case.

Algorithm 2 Analysis: The algorithm gives δ permutation matrices { P i } i ∈ [ δ ] such that ˜ A = ∑ δ i =1 P i , i.e., the set of non-zero entries of the P i 's are disjoint. Since O = ∑ i ∈ [ δ ] i · P i , this implies that each column has exactly δ non-zero entries such that these entries consists of elements of [ δ ] . Therefore, O is an ordering matrix. In addition, each row has δ non-zero elements from [ δ ] , so that all row sums and Q max ( O ) equal δ ( δ +1) 2 . A maximum matching can be in time O ( m 2 δ ) by converting it to a max-flow problem [38], so our overall complexity is O ( m 2 δ 2 ) . The complexity can potentially be reduced further by using more efficient max-flow algorithms.

4 Numerical Experiments and Comparisons

Our proposed protocol in Section 2, utilizes additional communication between the PS and the workers. We note here that ideas in [21] that are based on Lagrange interpolation can potentially be adapted to arrive at an exact GC protocol within our setting. However, the numerical instability of Lagrange interpolation is a serious issue with this solution, since it can be shown that the degree of the polynomial to be interpolated is at least m -ℓ . Even for values such as ℓ = 2 and m = 22 , 27 , 32 , the error in Lagrange interpolation is too high for the technique to be useful (see Appendix A).

In what follows, we present comparisons with the original GC protocol via a simulation of the distributed system for both the exact and approximate GC scenarios. All software code for recreating these results can be found at [40]. These simulations were performed within a MacBook Pro (M1 chip, 16 GB RAM). In both scenarios, we simulated node failures and slow-downs. We generated a random vector of lengthm where α workers uniformly at random are chosen to be failed ( α is chosen based on the scenario). For the other workers, the amount of time taken to process a chunk is

chosen i.i.d. from an exponential distribution with parameter µ = 1 . The entries of the matrix R are chosen i.i.d. from the standard normal N (0 , 1) .

Approximate GC simulation: We considered two different random regular graphs, G i , i = 1 , 2 with sizes m = 200 , 300 and degree ν = 8 . These graphs have second-largest absolute value of eigenvalues: 5.14 and 5.23, i.e., less than 2 √ ν -1 so they can be considered as Ramanujan graphs [41] (these were used in [6]). Their adjacency matrices were used as the assignment matrix. The number of failures α was set to ν -1 .

For the original GC protocol with arbitrary assignment matrices, [21] provides a communicationefficient approximate GC method that relies on rational interpolation. However, even for m = 100 , these will result in very complex basis functions. Furthermore, there are no numerical results in [21] (or posted software) and the approximation guarantees depend on the form of the function to be interpolated, i.e., the guarantees cannot be expressed in terms of the system parameters. Thus, in our comparison, we only consider the original GC protocol with ℓ = 1 .

For the original GC protocol, we compute the least squares solution ˆ r for minimizing || Ar -1 || 2 2 . Here, ˆ r such that ˆ r i = 0 if W i has not completed processing all its chunks at time T . For our proposed protocol, we leverage partial work completed by T , as described in Section 2. The overall error is computed as the sum of the errors for the least-squares solution for each z T i , i = 0 , . . . , ℓ -1 . Each data point on the curves was chosen by performing 1000 simulations of failures and slow-downs. We have also plotted the expected value of the error, derived in (6).

Fig. 3a shows the mean-squared-error (MSE) comparing the original GC approach with ℓ = 1 and our partial GC approach for ℓ = 1 , 2 , 3 for graph G 1 (see Appendix B for G 2 results). As can be observed, the MSE for our approach is several orders of magnitude lower with increasing T . Crucially, our estimate (6) closely tracks the behavior of the error of our method. Thus, it can easily be used by the PS as a way to decide when to send the encode-and-transmit message. We emphasize that our approach, even with ℓ ≥ 2 actually has a lower MSE than the original approach (that operates with ℓ = 1 ). Note that with ℓ ≥ 2 , we will enjoy a lower communication time in a real-world distributed cluster. However, as we are working with a simulation of the distributed system, at this time we do not have precise figures for the reduction in communication time on actual clusters.

In Fig. 6 in Appendix B, we compare the performance of our approach under the optimal ordering ( cf. Section 3) and an appropriately chosen random ordering. The random ordering is picked as follows. We generated 100 independent random orderings and selected the one with the best (smallest) Q max ( O ) . Our optimal ordering, which can be found efficiently, clearly has a better performance.

Exact GC simulation: We considered (i) the cyclic assignment [5] with N = m = 200 and δ = 8 , and (ii) the assignment based on graph G 1 discussed above. The number of failures α in the simulations is set to δ -ℓ so that exact gradient reconstruction is possible. For both approaches, we determined the time T such that each chunk is processed at least ℓ times across the cluster (for original GC we only consider workers that have processed all their chunks). These values were averaged over 1000 runs for each value of ℓ . In Fig. 3b we clearly observe that the exact gradient can be computed using our method is approximately half the time as compared to the original GC protocol. We note that the average completion time for the original GC protocol ( G 1 -based assignment) for ℓ = 1 is about T = 6 time units. However, the MSE for the original GC protocol in the approximate scenario continues to be high at T = 24 . The reason is that there are ν -1 failures that are introduced in the simulation. The approximate GC recovery operates by solving a least-squares problem for the fixed G 1 -based assignment. Thus, the MSE does not necessarily drop to zero even if one copy of each chunk has been processed in the cluster. However, there are exact recovery algorithms that one can use in this case. We note here that when ℓ ≥ 2 , the known techniques for exact recovery for the original GC protocol are based on Lagrange interpolation and are numerically unstable. Thus, the gradient recovered using the original approach will in general not be useful.

Fig. 7 in Appendix B compares the completion times of random vs. optimal ordering; the optimal ordering is clearly better.

5 Limitations of our work

Our chunk ordering algorithm (Section 3) is optimal only in the case when the assignment matrix has N = m and γ i = δ i = δ for i ∈ [ m ] . While several well known gradient coding schemes,

especially those from regular graphs, satisfy this property, there are others that do not. Our results in Section 4 with ℓ ≥ 2 are simulations of the actual distributed cluster and indicate lower MSE than the original GC protocol. However, we do not have actual cloud platform statistics on the reduction in communication time within our method. We do however expect this reduction to be quite significant.

6 Conclusions

We presented a novel gradient coding protocol that leverages the work performed by partial stragglers. Our protocol is simultaneously computation-efficient and communication-efficient, and numerically stable. Furthermore, we present rigorous analyses of the protocol's correctness and performance guarantees. Our protocol is one of the first to provide a satisfactory solution to the problem of communication-efficient, approximate gradient coding for general assignment matrices.

Acknowledgments and Disclosure of Funding

We would like to the anonymous reviewers and area chair whose helpful comments and suggestions helped improve the paper. This work was supported in part by the National Science Foundation (NSF) under grant CCF-2115200.

References

Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, Cambridge, MA, USA, 2016. http://www.deeplearningbook.org .

Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth Annual ACM Symposium on Theory of Computing (STOC) , pages 665-674, 2013.

Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on operating systems design and implementation (OSDI 14) , pages 583-598, 2014.

Google Spot Virtual Machines.

Rashish Tandon, Qi Lei, Alexandros G. Dimakis, and Nikos Karampatziakis. Gradient coding: Avoiding stragglers in distributed learning. In Intl. Conf. Mach. Learn. (ICML) , pages 33683376, August 2017.

Netanel Raviv, Rashish Tandon, Alex Dimakis, and Itzhak Tamo. Gradient coding from cyclic MDS codes and expander graphs. In Intl. Conf. Mach. Learn. (ICML) , pages 4302-4310, July 2018.

Wael Halbawi, Navid Azizan, Fariborz Salehi, and Babak Hassibi. Improving distributed gradient descent using reed-solomon codes. In IEEE Intl. Symp. on Info. Th. , pages 2027-2031, 2018.

Tianyi Chen, Georgios Giannakis, Tao Sun, and Wotao Yin. Lag: Lazily aggregated gradient for communication-efficient distributed learning. Adv. in Neural Inf. Process. Syst. (NeurIPS) , 31, 2018.

Shanuja Sasi, V. Lalitha, Vaneet Aggarwal, and B. Sundar Rajan. Straggler mitigation with tiered gradient codes. IEEE Trans. on Comm. , 68(8):4632-4647, 2020.

Baturalp Buyukates, Emre Ozfatura, Sennur Ulukus, and Deniz Gündüz. Gradient coding with dynamic clustering for straggler-tolerant distributed learning. IEEE Trans. on Comm. , 71(6):3317-3332, 2023.

Neophytos Charalambides, Hessam Mahdavifar, and Alfred O. Hero. Numerically stable binary gradient coding. In IEEE Intl. Symp. on Info. Th. , pages 2622-2627, 2020.

Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, May 2018.

Zachary Charles, Dimitris Papailiopoulos, and Jordan Ellenberg. Approximate gradient coding via sparse random graphs. 2017 [Online] arxiv:1711.06771.

Margalit Glasgow and Mary Wootters. Approximate gradient coding with optimal decoding. IEEE J. Select. Areas Info. Th. , 2(3):855-866, 2021.

Swanand Kadhe, O. Ozan Koyluoglu, and Kannan Ramchandran. Gradient coding based on block designs for mitigating adversarial stragglers. In IEEE Intl. Symp. on Info. Th. , pages 2813-2817, 2019.

Animesh Sakorikar and Lele Wang. Soft bibd and product gradient codes. IEEE J. Select. Areas Info. Th. , 3(2):229-240, 2022.

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. Adv. in Neural Inf. Process. Syst. (NeurIPS) , 30, 2017.

Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In Intl. Conf. Mach. Learn. (ICML) , pages 36058-36076, 2023.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, July 2019.

Min Ye and Emmanuel Abbe. Communication-computation efficient gradient coding. In Intl. Conf. Mach. Learn. (ICML) , pages 5610-5619, 2018.

Tayyebeh Jahani-Nezhad and Mohammad Ali Maddah-Ali. Optimal communicationcomputation trade-off in heterogeneous gradient coding. IEEE J. Select. Areas Info. Th. , 2(3):1002-1011, 2021.

V. Pan. How Bad Are Vandermonde Matrices? SIAM Jour. on Mat. Analysis and Appl. , 37(2):676-694, 2016.

Swanand Kadhe, O. Ozan Koyluoglu, and Kannan Ramchandran. Communication-efficient gradient coding for straggler mitigation in distributed learning. In IEEE Intl. Symp. on Info. Th. , pages 2634-2639, 2020.

Lev Tauz and Lara Dolecek. Multi-message gradient coding for utilizing non-persistent stragglers. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers , pages 2154-2159, 2019.

Hankun Cao, Qifa Yan, Xiaohu Tang, and Guojun Han. Adaptive gradient coding. IEEE/ACM Trans. on Networking , 30(2):717-734, 2022.

Haozhao Wang, Song Guo, Bin Tang, Ruixuan Li, Yutong Yang, Zhihao Qu, and Yi Wang. Heterogeneity-aware gradient coding for tolerating and leveraging stragglers. IEEE Trans. on Comp. , 71(4):779-794, 2022.

Qi Wang, Ying Cui, Chenglin Li, Junni Zou, and Hongkai Xiong. Optimization-based block coordinate gradient coding. In 2021 IEEE Global Communications Conference (GLOBECOM) , 2021.

Nuwan Ferdinand, Benjamin Gharachorloo, and Stark C. Draper. Anytime exploitation of stragglers in synchronous stochastic gradient descent. In IEEE Intl. Conf. on Mach. Learning and Appl. (ICMLA) , pages 141-146, 2017.

S. Lin and D. J. Costello. Error Control Coding, 2nd Ed. Prentice Hall, Upper Saddle River, 2004.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing) . Wiley-Interscience, 2006.

Roger A. Horn and Charles R. Johnson. Matrix Analysis . Cambridge University Press, 2012.

Zizhong Chen and Jack J. Dongarra. Condition numbers of gaussian random matrices. SIAM Journal on Matrix Analysis and Applications , 27(3):603-620, 2005.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge University Press, 2018.

S. Boyd and L. Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

Anindya Bijoy Das, Li Tang, and Aditya Ramamoorthy. C 3 LES : Codes for coded computation that leverage stragglers. In IEEE Info. Th. Workshop , pages 1-5, 2018.

Anindya Bijoy Das and Aditya Ramamoorthy. Coded sparse matrix computation schemes that leverage partial stragglers. IEEE Trans. on Info. Th. , 68(6):4156-4181, 2022.

Zachary Charles and Dimitris Papailiopoulos. Gradient coding via the stochastic block model, 2018 [Online] arxiv:1805.10378 [stat.ML].

J. Kleinberg and E. Tardos. Algorithm Design . Pearson, 2005.

J. H. Van Lint and R. M. Wilson. A Course in Combinatorics . Cambridge University Press, New York, 2001.

Aditya Ramamoorthy, Ruoyu Meng, and Vrinda S. Girimaji. Leveraging partial stragglers within gradient coding - software repository. https://github.com/flamethrower775/ Leveraging-partial-stragglers-within-gradient-coding , 2024.

F. R. K. Chung. Spectral Graph Theory . American Mathematical Society, Rhode Island, 1997.

Appendix A Numerical Instability of Lagrange Interpolation.

Even for values such as ℓ = 2 and m = 22 , 27 , 32 , the error in Lagrange interpolation is too high for the technique of [21] to be useful. To see this consider Fig. 4 which shows the error (averaged over 100 random trials) in first interpolating and then evaluating the interpolated Lagrange polynomial (degrees 20, 25 and 30) at specific points (as is done in [21]); the x -axis is the precision. It can be observed that the error even with full-precision is too high for the technique to be useful.

Appendix B Additional numerical experiments

Fig. 5 shows the mean-squared-error (MSE) comparing the original GC approach with ℓ = 1 and our partial GC approach for ℓ = 1 , 2 , 3 for graph G 2 with m = 300 vertices. The results are similar in spirit to the results for the case of G 1 that has 200 vertices. Namely, our MSE is orders of magnitude lower than the original GC approach, even when we consider ℓ ≥ 2 .

In Figs. 6a and 6b we study the impact of chunk ordering within the workers for the assignment matrices corresponding to graphs G 1 and G 2 respectively. Each data point on the curves corresponds to 1000 simulations (setup described in Section 4). In particular, in Fig. 6a corresponding to the case of ℓ = 1 , 2 , 3 for G 1 , we observe that the MSE of the optimal ordering consistently remains lower than the MSE of the random ordering and can in fact be multiple orders of magnitude lower when the

Lagrange Interpolation Error

106

Reconstruction Error

104

102

100

2

4

6

8

Full Precision

No. of decimal places in evaluation

100

10-2

Partial GC Est. (=1)

10

-4

Partial GC Est.(=2)

E

10-6

Partial GC Est.(f=3)

15

Orig.GC

M

10-8

Partial GC (=1)

Partial GC (l=2)

10-10

Partial GC(=3)

10-12

10-14

5

10

15

20

25

T

(a)

(b)

王士士重王

Opt. Ord. (l=1)

壬王壬壬重王

Opt.Ord. (l=1)

100

100

Rand. Ord. (f =1)

Rand. Ord. (f = 1)

10-2

Opt.Ord.(l=2)

10-2

Opt.Ord.(f=2)

Rand. Ord. (f =2)

Rand. Ord. ( =2)

10-4

Opt.Ord.(l=3)

10-4

Opt.Ord.(l=3)

Rand.Ord. (f=3)

Rand. Ord. (f =3)

E

10-6

E

10-6

10-8

10-8

10-10

10-10

10-12

10-12

10-14

10-14

5

10

15

20

25

5

10

15

20

25

T

T

encode-and-transmit signal is sent at certain time intervals. A similar pattern can be observed in Fig. 6b, which shows the case of ℓ = 1 , 2 , 3 and G 2 .

Fig. 7 shows the results of a similar experiment comparing the random chunk ordering and our optimal chunk ordering in terms of completion time for exact GC. The assignment matrix corresponds to the graph G 1 discussed in Section 4. The random ordering is chosen by sampling 100 independent random orderings and selecting one with the smallest Q max ( O ) . A data point is then generated by running 1000 simulations with the selected random ordering. The results indicate that in an average sense, the completion time of the optimal ordering is clearly lower.

9

Graph Assign.: Opt. Ord.

Graph Assign.: Rand. Ord.

8

Completion Time

7

6

5

4

3

2

1

2

3

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We present a new gradient coding algorithm. We have discussed the assumptions underlying the algorithm and outlined the main contributions of our work. The contributions are substantiated within the body of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a section titled Limitations in the paper that discusses these issues. Computational complexity has been discussed for all our algorithms.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper presents protocols and algorithms that are relevant to gradient coding. Within the main body of the paper, we have included analyses of these protocols and algorithms. The analyses include proof of correctness, performance analysis and computational complexity analysis.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided complete descriptions of all our protocols and algorithms. In addition, we also have an example of the main protocol that guides the reader through the main steps.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have uploaded commented code that recreates the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our protocols and algorithms do not have any hyperparameters that need to be tuned. All experimental settings and details are available in the body of the paper itself.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have included error bars for the relevant plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This information is provided on the section on Numerical Experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes. We have read the Code of Ethics and our submission conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: There are no negative social impacts of our work. Our work allows for faster distributed training on cloud platforms. The positive societal impact is that it will save computational cycles on such platforms.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not have such risks because the paper is about distributed training.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use any existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We are not releasing any new assets in this work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work is not related to crowdsourcing nor research with human subject Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn distributed learning, workers in large scale clusters may be slower than expected promised or are prone to failure. Gradient coding tries to address this by introducing redundancy within the assignment of chunks (of data) to workers. There, ideas from coding theory are used to recover the full gradient. \n\nExisting work is either not efficient or rejects \"failed\" workers, leading to suboptimal performance. In this work, the authors present a new gradient coding protocol, leveraging the work performed by partial stragglers. The approach offers redundancy in assignment of chunks, leading to worker calculating linear combination of the gradients of the chunks. Further, the workers know the assignment matrix and ordering. \n\nThe authors claim that the Algorithm is efficient in terms of compute and communication and numerically stable. Specifically, they claim that the algorithm is around 2x faster in the original setting and further, in the approximate gradient reconstruction setting, MSE is several orders of magnitude better.\n\nSTRENGTHS:\n- The proposed GC idea seems to be novel, particularly the distributed gradient encoding. \n- The math, as far as i checked is correct. \n- The example in Section 2.2 was helpful and improved readability. \n- Efficient and robust training in a distributed environment is becoming more and more important topic.\n\nWEAKNESSES:\n- The writing was not easy to follow. The example was helpful though. \n- To me, the biggest weakness of the work is its limited comparison to baselines. It is unclear to me, how the proposed algorithm compares computationally with a simple baseline, where workers share gradients for data chunks with the PS and if a data sample does not get processed, the PS sends it to a different worker. \n- It is unclear in which regime the approach offers benefits: The error rates of workers do not seem to be taken into account. This is strange to me, as an error rate of 0 would render the approach certainly not optimal. \n- Various aspects of the work remain unclear (see Questions). \n\nMinor:\n- Lines 46 and following: $nnz(v)$ is commonly typesettes (as a function) as $\\text{nnz}(v)$. Similarly for later occurances.\n\nQUESTIONS:\n- The redundancy leads to some overhead. What if one would not do this and redistribute data again that was not used in the previous iteration? \n- How does your work compare performance wise with simple baselines, like the PS distributes the chunks of data to different workers and if some don't report back, their chunks get redistributed again? How do the methods compare regarding memory, compute and communication?\n- I am surprised that the failure rate of nodes is taken into account when analyzing the algorithm. Experiments with varying failure rates and varying $\\ell$ would be good to compare different approaches. Specifically, under which regime is this Algorithm better than related work? What failure rates does one need to assume for this algorithm to be better? \n- The authors mention that variable spot instance pricing is an application. Can the authors provide a back of the envelope calculation how much more compute one would need compared to a standard setting and high the difference in spot prices would need to be in order for the method to be economically viable? \n- Just a clarification: The vector $\\psi$ does not only have the counts of how many data chunks a worker processed but which data chunks each worker did, yes? \n- It is unclear to me how data chunks are treated that where processed by more than $\\ell$ workers as this would then result in a $B^{(j)}$ that has more than $\\ell$ columns. \n- How would the protocol handle a case where a worker during or after the gradients get encoded becomes unresponsive?\n\nLIMITATIONS:\nWhile the authors provide a limitations section, cover potential limitations seen by me (see Questions and Weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a coded distributed computation solution when the parameter server needs to compute the gradient (exactly or approximately) over a very large dataset. The idea relies on distribution dataset into chunks and assigning chunks to different workers, such that each worker computes the gradient for only a subset of the chunks. The workers then send their computations to the server in a coded manner to reduce the communication load. In principle, redundant computations are introduced to compensate for slow straggling workers.\n\nSTRENGTHS:\nThe authors propose a new class of algorithms that benefits from additional communication between the server and the workers, where the worker tracks the chunks computed by the workers at each iteration, and informs all the workers about the state of others. The proposed protocol allows also partial computation of the gradients, which can be used to speed up optimisation steps at the expense of reduced accuracy, similarly to stochastic gradient descent. The authors provide numerical examples that illustrate the benefits of their proposed algorithm.\n\nWEAKNESSES:\n- One of the main weaknesses of the proposed algorithm is the additional communication overhead. The protocol requires communication from each worker to the server and from the server to each worker after each iteration. Even though these are packets of limited size, only containing information about the state of computation, not the computation results, they do still need to be communicated back and forth, and the algorithm relies on the correct reception of these packets by their recipients. In practice, this may cause significant load on the system particularly when the number of workers is large. \n\n- The proposed algorithm requires solving another optimization problem to find the right combining coefficients when the server requests the results from the workers. Each worker needs to solve a problem not only for its own coefficients, but also the other workers' problems to be able to track the coded solutions transmitted by them. This puts quite a bit of an extra computation load (probably much less then what is being computed for the gradient computation, but still an additional computation step). \n\n- Overall the presentation of the algorithm could be a bit more clear. It was difficult to follow the description. I believe it would be better to provide the illustrative example before presenting the details of the algorithm, and walk the reader through the steps of the proposed algorithm and highlight the benefits. \n\n- Some relevant literature is not considered.\n\nQUESTIONS:\n- The authors claim that no previous work considered approximate gradient computations. This is not exactly correct, as the following work makes almost the same claim:\n\nE. Ozfatura, S. Ulukus and D. Gündüz, \"Coded Distributed Computing With Partial Recovery,\" in IEEE Transactions on Information Theory, vol. 68, no. 3, pp. 1945-1959, March 2022, doi: 10.1109/TIT.2021.3133791.\n\nPlease see Section V of this paper that applies the proposed ideas to gradient coding, which is called coded communication (since computation results are coded before being communicated to the server).\n\nThe authors should revise their claims in light of the above work, clearly explain how/why their scheme is superior to the one proposed here, and also provide numerical comparisons.\n\nLIMITATIONS:\n- As noted above, one of the main limitation is the lack of comparison with other existing schemes in the literature. Also, one would like to see the results of the algorithm implemented on a more realistic scenario.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the gradient coding (GC) framework for distributed learning systems to address the issue of stragglers. Compared to the standard GC protocol, the authors propose new approaches that utilize the partial work done by partial stragglers. These protocols aim to enhance computation and communication efficiency while providing guarantees of numerical stability.\n\nSTRENGTHS:\nThe proposed protocols leverage partial stragglers, an aspect often ignored in traditional gradient coding methods. If the proposed computing design does not require a significantly stronger model from the communication perspective, it can potentially speed up the training process.\n\nWEAKNESSES:\n1. The issue of this work is the mismatch between the proposed computation design and the formulation. The idea used in the purpose framework is based on the concept of forward error correction, which means when redundancy is injected into the computation ahead of time in a certain way, the central node can retrieve the full gradient without communicating back to the individual workers. Such a communication framework is consistent with the earlier gradient coding paper, where the computation is one-shot. But in this work, the authors assume that feedback is easy and \"at regular intervals, the workers keep communicating to the PS, the number of chunks they have processed\". In this case, it would be better to coordinate the workers to send partial gradients from non-overlapping datasets, which gives ideal numerical stability as well. \n\n2. The overall presentation can be improved for readability. For example, currently, there is not a dedicated section to introduce the system formulation, and it is difficult for readers to find definitions for concepts such as $d$, which is introduced in the related work section. \n\n3. Algorithm 1 is not self-contained. In its current form, it would be better to revise it or describe it in conjunction with other detailed steps.\n\nQUESTIONS:\n1. The authors may want to justify why, when coordination is allowed, and the encoding is decided after knowing the straggler pattern, there are still benefits of encoding the results rather than directly feeding uncoded sums of non-overlapping partial gradients to the central node (e.g., in computation cost, communication, numerical performances).\n\n2. Remark 2 suggests that the communication cost only takes into account the overall load. If we take into account the rounds of communication, its cost could potentially be significantly larger than the classical gradient coding.\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses inefficiencies in gradient coding protocols for distributed learning systems, specifically targeting the issue of stragglers—workers that are slower than expected or fail to complete their tasks. The authors introduce novel gradient coding protocols that incorporate the partial work performed by \"partial stragglers\". The proposed protocols are designed to be both computation and communication efficient while remaining numerically stable.  \n\nFor exact gradient reconstruction, the new protocol is approximately a constant factor faster than existing protocols, and for approximate gradient reconstruction, the mean-squared-error (MSE) of the reconstructed gradient is significantly reduced. Additionally, the paper presents efficient algorithms for optimizing the order of data chunks within workers, enhancing further the performance of the gradient coding protocol. The authors provide theoretical analysis of the proposed algorithms as well as numerical experiments that demonstrate significant improvements in MSE and execution time compared to traditional gradient coding methods.\n\nSTRENGTHS:\n- The authors study a relevant problem of interest for the ML community.\n- The paper is well-structured and clearly explains the problem, the proposed solutions. It is easy to read and understand its main points.\n- The paper introduces innovative gradient coding protocols that leverage the partial contributions of slow workers, addressing a gap in existing methods that typically ignore these partial stragglers. Additionally, the approach of optimizing the order of data chunks within workers is a novel contribution that enhances the protocol’s efficiency.\n- The proposed protocols are theoretically analyzed for their computational and communication efficiency, as well as their numerical stability. The theoretical analysis is further supported by empirical results.\n- The protocols offer substantial improvements in gradient reconstruction performance, reducing mean-squared-error by several orders of magnitude and halving the execution time for inexact gradient reconstruction compared to existing methods. Further, the minimal additional communication introduced by the protocols ensures they have practical real-world applications.\n\nWEAKNESSES:\n- Although the paper is introducing a novel approach -as also mentioned by the authors - a part of this work builds heavily on existing works (long literature on GC as well as [34,35, 37,38]) which limits somewhat the novelty of this work.\n- The focus of this work is mostly theoretical without providing extensive experiments. Importantly, actual cloud platform statistics on the communication time reduction achieved by the proposed method are missing. This potentially limits the practicality of the proposed methods.\n\nQUESTIONS:\n- A major contribution of this work revolves around ensuring numerical stability. Could the authors elaborate more on the the measures taken to ensure the numerical stability of their protocols? \n- You mention that the additional communication cost is minimal compared to the length of the parameter vector. It would be very interesting to capture the effects of this overhead on experiments either in simulations or in real-world experiments. Although it is not necessary I think that the inclusion of such experiments would strengthen this paper.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-08-01"}