Title: DropCompute: simple and more robust  distributed synchronous training via compute variance reduction
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributed optimization

large-scale parallel SGD

synchronous training
Abstract: Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-Reduce), but these require waiting for all workers in each step. Thus, these methods are limited by the delays caused by straggling workers.
Results: We study a typical scenario in which workers are straggling due to variability in compute time. We find an analytical relation between compute time properties and scalability limitations, caused by such straggling workers. With these findings, we propose a simple yet effective decentralized method to reduce the variation among workers and thus improve the robustness of synchronous training. This method can be integrated with the widely used All-Reduce. Our findings are validated on large-scale training tasks using 200 Gaudi Accelerators.

DropCompute: simple and more robust distributed synchronous training via compute variance reduction

Abstract

Background. Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-Reduce ), but these require waiting for all workers in each step. Thus, these methods are limited by the delays caused by straggling workers. Results. We study a typical scenario in which workers are straggling due to variability in compute time. We find an analytical relation between compute time properties and scalability limitations, caused by such straggling workers. With these findings, we propose a simple yet effective decentralized method to reduce the variation among workers and thus improve the robustness of synchronous training. This method can be integrated with the widely used All-Reduce . Our findings are validated on large-scale training tasks using 200 Gaudi Accelerators. A reference implementation 2 is provided.

1 Introduction

Deep Neural Networks (DNNs) training continues to scale over size and computational footprint, as a result of a higher number of trainable parameters, wider and deeper models, and growing amounts of training data. As improvements in model quality (measured by test loss, for example) (Kaplan et al., 2020) lead over hardware capabilities (Hooker, 2020), this scale-up translates into a need for a growing number of training devices working in tandem (Chowdhery et al., 2022), turning distributed training to the standard approach for training DNNs on a large scale.

Distributed training typically refers to three parallelism paradigms - data parallel, model parallel and layer pipelining (Ben-Nun & Hoefler, 2019). Several variants and hybrid solutions exist in modern implementations such as tensor parallel (Narayanan et al., 2021) and parameter sharding (Rajbhandari et al., 2020; Rasley et al., 2020). These can be used separately or combined as they are orthogonal to each other. Mainly, data parallelism is straightforward, where the data is sharded among workers, and all workers share the same global model state. At each step, workers compute gradients locally and then aggregate them before taking an optimization step. When training synchronously, workers update their parameters in lockstep. This ensures that all workers hold a consensus on the same model and that gradients are averaged over all workers before being applied to the model. This approach is easy to implement and allows for good convergence properties, and correspondingly is the prevalent optimization method.

0

50

100

150

200

Workers

0

5

10

15

20

25

Speedup

Linear

Baseline

DropCompute

0

500

1000

1500

2000

Workers

0

50

100

150

200

250

Speedup

Linear

Baseline

DropCompute

Although state-of-the-art models use synchronous optimization for training, synchronous methods scale poorly, as stragglers and communication overhead might severely deteriorate system utilization. Many resources are invested in alleviating these issues in large-scale training systems since even minor improvements can be worth hundreds of thousands of dollars. These issues are exacerbated as the required scale grows, even in homogeneous high-performance computing clusters (Petrini et al., 2003; Hoefler et al., 2010). In this paper, we are interested in cases where significant computing variance between the workers exists. This includes (but is not limited to) straggling workers.

For instance, certain learning tasks entail heterogeneity in the required computation of data, such as varying sentence lengths in language processing, or different image sizes and frame numbers in computer vision. In addition, recent state-of-the-art models use all three parallelism paradigms (data (DP), tensor (TP), and pipeline (PP) parallelism), thus each data parallel node is a set of processing units (accelerators) communicating between them (via TP and PP) to calculate the model gradients collectively. This could potentially intensify compute variance between data parallel workers. Moreover, sub-optimal hardware systems can also lead to straggling workers. Although some sources of heterogeneity can be mitigated by non-trivial engineering work, they still persist as challenges that can introduce compute variance and substantial performance drops. This is further discussed in appendix A.

In this paper, we suggest a simple, yet effective method called DropCompute to improve the robustness and scalability of synchronous optimization in the face of compute variance. We model the compute time as a random variable and show that under reasonable assumptions, a tail of straggling workers slows down the system at a rate that is not proportional to the contributed compute by these straggling workers. We harness the gradient accumulation method widely used in Large Language Models (LLMs) (Ott et al., 2018; Liu et al., 2019) to implement the method in a few lines of code on a relevant large-scale learning task.

The contributions of our work include:

DropCompute : a novel, decentralized method to better handle heterogeneity or stragglers without additional hyper-parameters. DropCompute is hardware and framework agnostic, runs on top of existing optimizers, and can also be combined with other methods that improve other aspects of robustness such as communication overhead.

A theoretical convergence proof of SGD with stochastic batch size, as in DropCompute .

A theoretical runtime analysis on standard synchronous training and the proposed method. We find an approximation of the expected speedup using DropCompute , and show this speedup goes to infinity as the number of workers grows.

An empirical evaluation of the proposed method on a relevant large scale task, using up to 200 accelerators connected with high bandwidth communication. For example, when using DropCompute before and after system optimizations (for both software and hardware) we show accelerations of 18% and 5%, respectively.

2 Related Work

The challenge of training deep neural networks on a large scale has been extensively explored. With rapidly growing models and data sizes, numerous works tackled the weaknesses in synchronous DNN training on a large scale and suggested methods to alleviate these weaknesses.

Redundancy methods. This line of work addresses the straggling worker problem using a redundancy mechanism. Redundant workers or redundant data are used such that straggling workers will not slow down the entire system (Chen et al., 2016; Bitar et al., 2020). These methods provide better robustness to synchronous training, even in the event of a complete failure of a subset of the workers or considerable communication slowdown. However, the robustness is limited by the redundancy factor, and more generally, more compute resources are required and full utilization cannot be achieved. In addition, some coordination method is required to keep the training synchronous, i.e., keeping consensus between the model replicas of the workers. In particular, Chen et al. (2016); Bitar et al. (2020) use a centralized approach of a parameter server to determine which workers are left out at each iteration. Modern large-scale systems use decentralized variants of All-Reduce (von Luxburg et al.; Patarasuk & Yuan, 2009), so it is not trivial to determine which workers should be considered at each step, given that each worker can see a different subset of straggling workers. Moreover, combining redundancy with communication primitive collectives (e.g., All-Reduce ) requires adaptation to existing underlying frameworks (Sanders et al., 2019).

Asynchronous optimization. Another approach is introducing asynchrony to the optimization. Asynchronous training is inherently more scalable than synchronous training by being robust to all kinds of workers and communication faults. This includes periodic synchronization by exchanging parameters every τ optimization steps (Stich, 2019; Lin et al., 2020; Wang & Joshi, 2021; Zhang et al., 2015; Wang et al., 2020; Li et al., 2020b,a), approximate distributed averaging where each worker communicates with a subset of workers each step (Jiang et al., 2017; Lian et al., 2017; Assran et al., 2019; Yang et al., 2020), and many more. These works provide better scale-up properties and improve time performance. The main drawback is asynchronous optimization itself. In practice, the convergence is less stable, and more optimization steps are needed to generalize as well as synchronous training. In addition, hyperparameters should be chosen more precisely to guarantee convergence and generalization properties (Giladi et al., 2019; Mitliagkas et al., 2016). Due to these issues, asynchronous methods are less commonly used on a large scale.

Sparse and compressed communication. Alternatively, several works addressed only the communication overhead. A common technique is to reduce the amount of data exchanged by the workers at each step. This can be done by gradient pruning (Xu et al., 2021), gradient compression (Seide et al., 2014; Chen et al., 2020; Tang et al., 2021) or low-rank approximation (Vogels et al., 2019). These works reduce the communication overhead in a deterministic form while ignoring any compute variance across processing workers. This makes these works orthogonal to ours and potentially can be combined with our method.

3 Reducing Compute Variance

This paper proposes a method called DropCompute that improves the robustness of synchronous training by reducing compute variance. First, we describe the vanilla synchronous training framework. Then, we introduce the proposed approach.

3.1 Problem setup

We start with a model formulation for data-parallel synchronous SGD training with N workers, where each worker holds a replica of the model parameters θ . In parallelism paradigms that combine TP or PP with DP, N represents the count of data-parallel workers, not the total number of workers involved in training. Given a dataset D , we are interested in minimizing the empirical loss

where ℓ ( z, θ ) is the loss with respect to data-point z and the model parameters θ . At each step, the workers calculate gradients based on a local batch and then aggregate the gradients before taking an

optimization step. An equivalent strategy would be that the workers aggregate the parameters after taking an optimization step based on the local gradients. The aggregation is done in a decentralized fashion, such as AllReduce .

We also consider the use of gradient accumulation to increase the size of the local batch. Gradient accumulation breaks down each worker's local batch into M micro-batches for computation, which enables reaching a large global batch size beyond hardware capacity. This is common practice in training of LLM on large scale where a substantial number of accumulations are being utilized (Smith et al., 2022; Nvidia, 2023). In each iteration i and accumulation m , the gradients of the loss function with respect to the model parameters are computed, denoted by

where D ( m ) i,n ⊆ D is the micro-batch m of worker n , sampled without replacement from D , and we assume a constant micro-batch size |D m i,n | . The gradients accumulated by worker n at step i are

Finally, the workers aggregate and average the computed gradients to update the model parameters

where η is the learning rate. Equation 1 requires all workers to receive all gradients before updating the parameters. This communication restriction is what makes the training synchronous and ensures the model parameters are the same for all workers. However, due to this restriction, the slowest worker at each step dictates the iteration time. More generally, any variation in computation time among workers will lead to workers with idle time. Therefore, to improve the efficiency and robustness of the synchronous training process, we need to reduce the computation time variance among workers (namely, the compute variance).

3.2 Our method: DropCompute

To mitigate the effect of compute variance and to improve the robustness of synchronous training, we propose a simple yet effective method called DropCompute . DropCompute reduces the compute variance by introducing a compute threshold after which workers have to stop calculating local gradients and move directly to the communication phase, i.e., Equation 1.

In each iteration i , each worker n measures the time while calculating its local batch, and compares it against a given threshold τ , which is set as described in section 4.4. If that time exceeds the compute threshold , the worker stops and sends the gradients it has calculated so far. The rest of the training remains unchanged, thus synchronous training is intact. The pseudo-code of DropCompute is shown in Algorithm 1. This method maintains a decentralized approach while improving its robustness to compute variance and stragglers in particular. Since our method drops samples, the batch size is no longer deterministic. Specifically, the total batch size (the total number of samples used in one iteration by all workers) can be smaller than the maximal total batch size b max = NM |D ( m ) i,n | . Nevertheless, we show both theoretically and empirically that this makes no difference to convergence or generalization when the number of computed samples remains the same. Next, we analyze this method in section 4 and evaluate it in section 5.

4 Method Analysis

After establishing the notion of compute variance and the formulation of the proposed method, we analyze synchronous training and the potential value of the proposed method on the training time performance. To do so, we start with convergence guarantees when using DropCompute . Then, we theoretically analyze the computation time with synchronous training and when using DropCompute . Through this analysis, we estimate the potential speedup over vanilla synchronous training.

4.1 Convergence analysis of DropCompute

Using DropCompute , the batch size is no longer fixed, but a random variable, which can also potentially depends on the data samples. To the best of our knowledge, this setting is somewhat different from existing convergence guarantees. Therefore, in this section, we provide convergence guarantees for DropCompute by analyzing the convergence of SGD with stochastic batch size.

Assumption 4.1. Following the notations in section 3.1, consider a possibly non-convex smooth loss function L ( D , θ ) , with a global minimum θ ∗ ∈ R d , and the following (commonly used) assumptions

L-smooth : All functions L ( · , θ ) are with L-Lipschitzian gradients.

Unbiased estimation : Each ∇ ℓ ( z, θ i ) , z ∈ D is an unbiased estimate of ∇L ( D , θ i ) which is the true (full batch) gradient at θ i 2 . Namely, ∀ i : E [ ∇ ℓ ( z, θ i ) | θ i ] = ∇L ( D , θ i ) .

Bounded variance : The variance of the stochastic gradient is bounded by a constant σ ,

Theorem 4.1. Under the above assumption, applying SGD with DropCompute (Algorithm 1), ensures

where b max is the maximal total batch size, K is the total number of samples that are used throughout the training, θ 1 is the initial model, and ¯ θ is a random sample of θ i from the trajectory obtained by Algorithm 1, where i is selected with probability proportional to the total batch size at iteration i . The expectation is with respect to the randomization introduced due to sampling from D throughout the optimization process and with respect to choosing ¯ θ .

The bound in Theorem 4.1 is similar to existing fixed batch-size guarantees (Dekel et al., 2012). The second term in the bound does not degrade with the batch sizes, and behaves like O (1 / √ K ) , while the first term behaves like O ( b max /K ) . This implies that as long as b max ≤ O ( √ K ) , the second term is dominant and we are in the regime of linear speedup. This shows we can attain a linear speedup despite using changing batch sizes, as long as the maximal batch size is bounded.

Similarly, in Theorem D.1 in appendix D.1 we show that the loss itself converges, in the convex case along with a proof. The proof of Theorem 4.1 is provided in appendix D.2. Lastly, we discuss the impact of the stochastic batch size on generalization in appendix D.4.

4.2 Iteration time in standard synchronous distributed training

We start with finding a closed-form expression for the cumulative distribution function (CDF) of the iteration time, denoted as T , defined as

where T n represents the time taken by worker n to compute its local batch, which follows some cumulative probability function F T n ( x ) = P ( T n < x ) , which we assume is independent for each worker. Let F T ( x ) represent the cumulative distribution function and f T ( x ) represent the probability density function of the maximum iteration time T . The relation between F T ( x ) and F T n ( x ) is

Differentiating with respect to x and applying the chain rule gives:

̸

In the special case where all of the workers' iteration time distributions are identically and independently distributed (i.i.d.), this reduces to the well-known formula:

If the iteration time of each worker is distributed normally ( ∼ N ( µ, σ 2 ) ), the expected value of T can be approximated as shown by Bailey et al. (2014):

where Φ is the CDF of the standard normal distribution, and γ is the euler-mascheroni constant. Asymptotically the total iteration time is E [ T ] = Θ( √ log N ) . When the number of micro-batches M ≫ 1 , we can make a similar approximation under Central Limit Theorem (CLT) conditions. More details are in appendix C.2.

4.3 Iteration time and number of micro-batches with DropCompute

When using DropCompute with a constant threshold τ , each worker is preempted at ˜ T n = min { τ, T n } and joins the AllReduce . Therefore, the total iteration time with DropCompute is

where T c is a serial latency present in each iteration, which includes the AllReduce step. This upper limit serves to clip extreme values of T n , effectively constraining the range of potential outcomes for ˜ T . As a result, the compute time variability decreases, leading to a narrower distribution and enhanced compute efficiency. These effects are illustrated in Figure 2.

As a consequence of preempting each worker at ˜ T n , the number of micro-batches computed in each step varies. Denote as t ( m ) n the compute latency of a single micro-batch m for worker n , and T ( m ) n = ∑ m j =1 t ( j ) n . We can define the average number of micro-batches computed by each worker before reaching threshold τ as

Under CLT conditions, the expected value for ˜ M can be approximated in a closed form:

where µ, σ 2 are the mean and variance for a single micro-batch t ( m ) n compute latency, and Φ is the CDF of the standard normal distribution. This approximation closely fits the real value of ˜ M and can be used to analyze the expected gain from DropCompute . More details in appendix C.2.

7.4

7.6

7.8

8.0

8.2

8.4

8.6

8.8

step compute time

Tn

[sec]

0

1

2

3

density

Real

Simulation

7.8

8.0

8.2

8.4

8.6

8.8

9.0

max compute time

T

[sec]

0

5

10

15

20

density

no drops

1% drop

3% drop

5% drop

10% drop

4.4 Choosing the threshold

The throughput of the system can be seen as the number of micro-batches computed per second. For N workers, this can be written as NM/ ( T + T c ) . To evaluate the effectivness of DropCompute , we consider the difference in throughput between the baseline and when using DropCompute . Doing so, we can define the effective speedup for τ as:

Given the statistical characteristics of the training setting, it is possible to estimate analytically the expected value of the effective speedup E [ S eff ( τ )] by using Equations 5 and 4. Moreover, when plugging in the asymptotic form of E [ T ] , we find the expected speedup increases to infinity with N

As shown in figure 3b, Equation 4 is less accurate when samples deviate from a normal distribution. To find the optimal compute threshold, we synchronize the empirical distribution of micro-batch

(a)

2.5

5.0

7.5

10.0

Threshold

[sec]

0

5

10

15

20

S

eff[%]

simulation

analytical

(b)

5

10

15

Threshold

[sec]

0

5

10

15

20

S

eff[%]

simulation

analytical

analytical given

(

T

)

(c)

5

10

15

Threshold

[sec]

50

0

50

100

150

200

250

%

S

eff

(

)

T

+

T c

T

+

T c

M

(

)

M

compute latency between all workers after a few iterations. Given this distribution, we find T and ˜ M , and search in a decentralized way for the threshold τ ∗ that maximizes the effective speedup S eff ( τ )

defined in Equation 6. Overall, the cost of synchronizing the empirical distribution and finding τ ∗ is negligible, compared to a full training session, because it happens only once in a training session. Lowering the threshold leads to reduced compute time but higher compute drop rates. Figure 3c highlights this trade-off and the optimal τ ∗ is marked.

4.5 Compensating for dropped samples

The effective speedup metric S eff , accounts for dropped samples by treating them as a source of slowdown in direct proportion to the drop rate. This consideration enables us to execute additional computations to achieve the theoretical speedup. The extent of extra time spent on redundant calculations can be as much as R = ( M/ ˜ M -1) times the computational effort required when not applying DropCompute .

For instance, when 10% of the samples are dropped, we can expect to perform approximately 11% more calculations. Achieving this can be approached in several ways. One straightforward compensation method for LLM training involves adding an extra R · I base steps to the training process, where I base represents the number of training steps conducted without using DropCompute . In practice, achieving the original accuracy often requires even fewer additional steps, as illustrated in Figure 5, resulting in an even higher effective speedup.

Another method of compensating for the dropped samples is to increase the maximal batch size. When increasing the batch by R and dropping in average 1 -˜ M/M , we keep the average batch size the same as without using DropCompute , hence compensating for the lost samples. A third method, orthogonal to the first two, is resampling dropped samples before starting a new epoch to diversify the overall samples seen by the model. These approaches are rigorously tested and compared in Table 1b

5 Experiments

To be useful, DropCompute must possess two properties. First, it should not compromise the accuracy of the trained model. This property is put to test in section 5.1 where we fully train BERT-Large and ResNet-50 (Devlin et al., 2018; He et al., 2015), each on a different task, with different drop rates to compare accuracy. Second, DropCompute should maintain a high level of runtime performance, especially when compute variance or straggling workers exist and vanilla synchronous training time deteriorates. Section 5.2 tests runtime performance of DropCompute by training a 1.5 billion parameter language model, BERT1.5B (Devlin et al., 2018) with additive noise to the compute time of each worker.

Experimental setup. The analysis of all BERT models is performed on the same dataset as Devlin et al. (2018), which is a concatenation of Wikipedia and BooksCorpus with 2.5B and 800M words respectively. The finetuning of the pretrained models is performed on SQuAD-v1 (Rajpurkar et al., 2016). We verify the generality of DropCompute by additional evaluation of a ResNet-50 model for image classification on ImageNet (Deng et al., 2009). The experiments depicted in section 5.2 and section 5.1 are executed on Habana Gaudi-1 and Gaudi-2 accelerators, respectively, with high performance network (Habana, 2020).

5.1 Generalization performance

The sole difference in the optimization when DropCompute is applied is that the batch size is not deterministic, but stochastic, as explained in section 3.2. To complement theorem 4.1, we examine the generalization performance achieved with a stochastic batch size on two popular tasks.

Image classification. To evaluate the generality of stochastic batch size and DropCompute in particular, we evaluate the Top-1 accuracy of a ResNet-50 model on the Imagenet dataset using our method. Since it is not common to use gradient accumulation in large scale training of this task, we simulate the drops such that each worker randomly drops its local batch, so the total batch size is stochastic. This simulated environment enables us to examine the extent of drop rate we can use without compromising accuracy. Figure 10 in appendix B.2.2 shows that up to 10% drop rate, which is more than what DropCompute operates on, there is a negligible deterioration in accuracy.

Large language model. Training LLMs is resource intensive, typically using large batch sizes, which makes DropCompute appealing. We evaluate DropCompute method on this task by fully pretraining

(a) Varying drop rate, no compensation

(b) 10% drop rate, with compensation

BERT-Large model several times, each with a different drop rate. We follow the optimization regime described in You et al. (2019) with a batch size of 64K for phase-1 and 32K for phase-2 (more details are provided in appendix B.2). Each of the pretrained models is fine-tuned on the SQuAD task 3 times with different initializations. Fine-tuning is performed without drops, as it is not a large scale resource consuming task. Table 1a shows the average accuracy ( ± standard deviation) obtained for each drop rate. As shown, DropCompute at drop rates of up to 10% have negligible accuracy difference. Higher values measured up to 20% of dropped gradients provide acceleration with a small yet discernible hit on accuracy. We note that these results are for a fixed budget of steps. In the presence of compute variance, the effective speedup indicates that additional steps can be executed while still maintaining competitive runtime performance. This notion is demonstrated in section 5.2.

5.2 Runtime performance

The main purpose of our proposed method is to maintain runtime performance when compute variance is present. We examine this by measuring the speedup of DropCompute over standard synchronous training in several settings. First, we measure the potential speedup for different drop rates and training settings by post analysis of synchronous training without drops. In addition, we introduce compute variance by training with additive noise, and measure actual speedups using DropCompute . The experiments in this section are performed on BERT1.5B. Details are provided in appendix B.1.

Training with different number of workers and micro-batches. We evaluate the potential speedup of DropCompute on several training settings with natural heterogeneity and no drops. For each setting, we post analyze what would have been the speedup for different drop rates. As can be seen in Figure 4, DropCompute exhibits increasing benefits with a growing number of workers and compute requirements. However, there are diminishing returns in terms of speedup with more accumulations. This could possibly be explained by the amortization time of a large number of micro-batches.

0

5

10

15

20

% drop rate

0

1

2

3

4

5

6

% effective speedup

N=32

N=64

N=128

N=192

0

5

10

15

20

% drop rate

0

1

2

3

4

5

% effective speedup

M=8

M=16

M=32

M=64

Simulated delay environment. Although DropCompute may have value when the workers' compute latency variance is low, its significance becomes crucial when the workers' compute latency exhibits high variability. To evaluate our method, we introduce a delay environment where random latency is added to each micro-batch computation. This additive noise follows a bounded log-normal distribution. Detailed information and motivation regarding the additive noise are in appendix B.1. The experiments are executed with 12 gradient accumulations and a local batch size of 192. In Figure 1, the negative impact of compute variance on scalability is demonstrated and mitigated

using DropCompute . The results in Figure 1 also correspond to section 4 and Equation 11, where a theoretical extrapolation follows the same trend line. When utilizing DropCompute in this setup, achieving the same training loss as the baseline might requires additional training steps, however, it leads to a notable reduction in overall training time. Figure 5 demonstrates it in a training session with 64 workers, where approximately 3% more steps is needed to reach the same loss, in 13% less time.

0

1000

2000

3000

4000

5000

Steps

5

10

Training loss

baseline

DropCompute

0

10000

20000

30000

40000

Time [sec]

5

10

baseline

DropCompute

6 Discussion

Summary. Efficient scalable systems are a key component to enable the continued development of deep learning models. To this day, state-of-the-art models rely on synchronous distributed optimization. The challenge to maintain synchronous training as an efficient solution grows larger with the quickly growing model sizes and data. Therefore, improving the robustness and scalability of distributed synchronous training is an important endeavor. This paper tackles the challenge of maintaining synchronous training scalable in the face of compute variance. We propose DropCompute to improve the robustness of synchronous training. Workers drop their remaining compute when they reach a compute threshold, determined by exchanging and analyzing the compute latency distribution. We find that for a small percentage of dropped data, a much larger percentage of time can be saved, depending on the compute latency distribution of the workers. In addition, we provide theoretical convergence guarantees and runtime predictions. We further discuss the motivation behind DropCompute and how it effectively solves the problem in appendix A.

Limitations. While DropCompute is simple and straightforward, it deals with system efficiency, and as such, the user-level implementation provided is not optimal. Mainly, the provided implementation is limited by using many gradient accumulations and integrating compute timeout in between them. However, we believe that this is not a major concern since having multiple gradient accumulations is a common practice in training LLM on a large scale and is used in state-of-the-art training configurations (Smith et al., 2022; Nvidia, 2023). In addition, DropCompute addresses variance that originates from the compute stage of the training iteration and does not solve the potential issue of network variance during the all-reduce stage.

Future directions. DropCompute is described and analyzed in this paper as a method built on top of synchronous training. However, this method can be integrated with other possibly asynchronous methods such as periodic synchronization. In appendix B.3, we implement DropCompute on top of Local-SGD (Lin et al., 2020) and show that DropCompute can also improve the robustness of LocalSGD to stragglers. A different extension for DropCompute is to apply it during the model backward calculation and save the partial gradients that were already calculated. This would generalize DropCompute for workloads that do not utilize gradient accumulations. However, it will require further study as it differs from the stochastic batch-size setting where the entire data sample is either saved or dropped.

Acknowledgments

We thank Itay Hubara for technical advising and valuable comments on the manuscript. The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of Schmidt Career Advancement Chair in AI.

References

Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning , pp. 344-353. PMLR, 2019.

David H Bailey, Jonathan M Borwein, Marcos López de Prado, and Qiji Jim Zhu. Pseudomathematics and financial charlatanism: The effects of backtest over fitting on out-of-sample performance. Notices of the AMS , 61(5):458-471, 2014.

Tal Ben-Nun and Torsten Hoefler. Demystifying parallel and distributed deep learning: An indepth concurrency analysis. ACM Comput. Surv. , 52(4), aug 2019. ISSN 0360-0300. doi: 10.1145/3320060. URL https://doi.org/10.1145/3320060 .

Rawad Bitar, Mary Wootters, and Salim El Rouayheb. Stochastic gradient coding for straggler mitigation in distributed learning. IEEE Journal on Selected Areas in Information Theory , 1(1): 277-291, 2020.

Chia-Yu Chen, Jiamin Ni, Songtao Lu, Xiaodong Cui, Pin-Yu Chen, Xiao Sun, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Viji Srinivasan, Wei Zhang, et al. Scalecom: Scalable sparsified gradient compression for communication-efficient distributed training. Advances in Neural Information Processing Systems , 33:13551-13563, 2020.

Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981 , 2016.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Patch n'pack: Navit, a vision transformer for any aspect ratio and resolution. arXiv preprint arXiv:2307.06304 , 2023.

Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research , 13(1), 2012.

Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 , 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=ZBESeIUB5k .

Niv Giladi, Mor Shpigel Nacson, Elad Hoffer, and Daniel Soudry. At stability's edge: How to adjust hyperparameters to preserve minima selection in asynchronous training of neural networks? arXiv preprint arXiv:1909.12340 , 2019.

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.

Habana. Habana gaudi training whitepaper, June 2020. URL https://habana.ai/wp-content/ uploads/pdf/2020/Habana%20GAUDI%20Training%20Whitepaper%20v1.2.pdf .

Habana. Pre-training the bert 1.5b model with deepspeed, January 2023. URL https://developer. habana.ai/blog/pre-training-the-bert-1-5b-model-with-deepspeed/ .

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.

Torsten Hoefler, Timo Schneider, and Andrew Lumsdaine. Characterizing the influence of system noise on large-scale applications by simulation. In SC'10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis , pp. 1-11. IEEE, 2010.

Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems , 30, 2017.

Sara Hooker. The hardware lottery. CoRR , abs/2009.06489, 2020. URL https://arxiv.org/ abs/2009.06489 .

Zhanhong Jiang, Aditya Balu, Chinmay Hegde, and Soumik Sarkar. Collaborative deep learning in fixed topology networks. Advances in Neural Information Processing Systems , 30, 2017.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361 .

Matej Kosec, Sheng Fu, and Mario Michael Krell. Packing: Towards 2x NLP BERT acceleration. CoRR , abs/2107.02027, 2021. URL https://arxiv.org/abs/2107.02027 .

David A Levin and Yuval Peres. Markov chains and mixing times , volume 107. American Mathematical Soc., 2017.

Kfir Levy. Online to offline conversions, universality and adaptive minibatch sizes. Advances in Neural Information Processing Systems , 30, 2017.

Shigang Li, Tal Ben-Nun, Giorgi Nadiradze, Salvatore Di Girolamo, Nikoli Dryden, Dan Alistarh, and Torsten Hoefler. Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging. IEEE Transactions on Parallel and Distributed Systems , 32(7):1725-1739, 2020a.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020b.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Huangxing Lin, Weihong Zeng, Yihong Zhuang, Xinghao Ding, Yue Huang, and John Paisley. Learning rate dropout. IEEE Transactions on Neural Networks and Learning Systems , pp. 1-11, 2022. doi: 10.1109/TNNLS.2022.3155181.

Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. In International Conference on Learning Representations , 2020.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https://arxiv.org/abs/1907.11692 .

Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David A. Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojyoti Dutta, Udit Gupta, Kim M. Hazelwood, Andrew Hock, Xinyuan Huang, Bill Jia, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Carole-Jean Wu, Lingjie Xu, Cliff Young, and Matei Zaharia. Mlperf training benchmark. CoRR , abs/1910.01500, 2019. URL http://arxiv.org/abs/1910.01500 .

Ioannis Mitliagkas, Ce Zhang, Stefan Hadjis, and Christopher Ré. Asynchrony begets momentum, with an application to deep learning. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pp. 997-1004. IEEE, 2016.

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , SC '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384421. doi: 10.1145/3458817.3476209. URL https://doi.org/ 10.1145/3458817.3476209 .

Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks, 2015. URL https://arxiv.org/abs/1511.06807 .

Nvidia. mlcommons training results v3.0 gpt3. https://github.com/mlcommons/training_ results_v3.0/tree/main/NVIDIA/benchmarks/gpt3/implementations/pytorch , 2023.

Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers , pp. 1-9, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6301. URL https://aclanthology.org/W18-6301 .

Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. Journal of Parallel and Distributed Computing , 69(2):117-124, 2009.

Fabrizio Petrini, Darren J Kerbyson, and Scott Pakin. The case of the missing supercomputer performance: Achieving optimal performance on the 8,192 processors of asci q. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing , pp. 55, 2003.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020a. URL http://jmlr.org/papers/v21/20-074.html .

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020b.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pp. 1-16. IEEE, 2020.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pp. 3505-3506, 2020.

Peter Sanders, Kurt Mehlhorn, Martin Dietzfelbinger, and Roman Dementiev. Sequential and Parallel Algorithms and Data Structures . Springer, 2019.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176bparameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and application to data-parallel distributed training of speech dnns. In Interspeech 2014 , September 2014.

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 , 2022.

Pawel Sobkowicz, Mike Thelwall, Kevan Buckley, Georgios Paltoglou, and Antoni Sobkowicz. Lognormal distributions of user post lengths in internet discussions-a consequence of the weberfechner law? EPJ Data Science , 2(1):1-20, 2013.

Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations , 2019.

Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning , pp. 10096-10106. PMLR, 2021.

Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large-scale training with adam's convergence speed. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 10118-10129. PMLR, 18-24 Jul 2021.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems , 32, 2019.

U von Luxburg, S Bengio, HM Wallach, R Fergus, SVN Vishwanathan, and R Garnett. https://github. com/baidu-research/baidu-allreduce.

Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. Journal of Machine Learning Research , 22, 2021.

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-efficient distributed sgd with slow momentum. In International Conference on Learning Representations , 2020.

Hang Xu, Kelly Kostopoulou, Aritra Dutta, Xin Li, Alexandros Ntoulas, and Panos Kalnis. Deepreduce: A sparse-tensor communication framework for federated deep learning. Advances in Neural Information Processing Systems , 34:21150-21163, 2021.

Donglin Yang, Wei Rang, and Dazhao Cheng. Mitigating stragglers in the decentralized training on heterogeneous clusters. In Proceedings of the 21st International Middleware Conference , pp. 386-399, 2020.

Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 , 2019.

Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. Advances in neural information processing systems , 28, 2015.

Shuai Zheng, Haibin Lin, Sheng Zha, and Mu Li. Accelerated large batch optimization of bert pretraining in 54 minutes, 2020.

Peer reviews:
comments: SUMMARY:
This paper considers a typical scenario in which workers are straggling due to variability in compute time, in which the authors find an analytical relation between compute time properties and scalability limitations. They propose a decentralized method to reduce the variation among workers and   improve the robustness of synchronous training.

STRENGTHS:
1. This paper studies the distribution of iteration time and presents some insights into understanding distributed optimization.

2. Based on the analysis of the iteration time for the worker, the authors propose an algorithm that drops some data in computing local gradients to avoid waiting.

3. They present the analysis of distributed SGD with stochastic batch size, which is novel and interesting.

4. Numerical results are presented to verify the assumptions and demonstrate that the robustness of the convergence.

WEAKNESSES:
1. The convergence is proved under many extra assumptions than that of distributed SGD and cannot explain why the new algorithm can be better.

2. According to Figure 5, the proposed algorithm seems only a little better than the baseline.

QUESTIONS:
What is the computation cost of Algorithm 2? Does it consume much time, and is the consuming time recorded in  Figure 5?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a method to drop computation (DropCompute) of **microbatches** of a **minibatch** on decentralized workers if their execution time exceeds a certain threshhold. The author provided theoretical convergence proof of this method as well as theoretical/experimental study of the performance gain or speed up of this method.\n\nSTRENGTHS:\n* The proposed method is quite simple and straightforward in implementation, especially for decentralized SPMD distributed system.\n\n* The theoretical study of the convergence and speed up of this method is thorough and clear\n\n* The ablation study of the experimental section is clear\n\nWEAKNESSES:\n* If my understand is correct, the theoretical analysis (line 186) and efficacy (Figure 4 right) of the method relies on large enough minibatch size per replica, so that number of microbatches M >> 1. This probably worked fine for BERT training (line 241), however for modern LLM pretrainings, e.g., GPT3 XL (similar size as the model studied in this work), the global batchsize is 1M tokens/(2K tokens/sample) = 500 samples. Distributed on 200 replicas, minibatch / replica is only 2-3 samples, which fundamentally breaks the assumption. The global batchsize can be even smaller for fine-tuning. Addtionally, the evaluation on Image task (line 229) basically has only 1 microbath per minibatch, and does not serve a good experimental study for DropCompute (which is more like Federated Learning).\n\n* The proposed method drops data/samples per iteration. This might be fine if the data is trained multiple epochs like BERT scale, however for modern LLM, especially pretraining, the data is usually trained one epoch so if one micrabatch is dropped, it might be dropped from training forever.\n\n* The performance gain is marginal, e.g, theoretically at most 20% (Figure 3) by choosing the best threshold $\\tau$ or droprate, and in practice 5% (Figure 4) if maintaining final model quality. This is likely due to the compute variance of replicas follow a normal distribution (with small std) and the system w/o DropCompute is not significantly slowed down in this case. Things can be more interesting if the variance follows an extreme distribution, e.g., normal + delta function, but that's not likely true for datacenter clusters.\n\n* The author said no additional hyper-parameters are introduced (line 55), however proper droprate or threshold time $\\tau$ is still a new hyperparameter\n\nQUESTIONS:\n* in step 6 Algorithm 1 the authors mentioned step 7 and step 10 are in parallel, does it mean step 9 could stop in the middle if time out in step 10 or step 8-9 are still atomic? The description seems to suggest it is the former one (otherwise step 10 can be just a break condition in the forloop), but this results in gradients with i or i - 1 steps accumulation and breaks the theoretical analysis if my understand is correct.\n\n* is it possible to reuse dropped data/samples, e.g, there could be a synchronization of dropped sample indices of the dataloader of each replica so that next batch can be sampled with previously dropped data? Alternatively, can you cache the dropped data/samples locally on each replica, and use them in next batch. Both seem to be simple change to dataloaders.\n\nLIMITATIONS:\nThe authors didn't talk about their limitations. But from my understand, the main limitations include the first 3 points mentioned in Weaknesses:\n* large number of microbatches assumption\n* training data dropping\n* the problem solved is not significiant for LLM trainings which mostly happen in datacenter (compared to Federated Learning)\n\nAdditionally, the method is only applicable to traditional data parallel training, for modern distributed system that adopts FSDP/DeepSpeed-Zero and Tensor/Pipeline Parallellsim, the proposed method can't help with their parallel compute variance (because it depends on gradient accumulation, again, multiple microbatches in a minibatch).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DropCompute, a method to mitigate load imbalance across workers when doing distributed training. It splits mini-batches into micro-batches, and uses a dynamically-calculated time budget, which, when exceeded, results in the remaining micro-batches being dropped and communication to aggregate gradients beginning. A convergence analysis is provided, and experiments are conducted on ResNet-50 and BERT.\n\nSTRENGTHS:\n1. Load imbalance is an important emerging topic and somewhat understudied in distributed deep learning.\n2. DropCompute is a simple but appealing idea; integrating with micro-batching is a nice idea.\n3. The paper includes theoretical analyses and justifications for their choices.\n4. The experiments cover large, representative networks (ResNet-50, BERT) in different tasks/domains.\n\nWEAKNESSES:\n1. The motivation and significance of the paper is unclear to me: I am not sure load imbalance is an issue in practice for training. The paper gives the example of varying sentence lengths, but transformers typically concatenate sentences and pad to fixed lengths during training. Another example given is varying image sizes, but I do not think it is typical to train that way. The paper would be much stronger if there were concrete examples of widely-used and/or important use cases with load imbalance during training. This could be an issue in reinforcement learning, but that is a very different setting. Networks with conditional computation may also exhibit load imbalance, but I do not know any widely used examples.\n2. A related issue to this shows up in the experimental evaluation: Delays are manually induced through simulation, rather than actual load imbalance. While this is useful for ablation studies and accuracy analysis, it reinforces my prior point.\n3. There are no comparisons with other techniques to handle stragglers, such as asynchronous methods. For example, Li et al., \"Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging\", IEEE TPDS, 2021 (see also the references therein).\n\nQUESTIONS:\n1. Is load imbalance an issue in practice? Can you name concrete situations, which occur in practice, where it is an issue?\n2. Can you add an experiment showing DropCompute's benefit without artificially-induced delays?\n3. Please add experimental comparisons with asynchronous/wait-avoiding systems (see above).\n4. Why drop samples? It seems wasteful --- why not just use them in the next iteration? Is this due to a desire to follow synchronous training?\n5. It might be worth clarifying that DropCompute cannot address all causes of stragglers. In particular, it seems like it does nothing for delays caused by network issues (e.g., contention, dropped packets, routing issues, etc.). This is not a criticism so much as a limitation that should be acknowledged.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a strategy to improve the scalability of DNN training for highly distributed workloads by allowing nodes that are lagging behind, in terms of processing time, to stop computing updates and share partial results with the other nodes in the cluster. The proposed method, DropCompute, allows the training strategy to remain synchronous while supporting scalability. The primary method to scale train while maintaining reasonable performance typically relies on asynchronous processing and/or sharing of updates between nodes in the training cluster to avoid the performance impact of straggler nodes. By maintaining the synchronous training guarantees DropCompute avoids the less desirable divergent model convergence behavior that may exist with an asynchronous approach. The authors propose a simple strategy to implement DropCompute using modern DNN software and provide theoretical and empirical results to support the claim that dynamically dropping straggler computation does not result in a negative impact on the training accuracy of the final model while simultaneously enabling faster and more reliable computation times as the number of nodes in the cluster are increased. Evaluation results consisting of runtime performance and final model training accuracy are provided to support the authors' claims regarding the effectiveness of DropCompute on real-world models.\n\nSTRENGTHS:\n- The authors provide a clear description and mathematical model to understand the expected behavior of dropping straggler computations. The simplicity of the method lends itself to a simple description and implementation that is clear and easy to follow.\n- With the increased scale of training the influence of computational variability will continue to grow in importance. This fact leads to the natural importance of methods that are robust to possible large variations that may exist between various nodes in a cluster.\n- Asynchronous training has been the go-to strategy for a number of years but the adverse impact of this strategy has been downplayed to achieve more distributed training at higher performance. DropCompute is a refreshing novel take on the distributed computation problem and is amenable to simple implementation on top of existing DNN software, which is reminiscent of dropout or dropconnect training.\n- The theoretical analysis provides an interesting analysis of the expected behavior of dropcomute training and the overall impact on the final accuracy of the model.\n- Evaluation data using a simulated delay environment support the performance claims presented by the authors.\n\nWEAKNESSES:\n- Though DropCompute is interesting the community of researchers with access to training environments large enough and experience sufficient computational variability to truly benefit from the approach could be small.\n- How often do researchers with access to large distributed training systems experience high compute variability?\n- The implementation would benefit from a lower-level implementation to increase overall efficiency. This was noted by the authors.\n\nQUESTIONS:\nNo questions at this time.\n\nLIMITATIONS:\nAll limitations and societal impacts were addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}