Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees
Authors: Jue WANG
Binhang Yuan
Luka Rimanic
Yongjun He
Tri Dao
Beidi Chen
Christopher Re
Ce Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Communication compression
Quantization
SGD

Abstract: Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive studies of gradient compression for data parallel-style training, compressing the activations for models trained with pipeline parallelism is still an open problem. In this paper, we propose AQ-SGD, a novel activation compression algorithm for communication-efficient pipeline parallelism training over slow networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AQ-SGD compresses the changes of the activations. This allows us to show, to the best of our knowledge for the first time, that one can still achieve $O(1/\sqrt{T})$ convergence rate for non-convex objectives under activation compression, without making assumptions on gradient unbiasedness that do not hold for deep learning models with non-linear activation functions. We then show that AQ-SGD can be optimized and implemented efficiently, without additional end-to-end runtime overhead. We evaluated AQ-SGD to fine-tune language models with up to 1.5 billion parameters, compressing activation to 2-4 bits. AQ-SGD provides up to $4.3\times$ end-to-end speed-up in slower networks, without sacrificing model quality. Moreover, we also show that AQ-SGD can be combined with state-of-the-art gradient compression algorithms to enable end-to-end communication compression: All communications between machines, including model gradients, forward activations, and backward gradients are compressed into lower precision. This provides up to $4.9\times$ end-to-end speed-up, without sacrificing model quality.

Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees
Jue Wang" W.3 Binhann Yuan'; LLuk Rimanic ff; Yongjun He', Tri Dao",
Beidi Chen Christopher R Ce Zhang'
ETH Züiiich, Switzerland Stanford University, USA $Zhejiang University, China
*Carnegie Mellon University 5 Meta Al
(juewang, binhang. lukaniimanic, yongjun.he ce.zhang) @inf.ethz.ch (beidic, trid, chrismre stanforrdeduu
Abstract
Communication compression is a crucial technique for modern distributed learn- ing systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive studies of gradient compression for data parallel-style training, compressing the activations for models trained with pipeline parallelism is still an open problem. In this paper, we propose AQ-SGD, a novel activation compression algorithm for communication-efficienn pipeline parallelism training over slow networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AQ-SCD compresses the changes of the activations. This allows us to show, to the best of our knowledge for the first time, that one can still achieve O(1/vT) convergence rate for non-convex objectives under activation compression, without making assumptions on gradient unbiasedness that do not hold for deep learning models with non-linear activation functions. We then show that AQ-SGD can be optimized and implemented effi- ciently, without additional end-to-end runtime overhead. We evaluated AQ-SGD fine-tune language models with up to 1.5 billion parameters, compressing activation to 2-4 bits. AQ-SGD provides up to 4.3x end-to-end speed-up in slower networks, without sacrificing model quality. Moreover, we also show that AQ-SGU can be com- bined with state-of-the of-the-art gradient compression algorithms enable "end-to-end communication compression": All communications between machines, including model gradients, forward activations, and backward gradients compressed into lower precision. This provides up to 4.9x end-to-end speed-up, without sacrificing model quality.
Introduction
Decentralized or open collaborative training has recently attracted intensive interests Despite their great potential in leveraging geo-distributed powerful GPUs, the computation efficiency severely hindered by low network bandwidth - typically in the range of 10-400Mbps 22] 3 4
Recently, efforts in improving communication efficiency have significantly decreased the dependency on fast data center networks
the gradient can be compressed to lower precision or sparsified [5] 6
7 8) which speeds up training over low bandwidth networks, whereas the communication topology [3] [4], which speeds up training over high latency networks. Indeed, today's s state-of- toitte-ff-hhee offthhe-rt art training systems, such Pytorch [15 [6], Horovod (I7). Bagua (18).
can be decentralized [9] [0] [1] [2
and BytePS [9]. already support many of these communication-effiieenn training paradigms.
Equual contribution.
Now at Google.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Eljdekal] Elaattaaton
Table 1: Summary of techni- cal results. AC-GC (29 and TinyScript [30 assume that the re- turned gradient is unbiased, whereas AQ-SGD algorithm does not rely on such an assumption.
tost perplokaa
2.0
epach
(a) Fine-tune WikiText2
(b) Activation and delta
Algonithm
Ascumptions
Quant. Grad
Com: Rate
SGD 3D
NiA
0(1/TT)
Figure 1: (a) Fine-tuning GPT2-1.5B with different activation precisions in communication; (b) Average absolute value of activations and their changes for GPT2-1.55 during training.
AC GC (2
Unbiased
((1/VT)
TeayScript [0]]
Unbiased
(11vTT)
Aq-San
NIA
0(1/VT)
However, with the rise of large foundation models (0 (c.g., BERT 21], GPT-3 (2).. and CLIP (3 improving communication efficiency via compression becomes more challenging. Current training systems for foundation models such as Megatron (24, Deepspeed [25], and Fairscale [26], allocate different layers of the model onto multiple devices and need to communicate the gradients on the models the activations during the forward pass and the gradients on the activations during the backward pass, Compressing these activations leads to very different behavior compared with compressing the gradient simply compressing these activations a stochusticully unbiased way will lead biuses in the gradient that cannot be measured easily expressed in closed form. This either breaks the unbiasedness assumptions made by most gradient compression results 5 6 8 makes error compensation over gradient biases [27 28 difficult to adopt.
in addition to
35 36 illustrate, albeit mostly empirically, that large deep learning models can tolerate some compression errors on these activation values. However, when comes to the underlying theoretical analysis, these efforts mostly make assumptions that do not apply to neural networks with non-linear activation functions the only two recent efforts that claim theoretical convergence analysis [29 30 assume that an unbiased compression on activations leads to unbiased error on the gradient. Not surprisingly. these algorithms lead to suboptimal quality under relatively aggressive compression, illustrated Figure la many cases, using activation compression to fine-tune model might be worse than zero-shot learning without any fine-tumin toting at all.
Previous efforts on activation compression 32
33
34
In this paper, we focus on the problem of activation compression for training language models over slow networks by asking the following:
Q1. Can we design an algorithu for activation compression with rigorous theoretical guarantees SGD convergence?
Q2. Can such an algorithum be implemented efficiently without additional runtime overhead and oulperform today's 's activation compression algorithms without accuracy loss?
Our answers both questions are Yes. (Contribution 1) We propose AQ-SGD, novel algorithm for activation compression. The idea of AQ-SGD is simple instead of directly compressing the activa- tions, compress the change of activations for the same training example across epochs. Intuitively, we expect AQ-SGD to outperform simply compressing the activations because it enables an interesting "self-enforeing" dynamics: the more training stabilizes - the smaller the changes of the model across epochs + the smaller the changes of activations for the same training example across epochs -+ the smaller the compression error using the same #bits - training stabilizes more. (Contribution 2) The theoretical analysis of AQ-SGD is non-trivial since we have to analyze the above dynamics and connect to SGD convergence, which is quite different from most of today's results on gradient compression and error compensation. Under mild technical conditions and quantization functions with bounded error, we show that SQDSCD converges with rate of O(1v/T) for non-convex objectives, the same as vanilla SGD 31 37. To the best of our knowledge, AQ-SCH is the first activation compression algorithm with rigorous theoretical analysis that shows a convergence rate of O(1/VT) /VT)) (without relying on assumptions of unbiased gradient). (Contribution 3) We then show that AQ-SGD can be

---

Backward Gradient
For sample &
Direct Quantization [26, 27]: c(x) (uantize(x)
Forward
AQ-SGD: C(t) Quantize m(t:))
Activation
m(x.) m(t.) Q(x m(f+))
Transformer Block
Transformer
Transformer Block
Transformer
Transformer
Transfommer Block
Machine
Macro- batch 1
Block
Block
Block
Machine
Model
Gradient
Machine 4
Machine
Transformer Block
Transformer Block
Transtormer Block
Transformer Block
Transformer
Transformer
Macro- batch 2
Block
Block
Figure 2: The communication pattern of training large language models with both data parallelism and pipeline model parallelism. C denotes a compression module. The goal of this paper to understand the design of C' for forward activation and backward gradient.
optimized and implemented efficiently without adding additional end-to-end runtime overhead over non-compression and other compression schemes (it does require us to utilize more memory and SSD for storage of activations). (Contribution 4) We then conduct extensive experiments on sequence classification and language modeling datasets using DeBERTa-1.5B and GPT2-1.5B mod- els, respectively. We show that AQ-SGD can aggressively quantize activations to 2-4 bits without sacrificing convergence performance, where direct quantization of activations fails to converge; in slow networks, AQ- AQD achieves up 4.3x end-to-end speedup. (Contribution 5) Last but not least, we also show that AQ-SGD can combined with state-of-the-ee gradicnt compression algorithms to enable "end-to-end communication compression": All data exchanges between machines, including model gradients, forward activations, and backward gradients are quantized into lower precision. This provides up to 4.9x end-to-end speed-up, without sacrificing model quality.
Overview and Problem Formulation
Training large language models over multiple devices is a challenging task. Because of the vast number of parameters of the model and data examples, state-of-the-art systems need to combine different forms of parallelism. Filusreiil an example in which such a model distributed over four machines: (Pipeline Model Parallelism) The model is partitioned onto Machine 1 and Machine (similarly, Machine and Machine 4), each of which is holding a subset of layers. To compute the gradient over the model using backpropagation, these two machines need communicate the activations during the forward pass and the gradient on activations during the backward pass. (Data Parallelism) Machine and Machine 3 (similarly, Machine 2 and Machine 4) process the same set of layers for different macro- batches In this case, each of them will hold a replica of the same model. After the gradient over their model parameters ready, they need to communicate the model gradient, usually by taking an average 15 17 [8]
Communication Compression for Forward Activations and Backward Gradients. In slow networks, the communication among all machines often becomes the bottleneck [37]. To improve the speed of training, one can conduct lossy compression of the data before they are communicated, illustrated the C' module in Figure When the model fits into single machine, there have been intensive efforts on compressing the model gradient [5] 7 g. However, when it comes to pipeline model parallelism, such compression techniques are less studied. In this paper, we focus designing
Our code is available at: https://github.com hww.

---

efficient communication compression algorithms to compress both forward activations and backward gradients. As we will show later, both can be compressed significantly with AQ-SGD without hurting the model quality. We also show that it is possible to combine AQ-SGD with state-of-the-ar gradient compression techniques to enable the end-to-end compression scheme illustrated Figure
Problem Formulation. In this paper, we focus on the following technical problem. Note that, for the simplicity of notations, we present here the case where the model is partitioned onto K machines. AQ-SGD works for cases with K >> 2: (1) in experiments, we consider K = 8, i.e., single model is partitioned onto 8 machines; (2) in the supplementary material provide the theoretical analysis for K >> 2.
Given distribution of samples D, consider the following optimization task:
min ZE
(x) == Eg-D (((((,()) x.:)))
(2.1)
where F is loss function, a(-) and b(-) correspond to two sets of layers of the model - ( has model x²(0) and b(-) has model (²()) In Figure 2 Machine 1 would hold z²() and Machine would hold (²(b) In the following, we call the machine that holds r²(') Machine a and the machine that holds (b))) Machine b. In the standard backpropagation algorithm, the communication between these two machines
follows:
Given a data sample ç. Machine a sends to Machine the forward activation: a(E, (())) Machine b sends to Machine the backward gradient on a(S, x(()).
Difficulties in Direct Quantization. A natural way at compressing forward activations is to send,
instead of (()), quantized version m(g, ..(a) Q(a(E, x (l)))). This is the quantization scheme that state-of-the-ard methods such as AC-GC [29] and TinyScript [30] use. Both AC-GC (29
and TinyScript (30] assume that gradient is unbiased when mmg, x(")) ) is an unbiased estimator of
a(C, x()).. This enables their convergence rates of (1IVVT). However, because of the non-linearity
of and b in a deep learning model with non-linear activation functions, an unbiased mm(. wx(a)
will lead to biases on the gradient. In Appendix, we will provide an example showing that such gradient bias will hurt SGD convergence even for a very simple optimization problem. On the theory side, previous efforts on understanding gradient bias (88 have also shown that even bounded bias on gradient can impact the converges of SGD. As we will show later, empirically, this bias can indeed lead to suboptimal models under aggressive compression.
Notation.
Throughout the paper we use the following definitions:
f* is the optimal value of f.
N is the number samples.
Zz = (=)))) xt
(b) is the full model at iteration t.
Vf(() is the gradient of function f.
93, (Iz) v (f:: is the stochastic gradient.
Q(-) is the quantization function used to compress activations.
m(-) is the message exchanged between and b in the feed forward path.
|| denotes the L2-norm.
AQ-SGD: Theoretical Analysis and System Implementations
In this section we present the AQ-SGD, with the goal to mitigate the above mentioned difficulties that appear in direct quantization of the activation functions.
3.1 AQ-SGD Algorithm
Algorithm T Iillustrates the AQ-SCD algorithm. The idea behind Algorithm is simple- instead of quantizing the activations directly, quantize the changes of activations for the same training example across epochs. As illustrated in Algorithm I for iteration and the data sample &t, it is the first time

---

Algorithm AQ-SGD Algorithm
1: Initialize: za, learning rate 7; sub-network a( weights x1(), sub-network b(-) weights a:(h)
quantization function Q. array of previous messages m initialized to 0
2: t= I.... do
3: Randomly sample
4: if & not seen before then
5: Set m(t)) = a(Ei, (
6: else
7: Update m({)) + m(E)) + Q(a(Lt) x| m(E)))
8:
end if
9:
// Machine sends Q Qaa(Ei, m((i)) to Machine b, which knows m({)) through a local
version of m
10:
Update rt+1 + zt
(b)
V Vacss (s b
11:
I/ Machine sends Qf o m) to Machine
Update x/41 t 1t -7 Q(Va(J mm) V Viaia
(a)
(a)
13:
end for
(a) Output: r (xT (
that & is sampled, Machine a communicates the full precision activations without any compression: m(&)) a(E,, ( (Lines 4-5). Both machines will save m({:) in local buffer. If has been sampled in previous iterations, Machine a communicates quantized version:
(a((E, (()) mm())),
where m({)) the previous message, stored in the local buffer. Both machines then update this local buffer:
m(:) <- m(Ee) + Q(a(Ee, =("") m(E:))-
Machine b then use m({)) the forward activations, compute backward gradients, and communicate quantized version of the backward gradient to Machine (Line 11). We use
5g = a(Lt, xt (o)) - mm(E)
to denote the message error in sending the activations.
Update Rules. The above algorithm corresponds to the following update rules, at iteration t with sample &
(a) (a) QV (m V V2(wa af
F++1 zt
(b) = z; 7 V (f a ( (m((ss= (.)
2+1
where y is the learning rate, V Z (f b) (m( (())))) is the gradient (b) using the quantized forward activations (m(s, x;( (,) and Q(V o (m(s. (. is the quantized backward gradient.
(a) .t (b))), Setting 2, (xt
rephrase the update rule
Ft+1 = 7 (0e( + Ac(x:)) where g5 (xx) is the stochastic gradient and (x compression. We have Ag = (A (a)
) is the gradient error introduced by communication given by:
+ + (@)
A (2) (zx) Qf o b) (m((e.
al Valf o b)) (mm(ss V Va
4 (0) (xxe) Va(ʃ 6)) (
an va(fobo (20))
(b) (xx) V,oos (f bb) (mtt.))))) ( v (f (((eete..")) (oa"))
where A (2) (Ixz) is the enror introduced by the gradient quantization in the backpropagation part, whilst A (0) (xx) and A (b) (xx) the errors that the gradients of and b, respectively, inherit from the bias introduced in the forward pass.
5

---

3.2 Theoretical Analysis
We now prove the main theorem which states that, under some standard assumptions that are often used in the literature 37). the convergence rate of AQ-SGD algorithm O(LvVT) for non-convex objectives, same as vanilla SGD.
Assumptions. We make several assumptions on the networks and the quantization. It important to note is that we put no restrictions on either the message error óg, nor the gradient error Aç.
(A1: Lipschitz assumptions) We assume that f., ((f o b) and are Ly, fob-, and la Lipschhit, respectively, recalling that function is Lg Lipschitz if
[1g(z) - ا - 4, yll.
Vx, Vy.
Furthermore, assume that and o b have gradients bounded by C. and Cobb respectively, i.e. [|a((z)] & C., and [f 6)()) _ Coob-
(A2: SGD assumptions) We assume that the stochastic gradient ge is unbiased, i.e. [x)) =
f((),, for all z, and with bounded variance, i.e. EE [x]]g((2) f=x)l1 S o², for all
Theorem 3.1. Suppose that Assumptions AI, A2 hold, and consider unbiased quantization function
(z)) which satisfies that there exists eQ 11/2 such that E| Qxx)] S co||=]|, for all a Let
7 CC3 be the learning rate, where
C= Acqla(1 + Job
V1 1-2/
Then after performing T updates one has
B(V)(=)) (C (C+L))((o)) EE
o² +cooCooCoo1) VT
(3.1)
VT
We present the full proof of Theorem 3.1 in Appendix ??, whereas here we explain the main intuition. The usual starting point in examining convergence rates is use the fact that f has Li-Liscchitz gradient. It is well known that this implies
7 (VV((z)) ge. (xx)) + f(+++1) - f(x,) = - (VF(x.)) (x,)) + 2
9e, (xr) + De, (2:))1
2
After taking the expectation over all & and summing over all t = ,,....I7 T, we easily see that the (a)
key quantity to bound is Σ/ E Â [22 where (xx) = (A (zx), A (b) (mr.)) On the other
hand, the main object that we can control is the message error, dç. Therefore, we first prove an auxiliary result which shows that | (x.))l (2)al),,, (zz)]l. for all t. The key arguments for bounding o5, closely follow the self-improving loop described in the introduction, and can be summarized follows. Since are compressing the information in such way that compare with all the accumulated differences, this allows us to unwrap the changes which appeared since the last time that we observed the current sample, in an iterative way. However, since these are gradient updates, they are bounded by the learning rate - as long as we have a quantization method that keeps enough information about the signal, we can recursively build enough saving throughout the process. In particular, the more stability we have in the process, the smaller the changes of the model and the compression error gets, further strengthening the stability.
Tightness. The bound is tight with respect to quantization - setting cQ = 0 (implying C' = 0), i.e.
quantization does not incur any loss, recovers the original original SGD convergence (cf. 31 37
'Even for very simple quantization function Q(x) ||=||| [>/[|=]]]] where [-] denotes rounding to the closest k/2", stochastically. through simple bound cq vd/2", 6 bits suffice in low-dimensional (~ 10"). 11 bits in nigh-dimenniona. scenario (~ 10"), and bits in practice, as we show in the experiments, we observe that for 2-4 bits are often enough for fine-tuning GPT-2 style model. This leaves interesting direction for future exploration as we expect a careful analysis of sparsity together with more advanced quantization functions make this condition much weaker.
scenario (~ 10"). In

---

Regularization and other optimizers. Assuming A1 and A2 for f. and under further assumptions on b and V, one can prove that the Iz-reeularized loss f(x) f(x) + ||=||| which results in weight decay, satisfies Assumptions Al and A2 with slightly weaker constants. We note that a theoretical analysis for other regularization methods or optimizers such as Adam [39] could be an
independent study and represent an interesting line of future research.
3.3 System Implementations and Optimizations.
Additional storage and communication. AQ-SGD requires us to store, for each data example ç. the quantized activation m(() in a local buffer in memory or SSD. For example, in GPT2-XL. training, simple calculation shows that we need an approximately extra ITB storage. When using data parallelism, reduces to ITB # parallel degree, but also incurs communication overhead if data is shuffled in every epoch. In addition, when the example Ç is sampled again, m({) needs to be (1) loaded from this local buffer to the GPU, and (2) updated when new value for m({) is ready.
Optimization. It is easy to implement and optimize the system such that this additional loading and updating step do not incur significant overhead on the end-to-end runtime. This is because of the fact that the GPU computation time for forward pass is usually much longer than the data transfer time for GPT2-XL with 1.5 billion model parameters, a single forward pass on
load the activations layers require 44 ms, whereas loading mL{) need 0.2 ms from memory and 12 ms from SSD. One can simply pre-fetch m({) right before the forward pass, and hide it within the forward pass of other data examples. Similarly, updating m(g) can also be hidden in the backward computation. It is also simple to reduce the communication overhead by shuffling data only once or less frequently.
Evaluation
We demonstrate that SQD can significantly speed up fine-tuning large language models in slow networks. Specifically, we show: (1) on four standard benchmark tasks, AQ-SQI can tolerate aggressive quantization the activations (2-4 bits) and backward gradients (4-8 bits), without hurting convergence and final loss, whereas direct quantization converges to a worse loss or even diverge, (2) in slow networks, AQ-SGD provide an end-to-end speed-up up to 4.3x, and (3) AQ-SCD can be combined with state-of-the-art gradient compression methods and achieve an end-to-end speed-up of up 4.9x.
4.1 Experimental Setup
Datasets and Benchmarks. We consider both sequence classification and language modeling tasks with -tate-of-the-art foundation models. For sequence classification, fine-tune a 1.5B parameter DeBERTa on two datasets: QNLI and CoLA. For language modeling, we fine-tune the GPT2 model with 1.5B parameter on two datasets: Wiki'Tex12 and arXiv abstracts. All datasets are publicly available and do not contain sensitive or offensive content. Detailed setup can be found in Appendix ??.
Distributed Cluster: We conduct our experiments on AWS with 8-32 p3.pp..22......... 2xlarge instances, each containing V100 GPU. For single pipeline, we partition model onto 8 machines. When combined with data parallelism, we use 32 instances data parallel degree is and pipeline parallel degree is 8. By default, instances are interconnected with 10Gbps bandwidth. We simulate slow networks by controlling the communication bandwidth between instances using Linux traffic control.
Baselines. We compare with several strong baselines:
1. FP32: in which all communications are in 32 bit floating point without any compression.
2. DirectQ 29 30 in which activations and backward gradients are directly quantized.
We use simple, uniform quantization scheme, which first normalizes a given vector into [-1, and quantize each number into a b-bit integers by uniforming partitioning the range [-1, 1] into 26 intervals [35]. Additional details of the configuration can be found in Appendix ??.
A 'we use the 12-xxlarge checkpoint: httpppp///nnggggee.....oooo/o//s we use the extra large checkpoint: tttps::/bugggingtacc.oo///
xxlarge
7

---

666
Driec2 1ea syet
penci (H) Emo
Denett (in beel
Drec0 heds
044
40 500 3x0 vvt)
40-000 1N3 tw
40-520 (wd bxd)
50000 beel
61-
6.0
5.0K
10:K
kc
$50.
1.5
2
25c
200,
4000
900.0
500.
1.06
1.k
step
step
slep
(a) QNLI, DeBERTa-I. 1.B (b) CoLA, DeBERTa- 1.5B (c) WikiTex12, GPT2-1.5B (d) arXiv, GPT2-1.5B
Figure 3: Convergence (loss vs. # steps) of different approaches. x represents divergence.
0-5
Drieci you sutt
Wiici Hou beit 10-500 1Nd tw:)
penest (n2 keay devedd
creatt (nd sed
Drecc hed:
Dreend (wo
40 00000003000 3x0 sudi
AC-52D (w3 bwd
-+- A0-500 (wo hwel
02
00.
so20 Loc
ahe 25c 2.1
60
zbc
wia rie 1922 auds
1101 atio a ania soé
Limelmin
Limelmin
Lime/min
üimemin
(a) QNLI, 500Mbps
(b) QNLI, 100Mbps
(c) CoLA, 500Mbps
(d) CoLA, 100Mbps
2012
1912 Devetlz line
Binning Jist kuei
rerver the (iee
cheviity (icc
AD-CCD
A20200 (Nd
28
2300 uso
230.3
400.1
505.0
LOGD
200.0
2303
300.
(e) WikiTex12, 500Mbps
(f) WikiText2, 100Mbps
(g) arxi, 500Mbps
(h) arxiv 100Mbps
Figure 4: End-to-end training performance over different networks. x represents divergence.
Hyperparameter Tuning. We conduct careful tuning for all methods on all datasets. We perform grid search to choose learning rate from (2.5c-6, 3e-6, 5c-6, le-5} and macco-batch size from (32, 64, 96} for best model performance. We train all models using the Adam optimizer with weight decay.
4.2 Results
Convergence. We first compare the convergence behavior of different methods. For all compression methods, we try various settings: fwr bwy means that we use x bits for forward activation and y bits for backward gradients. Figure 3 shows the convergence behavior for the sequence classification and language modeling tasks. FP32 converges fast since it does not introduce any compression errors. DirectQ, under aggressive quantization, can converge to significantly worse model, or even diverge. This is not surprising, given the biases on model gradients that direct quantization introduced. On the other hand, AQ-SGD converges almost as fast as FP32 in terms of number of training steps.
End-to End Runtime. We show the end-to-end runtime of different methods under slow networks As illustrated in Figure 4 AQ-SGD achieves a 4.3x end-to-end speed-up comparing with that of FP32 (in terms of time to the same loss), illustrating the importance of communication compression in slow networks. Table 2 shows the training throughput and Table 3 shows the breakdown of our algorithm. We note that computation and communication can overlap, so the end-to-end time depends on the larger one of the two. Another interesting observation is that when the network is 100x x slower (from 10Gbps to 100Mbps), the training is only about 1.18x slower! This is exciting if AQDSGD were to be deployed in a in real-world geo-distributed decentralized networks, the training throughput would be almost as fast the performance inside a data center!
Moreover, AQ-SGD does not introduce significant runtime overhead over direct quantization. From Table we that SQ-SGD is essentially efficient direct guantization compression in throughput.
8

---

Table 2: Training Throughput of GPT2-1.5B. Oth- ers are similar and shown in Appendix.
Table Breakdown of AQ-SGD (fw4 bw8) on
GPT2-1.5B. We show the computation and com- munication time of each micro batch.
Network Bandwidth
DirectQ
AQ-SGD
FP32
Fw3 bw6 fw4 bws
3w bwó fw4 hw8
Network Bandwidth
Forward pass
Backward pass
10 Gbps
3.8
4.0
4.1
4.0
4.0
comp.
comm.
comp.
comm.
Gbps
3.2
4.0
4.0
4.0
3.9
500 Mhps
45 ms
13 ms
135 ms
25 ms
500 Mbps
2.7
3.9
3.9
3.9
3,9
300 Mbps
45 ms
21 ms
135
42 ms
300 Mbps
1.8
3.9
3.8
3.8
3.8
200 Mbps
45 ms
31 ms
135 ms
63 ms
100 Mbps
0.5
3.5
3.0
3.4
3.0
100 Mbps
45 ms
63 ms
135 ms
125 ms
5.0
Droat2 tted opd
A- 563 has 1p11
Dein 1055
4.-
3.5
semp Hee
100
2001
30.0
601.0
250.0 500.2
156.0 stop
Lic
zix
15x
stop
126ape
16dps
500ebbs
(a) WikiTex12, GPT2-1.5B
arxiv, GPT2-1.5B
(c) Training Throughput
Figure 5: Convergence and Throughput of AQ-SGD combined with gradient compression.
4.3 End-to-end Communication Compression: AQ-SGD + QuantizedAdam
AQ-SCD can also be combined with existing methods on gradient compression. This allows to compress all communications during training. We combine AQ AQ-SGD with QuantizedAdam [40), an error compensation- based gradient compression algorithm for data parallel training.
We quantize the forward activations with 3 bits, the backward gradient with 6 bits, and model gradient with 4 bits. Figure illustrates the convergence and the training throughput under different network configuration. We see that AQ-SGD converges well when combined with QuantizedAdam (Figure ( b)). On the other hand, DirectQ, when combined with gradient compression, converges to much worse model. In terms of training throughput, with both activation and gradient compression, we can achieve to 8.5x throughput improvement compared to the no-compression baseline (Figure(3(c) We also see that both activation and gradient compression are important in terms of improving end-to-end training throughput - as illustrated in Figure cjc), disabling any of them will lead to much lower training throughput.
5 Related Work
Distributed training of foundation models. Modern distributed training of deep neural networks goes beyond data parallelism [24] 41 42 due to the advance of the large-scale foundation models 20), such BERT 21.. GPT-3 22 and (3]. Popular systems to support foundation model
training include Megatron (24). Deepspeed 25 and Fairscale [26). To scale out the training of large-scale models, pipeline parallelism (e.g., Gpipe [1] Pipedream 43 44 is a popular option, where the model is partitioned into multiple stages, different stages are allocated on different GPUs and the exchange of activations and gradients on activations goes through network communication.
Communication compression of distributed learning. Communication compression is an effective 46
system relaxation for distributed training, especially in data parallelism 44
45
47
4g 49
50 51]
52
13
54
55
Popular techniques include quantization 6 8 sparsification
55
57 58 [59]
sketching 60
61 and error compensation [27]. Recently, TinyScript
30
proposes to compress
activations and gradients simultaneously.
Sparse Learning for activation compression. Sparse learning 30 62
63
66
67
68
69
has become increasingly popular for training neural networks, as can significantly reduce use of computation and memory while preserving the generalization of such models. In particular,

---

activation compression methods B32 33
34
35
36
29
are proposed to reduce the memory footprint
by adopting lossless [70,
74 compression techniques in the training of GNN[ZZ)). These approaches usually compute the activation with full precision forward propagation, adopt the compression method over the activation, and store the compressed version in DRAM for later use in backward propagation. Thus, compression does not introduce any error in forward propagation in contrast to the scenario of communicating compressed activation in the distributed setting.
7T 72 and lossy 73
29
various deep neural networks (e.g., CNN 66
75
76
Delta-based compression. Delta- bastd compression 78
classic solution various system problems. Recent research has also used the property of spatial proximity within activation in neural network training based on empirical observations [79 80 [4]. However, to our knowledge, no attempt has been made to consider the proximity of activation through training epochs to enable efficient compression with theoretical guarantee.
6
Conclusion
In this paper. we discuss how to adopt communication compression for activations in distributed learning. We proposed AQ-SGD, novel activation compression algorithm for communication--fficinnn pipeline parallelism training over slow networks. AQ-SGD achieves O(L/VT) convergence rate for non-convex optimization without making assumptions on gradient unbiasedness. Our empirical study suggests that AQ-SCD can achieve up to 4.3x speedup for pipeline parallelism. When combined with gradient compression in data parallelism, the end-to-end speed-up can be up 4.9x.
Acknowledgments
CZ and the DS3Lab gratefully acknowledge the support from the Swiss State Secretariat for Education. Research and Innovation (SERI) under contract number MB22.00033 (for European Research Council (ERC) Starting Grant TRIDENT 101042665). Swiss National Science Foundation (Project Number 200021 184628, and 197485). Innosuisse/SNF BRIDGE Discovery (Project Number 40B2-0 187132), European Union Horizon 2020 Research and Innovation Programme (DAPHNE, 957407), Botnar Research Centre for Child Health, Swiss Data Science Center, Alibaba, Cisco, eBay. Google Focused Research Awards, Kuaishou Inc.. Oracle Labs, Zurich Insurance, and the Department Computer Science at ETH Zurich. CR gratefully acknowledges the support of NIH under No. U54EB020405 (Mobilize). NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ARI. under No. W91 1011N-1-1-2000 (Interactive Human-Al Teaming); ONR under No. N000141712266 (Unifying Weak Supervision): ONR N00014-20- 1-0000---8800 Understanding and Applying Non-Euclidean Geometry in Machine Learning: 0000142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Govermmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, endorsements, either expressed or implied, of NIH, ONR, or the U.S. This work was done during Jue Wang's 's visit to ETH Zürich, which was supported by Key Research and Development Program of Zhejiang Province of China (No. 2021C01009) and Fundamental Research Funds the Central Universities. The computation required in this work was provided by Together Computer (https://together .xyz)).
References
[1] Max Ryabinin and Anton Gusev. Towards crowdsourced training of large neural networks using decentralized mixture-of-experts. Advances Neural Information Processing Systems, 33:3659-3677 2020.
[2] Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Anton Sinitsin, Dmitry Popov, Dmitry V Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Distributed deep learning in open collaborations. Advances in Neural Information Processing Systems, 34:7879-7897, 2021.
10

---

[3] Alexander Borzunov, Max Ryabinin, Tim Dettmers, Quentin Lhoest, Lucile Saulnier, Michael Diskin, and Yacine Jernite. Training transformers together. In NeurlPS 2021 Competitions and Demonstrations Track, pages 335-342. PMLR, 2022.
[4] Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-effiiee 2021.
[5] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication--ffiiceee sgd via gradient quantization and encoding. arXiv preprint arXiv:1610.02133 2016.
[6] Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Ji Liu, and Ce Zhang. Zipml: Training linear models with end-to-end low precision, and a little bit of deep learning. In International Conference on Machine Learning, pages 4035-4043. PMLR, 2017.
[7] Jeremy Bernstein, Yu- Yu-Xiang Wang. Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference Machine Learning, pages 560-569. PMLR, 2018.
[8] Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li. Terngrad: ternary gradients reduce communication in distributed deep learning. In Proceedings of the s1 International Conference on Neural Information Processing Systems, pages 5588-1518, 2017.
[9] Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference Machine Learning, pages 3478-3487. PMLR, 2019.
[10] Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. Pipe-sgd: decentralized pipelined sgd framework for distributed deep net training. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8056 8067, 2018.
[11] Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? case study for decentralized parallel stochastic gradient descent. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages $336-5346 2017.
[12] Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, pages 0043-3052. PMLR, 2018.
[13] Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, and Ji Liu. Communication compression for decentralized training. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 7663-7673, 2018.
[14] Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. D2: over decentralized data, International Conference on Machine Learning, pages 4848 4856. PMLR, 2018
Decentralized training
[15] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. Proceedings of the VLDB Endowment, 13(12).
[16] Pytorch-lightning.
ai/
[17] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv: 088800007999 2018.
[18] Gan, Shaoduo and Lian, Xiangru and Wang, Rui and Chang, Jianbin and Liu, Chengjun and Shi, Hongmei and Zhang, Shengzhuo and Li, Xianghong and Sun, Tengxu and Jiang, Jiawei and others. BAGUA: scaling up distributed learning with system relaxations. Proceedings of the VLDB Endowment, 15(4):804-813 2021.

---

[19] Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. A unified architecture for accelerating distributed [DNN] training in heterogeneous gpu/epu clusters. In 14th {USENIX) Symposium on Operating Systems Design and Implementation ((OSDI) 20), pages 463-479 2020.
[20]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bossclut, Emma Brunskill, Erik Brynjolfsson. Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and al. On the opportunities and risks of foundation models. CoRR, abs/2108.07255 2021.
[21] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04.00, 2018.
[22] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1990 2020.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[24] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.000 00053,, 2019.
[25] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 3505-3506 2020.
[26] Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, al. Fairscale: A general purpose modular pytorch library for high performance and large scale training, 2021.
[27] Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang, and Ji Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass error- cooor-oopensteedd compression. In International Conference on Machine Learning, pages 6155-6165. PMLR, 2019.
[28] Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce Zhang, Tong Zhang, and Ji Liu. Deep- squeeze: Parallel stochastic gradient descent with double-pass error-compensated compression. arXiv preprini arXiv: 1907.07374 2019.
[29] R David Evans and Tor Aamodt. Ac-ge: Lossy activation compression with guaranteed convergence. Advances in Neural Information Processing Systems, 34, 2021.
[30] Fangcheng Fu, Yuzheng Hu, Yihan He, Jiawei Jiang, Yingxia Shao, Ce Zhang, and Bin Cui. Don't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript. In International Conference on Machine Learning, pages 3304-3314, PMLR, 2020.
[31] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-3 2018
[32] Song Han, Huizi Mao, and William/ Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. 2016.
12

---

[33] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18())6686-6888, 2017.
[34] Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhimenko. Gist: Efficient data encoding for deep neural network training. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pages 776-789. IEEE, 2018.
[35] Ayan Chakrabarti and Benjamin Moseley. Backprop with approximate activations for memory- efficient network training. Advances in Neural Information Processing Systems, 32, 2019.
[36] Jianfei Chen, Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael Mahoney, and Joseph Gonzalez. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning, pages 1803-1813, PMLR, 2021.
[37] Liu and Ce Zhang. Distributed learning systems with first-order methods. arXiv preprint arX1v:2104.05245, 2021.
[38] Ahmad Ajallocian and Sebastian U, Stich. On the convergence of sgd with biased gradients, 2020.
[39] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[40] Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong Hc. 1-bit adam: Communication efficient sarge-scale training with adam's ' convergence speed. In International Conference on Machine Learning, pages 10118 101299 PMLR, 2021.
[41] Yanping Huang. Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukloong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[42] Bowen Yang, Jian Zhang, Jonathan Li, Christopher Ré, Christopher Aberger, and Christopher De Sa. Pipemare: Asynchronous pipeline parallel dnn training. Proceedings of Machine Learning and Systems, 3:269-296, 2021.
[43] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1-15, 2019.
[44] Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory- efficient aapelinee-prallee dnn training. In International Conference on Machine Learning, pages 7937-7947. PMLR, 2021.
[45] Yh Chen, Jiamin Ni, Songtao Lu, Xiaodong Cui, Pin- Yu Chen, Xiao Sun, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Viji Srinivasan, Wei Zhang, et al. Scalecom: Scalable sparsified gradient compression for communication-effiee efficiien distributed training. Advances in Neural Information Processing Systems, 13:13551-13563 2020.
[46] Cong Xie, Shuai Zheng, Sanmi Koyejo, Indranil Gupta, Mu Li, and Haibin Lin. Cser: Communication-efficiie sgd with error reset. Advances in Neural Information Processing Systems, 33:12593- 2020.
[47] Fartash Faghri, Iman Tabrizian, Ilia Markov, Dan Alistarh, Daniel M Roy, and Ali Ramezani- Kebrya. Adaptive gradient quantization for data-parallel sgd. Advances in neural information processing systems, 33:3174-3185, 2020.
[48] Zhize Li, Dmitry Kovalev, Xun Qian, and Peter Richtarik. Acceleration for compressed gradient descent distributed and federated optimization. In International Conference on Machine Learning, pages 5895-5904. PMLR, 2020.
13

---

[49] Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, lon Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-effiieeee federated learning with sketching. In International Conference on Machine Learning, pages 8253-8265. PMLR, 2020.
[50] Mher Safaryan and Peter Richtárik. Stochastic sign descent methods: New algorithms and better theory. In International Conference Machine Learning, pages 9224-9234. PMLR, 2021.
[51] Eduard Gorbunov, Konstantin P Burlachenko, Zhize Li, and Peter Richtárik. Marina: Faster non-convex distributed learning with compression. In International Conference Machine Learning, pages 3788-37909 PMLR, 2021.
[52] Mher Safaryan, Filip Hanzely, and Peter Richtárik. Smoothness matrices beat smoothness constants: better communication compression techniques for distributed optimization. Advances in Neural Information Processing Systems, 34, 2021.
[53] Xun Qian, Peter Richtárik, and Tong Zhang, Error compensated distributed sgd can be acceler- ated. Advances in Neural Information Processing Systems, 34, 2021.
[54] Ahmed M Abdelmoniem. Ahmed Elzanaty, Mohamed-Shie Slim Alouini, and Marco Canini. An efficient statistical based gradient compression technique for distributed training systems. Pro- ceedings of Machine Learning and Systems, 3:297-322, 2021.
[55] Hongyi Wang. Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication- efficient models at no extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 2021.
[56] J Wangni, J Liu, Wang, and T Zhang. Gradient sparsification for communication-effiee efficiin distributed optimization. Advances in Neural Information Processing Systems, 31:1299, 2018.
[57] Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Khirirat, Nikola Konstantinov, and Cédric Renggli. The convergence of sparsified gradient methods. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5977-5987, 2018.
[58] Hongyi Wang, Scott Sievert, Zachary Charles, Shengchao Liu, Stephen Wright, and Dimitris Papailiopoulos. Atomo: communication-efficicn learning via atomic sparsification. In Proceed- ings of the 32nd International Conference on Neural Information Processing Systems, pages 9883, 2018
[59] Jialei Wang, Mladen Kolar, Nathan Srebro, and Tong Zhang. Efficient distributed learning with sparsity. International Conference on Machine Learning, pages 4636-3655. PMLR, 2017.
[60] Jiawei Jiang, Fangcheng Fu, Tong Yang, and Bin Cui. Sketchml: Accelerating distributed machine learning with data sketches. In Proceedings of the 2018 ACM SIGMOD International Conference on Management of Data, pages 1289-1284, 2018
[61] Nikita Ivkin, Daniel Rothchild, Enayat Ullah, lon Stoica, Raman Arora, et al. Communication- efficient distributed sgd with sketching. In Advances in Neural Information Processing Systems, pages 13144-13151 2019.
[62] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1112, 2018.
[63] Md Aamir Raihan and Tor Aamodt. Sparse weight activation training. Advances in Neural Information Processing Systems, 33: 33:15625-15638, 2020.
[64] Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang. Zhao Song. Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. arXiv preprint arXiv:2 /2. 00009, 2021.
[65] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840. 2019.
14

---

[66] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646 4655. PMLR, 2019.
[67] LIU Junjie, XU Zhe, SHI Runbin, Ray CC Cheung, and Hayden KH So. Dynamic sparse train- ing: Find efficient sparse network from scratch with trainable masked layers. In International Conference on Learning Representations, 2019.
[68] Torsten Hoefter, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
[69] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34, 2021.
[70] Esha Choukse, Michael B Sullivan, Mike O'Connor, Mattan Erez, Jeff Pool, David Nellans, and Stephen W Keckler. Buddy compression: Enabling larger memory for deep learning and hpc workloads gpus. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 926-939. IEEE, 2020.
[71] Minsoo Rhu, Mike O'Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W Keckler. Compressing dma engine: Leveraging activation sparsity for training deep neural networks. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 78-91. IEEE, 2018.
[72] Alberto Delmás Lascorz, Sayeh Sharify, Isak Edo, Dylan Malone Stuart, Omar Mohamed Awad, Patrick Judd, Mostafa Mahmoud, Milos Nikolic, Kevin Siu, Zissis Poulos, et al. Shapeshifter: Enabling fine-grain data width adaptation in deep learning. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pages 28-41, 2019.
[73] Sian Jin, Guanpeng Li, Shuaiwen Leon Song, and Dingwen Tao. A novel nemory-efficieen deep learning training framework via error-bounde boundd lossy compression. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 485-487, 2021.
[74] R David Evans, Lufei Liu, and Tor M Aamodt. Jpeg-act: accelerating deep learning via transform-ben based lossy compression. 2020 ACM/IEÉE 47th Annual International Symposium on Computer Architecture (ISCA), pages 860-873, IEEE, 2020.
[75] Georgios Georgiadis. Accelerating convolutional neural networks via activation map compres- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7085-7095, 2019.
[76] Denis Gudovskiy, Alec Hodgkinson, and Luca Rigazio. Dnn feature map compression using learned representation over gf (2). In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[77] Zirui Liu, Kaixiong Zhou, Fan Yang, Li Li, Rui Chen, and Xia Hu. Exact: Scalable graph neural networks training via extreme activation compression. In International Conference on Learning Representations, 2021.
[78] Gennady Pekhimenko, Vivek Seshadri, Onur Mutlu, Michael A Kozuch, Phillip B Gibbons, and Todd C Mowry. Base-delta-immediate compression: Practical data compression for on- chip caches. In 2012 2tt international conference on parallel architectures and compilation techniques (PACT), pages 377-388. IEEE, 2012.
[79] Omar Mohamed Awad, Mostafa Mahmoud, Isak Edo, Ali Hadi Zadeh, Ciaran Bannon, Anand Jayarajan, Gennady Pekhimenko, and Andreas Moshovos. Fpraker: A processing element for accelerating neural network training. In MICRO-54 54th Annual IEFF/ACM International Symposium Microarchitecture, pages 857-869, 2021.
15

---

[80] Andrei Bersatti, Nima Shoghi Ghalehshahi, and Hyesoon Kim. Neural network weight com- pression with nnw-bdi. In The International Symposium Memory Systems, pages 335-340, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] AQ-SGD requires additional
storage to store the activation for each data example. We describe the limitations from the perspectives of algorithm and implementation in Section 3.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Language models increase the inequality acceleration of training may lead to more misuse of language models. Our work also makes positive social impact privacy protection because it can also be applied in split learning as shown in Appendix.
employment suggested by DeepMind. The
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We repeated some of the experiments for at least three times and reported the average in the paper. Details are reported in Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Our system was built on PyTorch and CuPy and was evaluated using publicly available datasets (QNLI, CoLA, WikiTex12 and arXiv abstracts). All of them use open-source licenses and can be used for non- commercial educational purposes.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "QwNp9ps06Dn", "reviewer": "Reviewer_6E4o", "summary": "This paper proposes an activation quantization method, AC-SGD, for the fine-tuning step on language models with slow network set-up. AC-SGD compresses the change of the activations instead of activation values directly. This paper shows this compression method can converge well and it can lead to throughput improvement under slow network systems. ", "questions": "- How about model (tensor) parallelism (ex. Megatron)? \n", "limitations": "- As I mentioned, this paper should describe why they assume this slow network system in detail. As far as I know, it seems to be not common. \n- The experimental results are not sufficient to show that this method works well.\n- Showing profiling results of the fine-tuning process with this method and various speeds of the network would be better for understanding. \n", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This paper doesn’t aim at compressing activations directly. Instead, by compressing the changes of the activation, the proposed algorithm can achieve efficient convergence.\n- With a slow network system (i.e., the portion of communication overheads is quite big during end-to-end distributed fine-tuning process), this algorithm can achieve higher throughput with smaller communication overhead.\n\n\n### Weaknesses\nAs weaknesses of this paper, I have two concerns as below:\n- I’m very confused about what slow networks are. It seems to be hard for me to imagine which scenario this paper assumes, V100 multi-gpus with 100Mbps network.  The only network I can imagine is that single GPUs are distributed and inter-connected with slow ethernet systems. It is so weird. I don’t understand why we should assume extremely slow interconnection between high-end GPUs for fine-tuning large PLMs. \n- For the GPT results in Figure 4, I think showing training loss is not sufficient to prove that the compression method works well for generative language models, especially for fine-tuning steps. Since the pre-trained model consists of an enormous number of parameters and is trained with a large dataset, it is hard to trust the train loss and validation loss for fine-tuning dataset. It tends to be easily overfitted and there is rare correlation between loss results and measured scores. Since there is a limitation on scale of BERT-like models, this paper should show the inference results on generation tasks and large generation models. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "L5QJRzvU-z5", "reviewer": "Reviewer_uCZr", "summary": "This work examines the feasibility to compress the activations for models trained with pipeline parallelism. To this end, the scheme AC-SGD is proposed, which aims to make pipeline-parallel training more communication-efficient over slot networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AC-SGD compresses the changes of the activations. The most novel insight is that: one can still achieve O(1/pT) convergence rate for non-convex objectives under  activation compression, without making assumptions on gradient unbiasedness that  do not hold for deep learning models with non-linear activation functions. The this work shows that AC-SGD can be optimized and implemented efficiently, without additional end-to-end runtime overhead. The evaluations on AC-SGD are shown on fine-tuned language models with up to 1.5 billion parameters, compressing activations to 2-4 bits. AC-SGD provides up to 4.3⇥ end-to-end speed-up in slower networks, without sacrificing model quality.", "questions": "- Can you provide more evidence to back up your simulation setups on the slow work?\n- How your optimizations benefit the overall system (separately and collectively)? ", "limitations": "- I think the paper has good potentials, and it can be strengthened by addressing the two points (raised in the weakness part).\n- I assume the geo-distributed network is a more realistic scenario for this proposed approach. Hence, I would suggest the authors to get more literature on this part, and back up both their motivation and simulation setup better (both qualitatively and quantitatively).\n- Also, I believe breaking down the optimizations in AC-SGD can add more values in this work, so that the readers can better understand how essential your optimizations are. \n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- An interesting insight on pipeline-parallel training.\n- The solution is effective and the optimization space is covered.\n- Extensive evaluation results.\n- Source codes available\n\nWeaknesses:\n- The work may need more rationales upfront to motivate the problems (i.e. slow networks)\n- More breakdown analysis to showcase the effectiveness of the approach\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BdxZIshVqSZ", "reviewer": "Reviewer_XUdm", "summary": "The authors proposed a differential activation quantization scheme with proven convergence, for distributed model-parallel training.  \n", "questions": "- A biggest limitation of differential compression is the necessity for storage of the entire epoch of activations, and second biggest, the uselessness in online learning scenarios where data never repeat.\n- Numerical stability probably needs more validation.  Though more stable than DirectQ, AC-SGD did present non-monotonic training loss, e.g. Fig. 5b.  Given that these are fine-tuning of pre-trained LMs, which are known to be already close in parameter space to convergence, it is questionable whether AC-SGD was stable at initialization for training from scratch.\n- Following the same thought, as the authors mentioned in the manuscript, near-convergence training dynamics is probably more tolerant to aggressive activation quantization.\n- Though SGD convergence proof is in the order, numerically it is usually more complicated in practice when diverse regularizers and different optimizers were used.  Is there any commonly used regularization or optimizers that do not agree with AC-SGD? \n- What is the $Q(\\cdot)$?  Does it involve affine calibration from the differential data statistics?  If so, what's the procedure? \n- What if activation is already in low precision, say in FP16 in mixed-precision training?  Is AC-SGD still better off?  What about FP8?  \n- \"Activation quantization\" is perhaps more explicit and accurate than \"activation compression\".\n", "limitations": "See above.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] Clear presentation in general\n\n[+] Real-world demonstration\n\n[-] Limitation of practical usefulness from storage requirement\n\n[-] Lack of stability study\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5XtAFPegncw", "reviewer": "Reviewer_dFk7", "summary": "Large Language models are trained or finetuned using a combination of pipeline and data parallelism methods. Pipeline parallelism leads to activation and activation gradient being communicated between devices, while data parallelism leads to weight gradients being communicated between devices. A lot of work has focussed on the later, but the former has received less focus. Compressing activations is non trivial as compressing it in a stochastic unbiased way will lead to biases in the gradient  and breaks the unbiasedness assumption made by most gradient compression results. Previous work either failed to provide theoretical guarantees \n\nThis paper proposes AC-SGD, an activation compression algorithm for communication efficient pipeline parallelism. This method helps accelerate pipeline parallelism over slow networks. They provide theoretical guarantees to prove convergence and show that the method can be implemented with minimum runtime overhead (albeit at a huge memory cost). They compare their method with recent work and show better accuracy at lower precision. Depending on the communication BW used as baseline, they can achieve 4.3x end to end speedup.", "questions": "Questions -\n\n1. What was the storage requirement for each of the benchmarks evaluated in the paper?\n2. What motivated the path to explore compressing the delta of activation per sample rather than the activation itself\n", "limitations": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The claims in the paper are evaluated against strong baselines\n2. The method comes with a theoretical guarantee of convergence \n\nWeakness:\n\n1. The idea relies on data being repeated over multiple epochs. This can limit the applicability of this techniques to more important workloads like pre-training and limit to the idea to only fine-tuning of LLM. This reduces the potential impact of the work.\n\n2. The requirement to store activations for new samples in the finetuning datasets makes it harder to scale this technique with larger finetuning datasets", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept