AC-GC: Lossy Activation Compression with Guaranteed Convergence
Authors: R David Evans
Tor Aamodt
Venue: NeurIPS 2021 Poster (2021)
Keywords: DNN training
lossy compression
SGD
convergence
memory efficient training
training
deep learning

Abstract: Parallel hardware devices (e.g., graphics processor units) have limited high-bandwidth memory capacity.
This negatively impacts the training of deep neural networks (DNNs) by increasing runtime and/or decreasing accuracy when reducing model and/or batch size to fit this capacity. Lossy compression is a promising approach to tackling memory capacity constraints, but prior approaches rely on hyperparameter search to achieve a suitable trade-off between convergence and compression, negating runtime benefits. In this paper we build upon recent developments on Stochastic Gradient Descent convergence to prove an upper bound on the expected loss increase when training with compressed activation storage. We then express activation compression error in terms of this bound, allowing the compression rate to adapt to training conditions automatically. The advantage of our approach, called AC-GC, over existing lossy compression frameworks is that, given a preset allowable increase in loss, significant compression without significant increase in error can be achieved with a single training run. When combined with error-bounded methods, AC-GC achieves 15.1x compression with an average accuracy change of 0.1% on text and image datasets. AC-GC functions on any model composed of the layers analyzed and, by avoiding compression rate search, reduces overall training time by 4.6x over SuccessiveHalving. 

AC-GC: Lossy Activation Compression with Guaranteed Convergence
David Evans
Tor M. Aamodt
Dept. of Electrical and Computer Engineering University of British Columbia
Dept. of Electrical and Computer Engineering University of British Columbia
Vancouver, BC V6T 1Z4
Vancouver, BC V6T 1Z4
rdevansšece. ubc..cc..bbbcc..
aamodtêece. .bc.
Abstract
Parallel hardware devices (e.g., graphics processor units) have limited high- bandwidth memory capacity. This negatively impacts the training of deep neural networks (DNNs) by increasing runtime and/or decreasing accuracy when reducing model and/or batch size fit this capacity. Lossy compression is promising approach to tackling memory capacity constraints, but prior approaches rely on hyperparameter search achieve suitable trade-off between convergence and compression, negating runtime benefits. In this paper we build upon recent devel- opments on Stochastic Gradient Descent convergence prove an upper bound on the expected loss increase when training with compressed activation storage. We then express activation compression error in terms of this bound, allowing the compression rate to adapt to training conditions automatically. The advantage of our approach, called AC-GC, over existing lossy compression frameworks that, given preset allowable increase in loss, significant compression without significant increase in error can be achieved with a single training run. When combined with error-bounded methods, AC-GC achieves 15.1 compression with average accuracy change of (. 0.1% on text and image datasets. AC-GC functions any model composed of the layers analyzed and, by avoiding compression rate search, reduces overall training time by 4.6x over SuccessiveHalving.
Introduction
Stochastic Gradient Descent (SGD) has proven efficient and effective for optimizing Deep and Con- volutional Neural Networks (DNNs and CNNs). However, due to deeper and automatically generated networks [20, 22, 44, 52), improvement of accuracy has caused rapid increase in training memory requirements, which are dominated by the temporary storage of activations between the forward and backward pass of the back-propagaan faster training and, thus, more effective research of DNN models and applications. However, doing this by decreasing the batch size has many drawbacks. On parallel processors, such as GPUs, a small batch size can lead poor compute saturation, and reduced training throughput [47]. Smaller batch sizes also introduce errors that impact convergence and accuracy [18]. Over 50GB of memory is required to train some networks, e.g., GPIPE [22].
algorithm [45, 48). Reducing memory consumption leads to
Many works have examined reducing activation storage overheads. Lossy compression of activations in memory can reduce memory footprint without network modifications [6, 14, 25, 27). Error bounded lossy compression (EBC) [27] has bounded activation error, however, it uses an empirical study to select error target. Activations can also be offloaded to external memory (c.g. CPU DRAM), using either an uncompressed link [31, 45] or compressed link [14, 46). Activation compression and offloading have performance overheads from 5% to 60% [7, 14, 25, 27, 45]. Reduced precision
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

training has the side effect of reducing activation size [9, 57]. Finally, restructuring networks to be reversible [17] or efficient scheduling of network layers [8] can reduce memory use.
Prior lossy and reduced precision approaches [6, 9, 14, 25, 51, 57] utilize automated searches or hand-tuning determine compression rates, which increase training time and have the potential to select poor compression/accuracyy trade-offs.. Ideally, an activation compression method has high compression and minimal decrease in trained accuracy. Tuning to achieve this is pro- hibitively expensive. For instance, selecting fixed-point integer (fixpoint) compression rate for ImageNet/ResNet50 using Grid Search uses 16 training runs (Figure 1). Using Successive- Halving [26] can decrease training time, how- ever even with aggressive resource allocations (e.g. GPU-days, Figure 1) total training time is still high. With low resource settings, meth- ods such as SuccessiveHalving [26] and Hyper- band [36] allocate little time to some configura- tions, increasing the likelihood that compression
Grid Search Fixpoint Act. Compr.
Successive Fixpoint Act. Compr.
This Work
Automatic Fixpoint
20
40
Training Time (days, RTX 2080 Ti)
Figure I: Activation compression rate search cost for ImageNe//ResNet00 [20]. Each box indicates a different compression from 1- to 16-bit fixpoint.
artifacts [6. 25] can be missed, resulting poor accuracy. Additionally, when tuning hyperparameters, lossy compression makes it difficult to determine the cause of degraded accuracy. Finally, prior compression methods have unknown impact convergence behavior.
In this work, we present a framework for lossy Activation Compression with Guaranteed Convergence (AC-GC). To our knowledge, our work is the first to prove convergence bounds on SGD with activation compression. AC-GC involves allowing an increase in the bound on the expected loss, which we trade-off for increased compression. We formulate this as a constrained optimization problem: maximizing compression subject to bounded increase in loss. Doing so allows using a single hyperparameter to correlate convergence bounds with the activation error, which creates compression methods that are iteration, network, and dataset agnostic. Having convergence bounds known to set tolerable error rate before training, avoiding compression rate search cost
priori allows entirely. Our contributions:
We prove convergence bounds SGD under error bounded activation compression with weak assumptions on convexity.
We express activation compression and convergence as a constrained optimization problem tolerance of common DNN layers within this framework.
and analyze the activation
We combine these error bounds with fixpoint, image, and error bounded compression, to create methods with guaranteed convergence and a compression/accuraa trade-off known prior training.
2 Preliminaries
DNNs commonly used on problems involving sum, for instance, minimize the total error set of training images. The loss L for such a problem takes the form
(()) E ((0
where / represents the loss of one example input X,, with weights 0.
Stochastic Gradient Descent (SGD) is typically used to optimize these finite sums, using the iteration
g(++1) = g()) - a aVa .X.))
(2)
where is the learning rate, t is the iteration, Vef represents the gradient of f with respect to and n, randomly chosen training example index from the distribution over n such that E (0()), Xx,)) VoL()()))
Figure shows the computation graph for the paoo-propngttion [48] algorithm for a single DNN layer without compression. Back-propagation is often used as it allows efficient calculation of

---

Direct activation storage
Loss
X)
Stored Activations
Forward
fwd
Backward
Parameters
bwd act
b) With activation compression
bwd param
x)
Compressed Activations
Vxfld,
X AX
Inputs
Figure 2: Computation graph for training of a DNN layer. Activations are a) stored between the forward (left) and backward pass (right) or compressed (C) the forward pass, and decompressed (D) in the backward pass. Red indicates paths potentially affected by compression errors.
parameter gradients at the expense of storing activations [48]. A DNN layer is any linear or non-linear function, e.g., a convolution or ReLU activation. The functions fwd, bwd_param, bwd_ bwd_act act are algorithmic implementations of the layer function and the gradients w.r.t. 0 and X. In the forward pass, each layer calculates an output activation Y fwd(X) which is fed to subsequent layers. These activations are temporarily stored after use to avoid performance penalty from recalculating them in the backward pass. To our knowledge, all frameworks opt to store activations [42, 53]. In the backward pass, parameter gradients and activation gradients calculated using bwd_param and bwd_act. Parameter gradients are used to update the parameters (Eqn. (2)), and activation gradients sent downward to the next layer. Depending the layer type and its derivatives, the bwd_ par_paarm and bwd_acc functions may require activations to be stored. For example, it is computationally efficient to store the input X for convolution layers [42, 53]. The many layer types place diverse set of constraints on the activation storage.
Activation compression addresses one of the most significant contributors to memory consumption DNNs. In the forward pass, an activation can lazily compressed after its last usage (C, Figure 2b). Eagerly compressing activations would require storing both compressed and uncompressed copy until its last use. In the backward pass, activations are decompressed before their first use (D, Figure 2b). The backward pass begins only after the forward pass is completed for all layers, resulting in large reuse distance for stored activations. Compression can thus be performed off the critical path in parallel with compute, with low performance overheads from 4%-30% [7, 25], provided that sufficient resources are available.
We denote the uncompressed activation as X = (Inehw) E R XxCxxxX where N, C, H, and W represent the batch size, channel, height and width, respectively. In the uncompressed backward pass, gradients are calculated from the saved activations, parameters, and gradients from the upward layer (Figure with a); we write this as Vol (0, X) bwd_param/ X,0, 0, Vy f(0, X)).
Lossy compression involves discarding some information of the activation to increase the compression rate. In information theory, this is referred as the rate-distortion trade-off. our model, the rate refers to the activation error, and the distortion refers to resulting impacts on gradient error and thus accuracy after training. We model lossy compression between the forward and backward pass as an independent perturbation on each value in the activation, AX E R NxxxXXXx the compressed backward pass, the perturbed activation is X + AX. We denote the approximate gradient resulting from lossy error as (0, X)= = bud_ aad_parammX" X + AX, 0, f0, X)), and the corresponding gradient error as A of X)= == Vof(e. (0,
Thus, in
Guaranteed Convergence
This section details how gradient error AVof(0, impacts convergence of SGD. Following this, the lossy compression error AX can be expressed in terms of the gradient error, and the compression rate for many methods can be determined. For example, the bitwidth b of fixpoint compression of an activation with range (-1, is
b logg AInehwol
(3)
3

---

Fast Convergence
Convergence to within CzaV?
1-bit Fixpoint
+ yygav
Loss 2(ol!)
Loss
2-bit Fixpoint
3-bit Fixpoint
Uncompressod
c(dl+)
10
a) Early training
Late training
c) Late training Lossy Compression
Epochs
d) CIFARIO/ReSNet50 Training
Figure 3: SGD convergence behavior. a) and b) without compression, c) this work, and d) ResNet50 training with 0.25 and fixpoint activation compression (average over five runs, shaded indicates the minimum and maximum training loss).
where the remaining terms are constants determined by the rounding mode, sign, etc. Any compres- sion method with bounded activation error for a given rate can be combined with the error bounds from this work to guarantee convergence (Sections 3.2 and 4).
3.1 SGD Convergence
We will briefly summarize uncompressed convergence of SGD from Karimi et al. [28]. Consider training using iterations of SGD with loss L((()),, with a constant learning rate a and initial point g(o), We assume that L has an optimal point 8(+) and satisfies Vel xx,))l < V2 for all and some V We refer to V as the variance. With some assumptions and problem-defined constants C1 and C2 (Appendix A), Karimi et al. [28] demonstrate that the expected error at iteration t
[[C()(()) - )( s (1 C.a)) (E(( - L(((+)) + C2aV2
Initially when training, fast convergence occurs as (1 Cia) approaches zero (Figure 3a). Later in training, C2 12 dominates, resulting in an approximately constant expected error (Figure 3b). The key observation of this result that the final loss scales with gradient variance, E[[(((x)) x V2.
Many DNN classes fall under this progress bound as it uses relatively weak assumptions and does not require convex DNNs using ReL.U activations Lipschitz continuous [55]. Furthermore, those with an 1.2 loss are piecewise strongly convex, which implies that the required Polyak-LLojssewicc condition is satisfied locally [40]. Networks this does not apply to could use another progress bound [3, 28, 41].
3.2 Compressed Convergence
Lossy compression trades accuracy for compression. This trade-off can be empirically observed with fixpoint compression on CIFAR 1 (Figure and Section 6). Our method functions by allowing the loss increase by multiplicative error (1 e²), where e² > 0, and determining how much compression can extracted from the change (Figure 3c). The error + e² is chosen such that the compressed loss converges to the uncompressed loss c² 0.
Our process for translating the loss bound into maximum activation errors is illustrated in Figure 4a. This section outlines how the loss can be bounded by bounding the gradient error using intermediate bounding function D(AX) to simplify the problem. We follow this Section by deriving an activation error (+) for each network layer that satisfies this bound. e² becomes the sole hyperparameter in our method, and selection determines the maximum increase in loss and the compression rate.
Compressed convergence with increased loss can be viewed as an uncompressed problem with an increased gradient variance bound, (1 + e²) To determine the activation error which satisfies this gradient variance, we must express the variance in terms of bound on the gradient error /AVo/(0) X) Theorem demonstrates that maximum gradient error e²/² satisfies the (1+e2) loss bound. There may be multiple regions where the gradient error is below e² (Figure 4b), which would require iterative solvers to determine suitable activation errors. apply our technique in practice we introduce convex function D(AX) > a (0, x] which provides a flexible

---

Loss Bound, 1+
Pey-Layer Equations
Grad. Ertor
D(Az)
Implicd
Used choose
Section 3.2
ev2
Gradient Variance Bound
Bounding Function, DIAX)
Implied by
1AV.J x)ll
Gradient Firor Bound
[(, 222
Constrained Optimization
Act. Error Ar
Ax
Maximum Activation Error Compression Rate
a) Flowchart Derivations
b) Bounding Function Example
Figure 4: a) Flowchart of derivations for obtaining the maximum activation error AX (+) as function of the loss bound 1 + e². b) ID Example for the relationship between the error bound, bounding function, and gradient eтror for an activation error Az. Shaded area satisfies the loss bound, and the hatched area satisfies D(Az) S e²V².
proxy for the gradient error (Figure 4b). Using D(AX) allows defining the problem so that it has unique solution. The gradient variance bound, bounding function, and gradient error related in Theorem I.
Theorem 1. Given f which obeys (4), and convex D(aX) which bounds the gradient error from above for all X, 8, and AX:
L.Vef(0, 5
(5)
then any activation error A (+) where D(Ax(") < V22 satisfies
Vl )1 e
(6)
assuming that [E[1 AV of (..))l = 0 for all with positive value e?, and variance V2 satisfying of X,,, 11 S V2. All expectations taken over the training examples n4-
Proof: See Appendix A.
Figure 4b demonstrates the relationship between the various bounds, well as the motivation for bounding function. The gradient error derived from per-layer equations, and in many cases is highly non-convex. Due to this, local minima can cause difficulties when optimizing with the constraint (e, X3I1 I e2v² (Figure 4b). D(AX) is defined to be convex, making the region defined by the constraint closed region (hatched, Figure 4b). Although the original constraint could tolerate a higher activation error (and thus compression), using a bounding function provides favorable conditions for obtaining closed-form solution for the activation error.
4 Framework for Activation Compression
For our evaluation of AC-GC, we develop activation error bounds for various commonly used DNN layers and apply them several recent networks. This section summarizes bounds for common layers, and derivations and additional layer bounds are provided in Appendix B, Calculating compression eтror from the gradient error bound e²v² requires expressing and solving for the compression/converennc trade-off. We tackle this by formulating the trade-off as constrained optimization problem: maximizing the compression subject to bounded gradient error. The problem must be solved once per layer type and can be formally defined as
AX* argmax B(AX) s.t. D(AX) = e2 AX
where D(AX) (0,
(8)
where B(AX) a continuous convex function that measures compression rate as a function of
activation error AX E XBX A closed-form solution can be found for many systems of this type using the method of Lagrange multipliers. The constraint D(AX) S e²V² defines convex region of potential activation errors where the convergence constraint is satisfied (hatched, Figure 4b).
5

---

However, as both the B(AX) and D(AX) functions are convex, the maximum value must oceur along the boundary, hence the equality constraint ((AX)= convexity of D(AX) also implies that Ax(") is the maximum activation error, i.e., any error (AX)² < (Ax(+))² also satisfies the variance bound (6).
= e2 in (7) (Az!*), Figure 4b). The
Many compression methods use variation of fixpoint. Hence, we select B to measure the number of bits removed from the activation when compressed with reduced precision fixpoint (9). Rounding mode and sign constant factors that do not affect the result, however, ignore clipping. The target compression method loosely influences the objective, hence, non-fixpoint methods may fare better with another error objective.
N.C.H.W
B(AX)=
Σ
log ATnehwl
(9)
To derive AC-GC error bounds for DNN layers not presented in this work, one can:
1. Derive
x2 for the layer type
2. Choose suitable convex bounding function, D(AX) N [0)of(0,X]]]
3. Obtain the maximum error (+ by solving (7) using the method of Lagrange multipliers
For the sake of brevity, we will summarize the notation, assumptions, and AC-GC error bounds for fully connected, convolution, and batch normalization layers (Table 1). We aim to locate closed- form solutions with low computation overheads, although tighter bounding functions likely exist. Henceforth we omit arguments of f and use the following definitions: Batch size N, input channels C, output channels K, input and output activations X and Y, and compression error AX.
A) Fully Connected: Table 1A relates the error for guaranteed convergence with compression error for fully connected layer, with weights 0 = (0ke) E R R input activation X = (Inc) E NxC, and output activation gradient y (0f E RRN As the error bound (e21/2/2 decreases, the compression must decrease to compensate. All activations for fully connected layer have the same error tolerance.
B) Convolution: Convolution with no padding follows a similar trend to linear layers, with the addition of stride T, filter size of R S, and increased dimensions X (Inchw) IR N and V (af )0ynkhu)) E B X We assume an average usage of activations due to stride, as uneven usage leads to uneven compression, which would require tracking per-element compression rates. Comparing linear and convolution reveals that convolutions have lower error tolerance due to the increased number of activations (HW) and weights (RS).
To fully cover cases encountered in CNNs, we also derive error bounds for cases where activation errors affect multiple convolution layers (e.g., in ResNets [20]).
C) Batch Normalization: Batch normalization [24] re-normalizes the activation from per-channel standard deviation E RC to learned E RC There is different dependence on activation error from convolution and linear layers. Instead of causing errors in the parameter gradient v,f exclusively, activation error propagates to the activation gradient Vx f and then all subsequent layers in the network. To avoid bounding all weights in the network, we isolate the layer and bound the convergence of the batch normalization activations using
| < 222
(10)
Arriving at a closed-form solution further requires a bound on the parameter gradient error within the layer using positive values (o?) 2 (V, (Appendix B). Although not observed for the networks in this work, as network parameter gradients are not directly bounded, the convergence bounds on networks with batch normalization may be violated.
Layer Normalization: This layer type is similar to batch normalization and requires similar set of assumptions and derivations (Appendix B).
ReLU, Dropout, Max Pooling, and Summation: Summation does not require storage of any activation, and the remaining layers (Dropout, Max Pooling, and ReLU) only require a bitmask to calculate their respective gradients. For instance, ReLU requires the storage of the bitmask X 2 [14, 25), and Max Pooling requires bitmask of the locations of maximal values. As these layers have an efficient lossless high compression method available, do not analyze them.

---

Table 1: Guaranteed convergence equations for common network layers. See Appendices for full derivations and assumptions. Empty sums are over all indicies, i.c. Σ Emchh M := NHW
N.C.H.W
^) FULLY CONNECTED
c) BATCH NORMALIZATION
QUANTITY
B) CONVOLUTION
D(AX):
RS T
Σ
Mo!! Arhchw
e²v2
2V222
eV
Ar (r)2))
(+)2
2NCIVyyfI
2
CreggE
5 Practical Automatic Lossy Compression
Using the AC-GC error bounds, create convergence bounded compression methods with auto- matic compression rates, collectively referred to as AutoX. The first two methods (AutoQuant and AutoCuSZ) adapt scaled fixpoint [14] and error-bounded compression [27], which have bounded errors for a given compression rate. The errors for these methods are unbiased provided that unbiased rounding fixpoint is used [5, 27]. The third method (AutoJPEG) uses lossy JPEG compression [14]. We bound JPEG error using an empirical error-compression relationship using activations sampled from uncompressed training of SEFFARO//ReNeete [20]. Samples used offline with JPEG compression to establish the compression-crrv relationship. This empirical compression-error relationship used to calculate the JPEG compression levels, which approximately satisfy the error bounds in Table 1. We chain quantization and JPEG with lossless Zero Value Compression [46] to compress sparse activations better, creating AutoQuantZ and AutoJPEGZ.
As all AutoX used some form of fixpoint, we can express activation error in terms of bits. For any layer type, the relationship between bitwidth b and the convergence bound can be described
b 4 logg nenhw + ... logg
loga Vefll +...
(11)
Using the results from Table 1, it can be seen that the bitwidth scales additively with the batch size as + (IV)(NV) for convolution layers, and (N) for normalization layers.
Two issues with using AC-GC in compression method are: 1) the various norms required are computationally expensive, and 2) the formulation assumes exact gradient information is available during the forward pass. We address both issues by statistically estimating activation error bounds. Instead of evaluating at every iteration, statisties are calculated at the end of every recalculation interval, specified in iterations. In the forward pass, a summary (mean maximum) of the last ten recalculations is used when calculating the errors. This also allows approximating V x [[ Although some quantities are available in the forward pass, we estimate all of the activations, parame- ters, and gradients to avoid performing norm calculations at every iteration. Despite norms being estimated, we do not observe that the convergence bound (6) is violated for any network examined. A few training iterations can be used to verify correctness of the norm estimates (Figure 5c).
Evaluation
We examine activation compression by modifying the Chainer framework [53] to compress and decompress activations during training. We measure compression rates every 100 iterations, and otherwise perform paired compressinndeccopressooo to maintain the highest performance for our experiments. We focus our analysis on CNNs with image and text datascts, as they have large activation memory requirements, but avoid the largest networks [22, 52] due to limited resources. We create a performance implementation based off Chen et al. [7] to measure throughput.
For ImageNet [11], 10FAR10 [2] and Div2K [1], we use SGD and 0.9 momentum for VGG16 [50]. ResNets (RN18 and RN50) [20], Wide ResNet (WRN) [59], and VDSR [29]. IMDB [39] and Text Copy [4] are trained using ADAM with CNN [53], RNN [53], and transformer heads [54]. All image datasets are augmented with random sizing, flip, and crop, as well as whitening and PCA for ImageNet [30], and 8 x 8 cutout for CIFAR 10 [12]. Learning rates, batch sizes, and epochs are 0.05, 128, 300 (CIFARTO, [49]. 0.1, 64, 105 (ImageNet, [58]), 0.1, 32, 110 (Div2K, grid search), 2.0, 64, 100 (Text Copy, [4]), and 0.001, 64, 20 (IMDB, [53]).
7

---

Lass (2)
(1 (1-e))
AQ max
AQ mean
Loss @ Epoch 10
Epoch 10
0.015
Variance Ratio
I !| |V/l?
0.03
1.0
TD
0.014
Loss @
0.02
0.013
0.8
-2
2
10 10'
10
10"
10"
20
40
50
60
70 80
90100
a)
Change in AQ bitwidth (-logle))
b)
Recaleulation Interval (iterations)
Epoch
Figure 5: a-b) Average loss over the 10th epoch MNIST/LeNet, where shaded regions indicate the min/max loss over five runs. a) Loss as function of e. Under compression, the empirical loss (AQ max: AutoQuant with maximum summary) falls below the theoretical loss bound (1 e²) b) Loss where statistics are calculated every recalculation interval iterations. The max or mean the last intervals is used to calculate the bitwidth. c) Ratio of compressed uncompressed gradients for the first convolution for ImageNet/ResNet16 where the compressed gradient is used to update the parameters. Clusters indicate the first 50 iterations of every epochs.
All forward and backward pass calculations use mhating--point using activations that have been compressed and decompressed between the two passes. Baseline refers to uncompressed training, i.e., 32-bit floating-poin point activations. GridQuantZ uses the same implementation of AutoQuantZ, however, uses grid search over eight bit-widths of 2, 3, 4, 6, 8, 10, 12 and 16 bits, and then chooses the lowest with accuracy within 0.1% % of the baseline. These grid points were selected to give good coverage of low and medium bit- bidths, and are approximately and Hyperband [36] produce similar accuracy/compression otherwise stated, all experiments use e² 0.5, parameter estimates from the mean of a ten entry window, and recalculation interval of 100 iterations. A value of e2 0.5 allows small increase in loss over the baseline (+50%), but other values could be chosen depending on the desired error tolerance. Appendices C and D contain additional detail on hyperparameters and implementations.
searrtthmm--spaced SuccessiveHalving [26] grid search but take less time. Unless
6.1 Parameter Sensitivity
We isolate the impacts of parameter estimation and selection by training LeNet [33] on MNIST [32], and RN18 [20] on ImageNet [11]. We train MNIST to convergence by using SGD with momentum, a learning rate of 0.001, a batch size of 64, and 10 epochs.
The effect of recalculation interval, and summary method are evaluated by training MNIST/Lenet with different configurations of AutoQuant (AQ) (Figures 5a and 5b). Decreasing (which increases bitwidth) causes the loss to increase, however, the average loss does not violate the bound in Theorem I (Figure 5a). In general, all networks examined with e² 0.5 have loss changes below 2% and validation score changes below 0.5% (Sections 6.2 and 6.3), which demonstrates that AC-GC error bounds are not violated. Both interval and summary method have a minimal impact on the training loss for MNIST (Figure 5b). As there is an insignificant change in loss, we the mean for AC-GC as it has higher compression.
We evaluate ImageNet/ResNet18 using a dual training approach to ensure the correctness of parameter estimation (Figure 5c). This involves training using AutoQuant, while evaluating the true weight gradients V offline and comparing against their compressed counterpart Vaf. The ratio is 1 for the duration of training, and does not violate the bound for e? = 0.5 (i.e., S 1.5). The mean ratio is less than onc, likely due to decreased activation variance from compression to a discrete set of values. We observe similar behavior for the other layers in the network (not shown).
6.2 CIFARIO, Div2K and IMDB
We compare the AutoX methods with fixpoint grid search (GridQuantZ), and with prior works on lossy JPEG compression [14] (Table 2). Grid search requires oracle knowledge of the baseline, and 8x the training iterations of any other method in Table 2. Compared to GridQuantZ, AutoQuantZ. uses single run of training, and provides similar compression rate of 7.5x. AutoCuSZ has

---

Table 2: Test/validation score and compression rate (bracketed) for fixpoint with grid search (GridQuantZ), AutoX methods, and JPEG-ACT (optLSH from [14]), averaged over runs (Im- ageNet) or runs (remainder). The highest aceuracy and compression are bolded. Trained using 900 GPU-days (RTX 2080 Ti). N/A indicates either not run in [14], or lack of spatial activations.
AUTO
AUTO CUSZ
AUTO JPEGZ
JPEG- ACT[14]
GRID QUANTZ
MODEL
BASE
QUANTZ
CIFAR10
% Tor-1 TEST ACCURACY
VGG
93.6
93.5 (7.4x)
93.5 (9.4x)
(2.9
(11.9))
93.5 (6.3x)
RNS0
94.9
95.0 (4.2x)
94.7 (15.5x)
94.3 (9.2%)
94.4 (7.5x)
95.0 (5.7x)
WRN
95.8
95.9 6.5x)
95.8 (14.6x)
95.3
(1.2
96.0 (7.6x)
DIV2K BEST VAL. PSNR
VDSR
36.1
36.1 (5.1x)
35.8 (25.2x)
36.1 (7.9x)
35.4 (9.1x)
36.0 ( 6.7x)
IMDB
BEST VAL. ACCURACY
CNN
61.4
61.6 (12.2x)
61.8 (19.3x)
61.4 (11.2x)
N/A
61.7 (16.5x)
LSTM
60.3
60.1 (10.0x)
60.9 ( (.8x)
N/A
N/A
60.4 (14.7x)
TEXT COPY % BEST TEST ACCURACY
TRANS
98.8
98.6 (..6(7.) (7.1x)
98.3 (12.7x)
N/A
N/A
98.9 (5.1))
IMAGENET % Top-1 CENTER CROP VAL. ACCURACY
RNI8
68.6
68.5 (4.2x)
68.1 (6.8x)
68.1(8.1X)
67.3 (7.2x)
68.5 (2.9x)
RNSO
72.3
72.7 (4.8x)
72.5 (10.1x)
71.5 (8.5x)
71.6 5.9x)
72.5 (4.9))
AVERAGE 4-POINT CHANGE AND COMPRESSION RATIO
ALL
0
+0.0 (7.5x) -0.1 (15.1x) -0.6 (10.5x)
-1.0 (7.4))
+0.1 (7.8x)
compression 2.0x higher than JPEG-ACT, while maintaining accuracy to within 0.1 of the baseline on average. With suitable error bound, CuSZ can extract significant compression from zeros and spatial information. On non-spatial data and non-image datasets (Text Copy and IMDB, Table 2) we observe that AutoCuSZ extracts similarly high compression with little accuracy change. Finally, AutoJPEGZ vs. JPEG-ACT demonstrates that using AC-GC error bounds gives higher accuracy and compression than using heuristics to select JPEG hyperparameters. In general, we find that using 22 0.5 decreases accuracy, leaving little reason to modify it.
6.3 ImageNet
On ImageNet training with ResNets (Table 2), the AutoX methods obtain high accuracy. Our ImageNet accuracies are lower than other works as we do not use random scaling (which improves performance), and we report 1-crop accuracy. Our 10-crop accuracy for the ResNet50 baseline is 75.2%. Reduced compression rates on ResNet18 vs. ResNet50 are due to lower sparsity in ReLU activations, which we hypothesize is caused by the different bottleneck structures of the two networks [20]. The higher-than basalle accuracy of AutoQuantZ (Table 2) is caused by a large standard deviation for the ImageNet baseline (+..21). On ImageNet, AutoCuSZ gives a high compression in exchange for small decrease in accuracy, 0. 1.%5%--osss.. AutoQuantZ, provides high accuracy, at moderate compression rate of 0.7x, with 1.2%--poi better accuracy than JPEG-ACT. However, the primary advantage of AutoX methods is that the pre-training bound on loss increases.
6.4 Overheads
The AutoX methods require eтror bound calculation (common to all methods) and compression.
Our unoptimized AutoQuantZ implementation achieves throughput of 1.66x (N = 128) vs. naive swapping to the CPU (N 128), and 0.64x vs. uncompressed training (N 32) (ImageNet/ ResNet50). Our primary contribution, AC-GC error bound calculation, uses 0.4% of total training time. This negligible when compared to compression overhead, e.g., 4% for fixpoint [14, 25), 17%
for CuSZ [27], 33% for AcNNN-L [7], or 13% for hardware accelerated JPEG [14]. Unoptimized AutoQuantZ 23% slower compared to ActNN-L.3 [7]. Compression rate search time is decreased by x.6x when compared to SuccessiveHalving (Figure 1).

---

Table 3: Comparison with prior works on CIFARIO (C10) and ImageNet (IN). + indicates standard
deviation, if available. Accuracy is presented relative to the baseline accuracy of each work. Does not include 2 mememrr reduction from recalculating activations, which is orthogonal to this work.
AUTO
WAGE [57]
BAA [6]
JPEG-
ULP
EBC [27]
ACTNN -L([7]
CUSZ
ACT[14]
[51]
DATASET
IN
C10
IN
IN
IN
IN
IN
MODEL
RN50
VGGI6
RN152
RN50
RNS0
RN50
RN152
METHOD
AUTO
8-BIT
4-BIT
JPEG
4-BIT
CUSZ
2-BIT
-CuSZ
+RECALC.
+ZVC
Mix. PREC.
Acc. (%)
-.2
-0.
-0.5
-0.1
-0.3
-0.9
-02
COMPR.
10.1x
4x
8x
5.9x
8x
11.0x
12x
7
Related Works
We compare AutoCuSZ with the most recent works in activation compression [6, 14, 25, 27] and reduced precision training [51, 57] (Table 3). AutoCuSZ obtains better accuracy and compression than most works, with the added benefit of not requiring a search over compression rates. BAA [6] presents technique of recalculating activations similar to Chen et al. [8]. which could combined with any method in Table 3. including the AutoX methods. AutoCuSZ obtains higher compression
and accuracy than EBC [27], demonstrating that AC-GC error bounds are better than hand-tuning in this case. WAGE [57] and ULP [51] and reduced precision training with theoretical convergence bounds [35, 37] reduce training compute through reduced precision, however, have lower accuracy due to accumulating errors in both the forward and backward pass. AutoX methods can potentially be combined with low precision training and other techniques to reduce memory requirements further.
ActNN is most similar to this work in that it invokes gradient variance to partition bits among compressed activations [7]. However, ActNN approaches the problem in an opposite manner this work: it finds the highest accuracy for given compression rate. The ActNN approach works well when the goal is to fit within a memory budget. Our approach allows for selecting the target loss increase priori, and gaining information on the uncompressed accuracy from a single training run. AC-GC will also converge similarly to the uncompressed case on an unknown model, providing assurances that loss changes are due to mooel/hyyprpprrane Additionally, ActNN is specific to group-wise fixpoint compression, whereas AC-GC quick to adapt to any lossy compression method (including group-wiss fixpoint). On lmageNet/Resseet, AutoCuSZ and AutoQuantZ obtain half the accuracy change of ActNN-L3, albeit at 0.9x and 0.4x the compression, respectively.
configuration, not compression.
Works targeting inference, such as precision reduction [10, 15, 23, 34), compression [13, 19, 38, 56), and sparsification [16, 43] increase memory requirements due to tracking additional state. Other works that directly address memory can be grouped into scheduling [8, 25], offloading [31, 45), and restructuring [17, 21]. Generally, these come with performance Any non-lossy method is partially orthogonal to compression and can be potentially combined with AC-GC for increased memory reduction.
penalty,
8 Conclusions
The AC-GC automatic compression methods described in this work provide high compression rates with error trade-offss known before training. Avoiding compression rate search comes computational overhead of only 0.4%, many times less than tuning techniques. Provided that the assumptions of AC-GC hold, it can be further combined with any lossy compression method, layer, or network to guarantee convergence with high activation compression. Although convergence is theoretically guaranteed, some factors could impact convergence, such floating-point errors. By lowering training costs, this framework allows for training larger models and faster exploration of the machine learning field's landscape. Detailed derivations and implementations can be found in the supplemental material. Code is available https: //gi thub. com / rwt.//dddaannaccc....
10

---

Funding Transparency Statement
This research was funded in part by the Computing Hardware for Emerging Intelligent Sensory Applications (COHESA) project financed under the National Sciences and Engineering Research Council of Canada (NSERC); grant number NETGP485577-15.
References
[1] Agustsson, E. and Timofte, R. NTIRE 2017 Challenge on Single Image Super-Resolution
Dataset and Study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Pp. 1122-1131, July 2017. doi: 10. D0.109/CVPRW.2007.12
[2] Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. PhD thesis, University Toronto, Toronto, Canada, 2009.
[3] Bach, F. and Moulines, E. Non-strongly-conver smooth stochastic approximation with conver- gence rate O(1/n). Advances in Neural Information Processing Systems, 26:773-781, 2013,
[4] Bartz, C. Shainer-transformer http://ghhab.oommorrrr/cccccccat ENSE, January 2021.
[5] Burks, A. W., Goldstine, H. H., and Neumann, J. Preliminary Discussion of the Logical Design of an Electronic Computing Instrument. In Randell, B. (ed.), The Origins of Digital Com- puters: Selected Pupers. Texts and Monographs in Computer Science, pp. 399-413, Springer, Berlin, Heidelberg, 1982. ISBN 978-3-642-618 6112-3. doi: 10.
[6] Chakrabarti, A. and Moseley, B. Backprop with Approximate Activations for Memory-effin Network Training. Advances in Neural Information Processing Systems, 32, 2019.
[7] Chen, J., Zheng, L., Yao, Z., Wang, D., Stoica, I., Mahoney, M. W., and Gonzalez, J. E. Actnn: Reducing training memory footprint via 2-bit activation compressed training. arXiv:2104. 14129 [cs.LG], April 2021.
[8] Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06177 [cs]. April 2016.
[9] Courbariaux, M., Bengio, Y., and David, J.-P. Training deep neural networks with low precision [cs], December 2014.
multiplications. 1412:
[10] Courbariaux, M., Bengio, Y., and David, J.-P. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 28, pp. 3123-
3131. Curran Associates, Inc., 2015.
[11] Deng, J.. Dong, W. Socher, R., Li, L., Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image datahase.
2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, June 2009. doi: 10. RBWCCVPR.200952066220
[12] DeVries, T. and Taylor, G. W. Improved Regularization of Convolutional Neural Networks with Cutout. arXiv: 17X8v 04552 [cs], November 2017.
[13] Ding, C.. Liao, S., Wang. Y., Li. Z., Liu, N., Zhuo, Y., Wang. C., Qian, X., Bai, Y., Yuan, G., Ma, X., Zhang, Y., Tang. J., Qiu, Q.. Lin, X., and Yuan, B. CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-circulant Weight Matrices. In Proceedings of the 50th Annual IEEF/ACM International Symposium on Microarchitecture, MICRO-50 '17 pp. 395 408, New York, NY, USA, 2017. ACM. ISBN 378-1-4503-499 doi: 10.1145/3123939, 3124552.
[14] Evans, R. D., Liu, L., and Aamodt, T. M. JPEG-ACT: Accelerating Deep Learning via Transform-based Lossy Compression. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pp. 660-873, May 2020. doi: 10.1109//CCA44567.2222 00075.
[15] Fang, J., Shafiee, A., Abdel-Aziz, H., Thorsley, D., Georgiadis, G., and Hassoun, J. H. Post- training Piecewise Linear Quantization for Deep Neural Networks. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.-M. (eds.), Computer Vision - ECCV 2020, Lecture Notes in Computer Science, pp. 69-86, Cham, 2020. Springer International Publishing. ISBN 978-3--88 030--58635-- doi: 10.1007/

---

[16] Frankle, J. and Carbin, M. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. arXiv:1803.0388 0333333..6355/0 [cs], March 2019.
[17] Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. The Reversible Residual Network: Backpropagation Without Storing Activations. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30, pp. 2214-2224. Curran Associates, Inc., 2017.
[18] Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L.,, Kyrola, A., Tulloch, A., Jia. Y., and He, K. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.
[cs), June 2017.
[19] Han, S., Mao, H., and Dally, W. J. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv: [cs], October 2015.
[20] He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. 11Xv::1522003885 [cs]. December 2015.
[21] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv: 04861 [cs], April 2017.
[22] Huang, Y.. Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee. H.. Ngiam, J., Le, Q. V., Wu, Y., and Chen, z. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Wallach, H., Larochelle, H., Beygelzimer, A., d'textquotesingle Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp. 103 112. Curran Associates, Inc., 2019.
[23] Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized Neural Networks. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29, pp. 4107-4115 Curran Associates, Inc., 2016.
[24] loffe, S. and Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning, pp- 448-456. PMLR, June 2015.
[25] Jain, A., Phanishayee, A., Mars, J., Tang, L.,, and Pekhimenko, G. Gist: Efficient Data Encoding for Deep Neural Network Training. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 776-789, June 2018. doi:
201 1.18..000 8.00770.
[26] Jamieson, K. and Talwalkar, A. Non-stochastic Best Arm Identification and Hyperparameter Optimization. 1rXiv:1502.0733
[cs, stat), February 2015.
[27] Jin, S., Li, G., Song, S. L., and Tao, D. A Novel Memory-Efficient Deep Learning Training Framework via Berror-Bounded Lossy Compression. arXiv:2011. T.V::2011.00017 1.09017 [cs], November 2020.
[28] Karimi, H., Nutini, J., and Schmidt, M. Linear Convergence of Gradient and Proximal-Gradient Methods Under the Dolyak-Lojasiewicz Condition. In Frasconi, P., Landwehr, N., Manco, G., and Vreeken, J. (eds.), Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, pp. 795- 811, Cham, 2016. Springer International Publishing. ISBN 0000077788-331--411111 1_50.
978-3-319-46125 1. doi:
[29] Kim, J., Lee, J. K., and Lee, K. M. Accurate Image Super-Resolute Using Very Deep Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition, pp. 1646-1654, 2016.
[30] Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1, NIPS' 12, pp. 1097-1 105, Red Hook, NY, USA, December 2012. Curran Associates Inc.
[31] Kwon, Y. and Rhu, M. Beyond the memory wall: A case for memory-centric HPC system for deep learning. In Proceedings of the 5/ Annual IEEF/ACM International Symposium on Microarchitecture, MICRO-51, pp. 148-161, Fukuoka, Japan, October 2018. IEEE Press. ISBN 978-1-5386-62400 doi: 10.1109/MCCR02200 2018 8.0021..
[32] LeCun, Y. and Cortes, C. MNIST handwritten digit database. http://yann.lecu..comm/ee 2010.
12

---

[33] Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2224,, November 1998. ISSN 1558-2256. doi: 10.1 109/577667..
[34] Li, F., Zhang, B., and Liu, B. Ternary Weight Networks. arXiv:1605.04711 [cs], May 2016.
[35] Li, H., De, S., Xu, Z.,, Studer, C., Samet, H., and Goldstein, T. Training quantized nets: A deeper understanding. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 5813-5823, Red Hook, NY, USA, December 2017. Curran Associates Inc. ISBN 978-1-5108-6096-
[36] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1 116765-6816, January 2017. ISSN 1532-4435.
[37] Li, Z. and De Sa, C. M. Dimension-Free Bounds for Low-Precision Training. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[38] Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W. J. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. arXiv: 1tt.111122.88888 [cs, stat), December 2017.
[39] Maas, A. L., Daly, R. E., Pham, T., Huang, D., Ng. Y., and Potts, C. Learning word
vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies Volume HLT '11, pp. 150, USA, June 2011 Association for Computational Linguistics. ISBN 978- 1-932432--8---9
[40] Milne, T. Piecewise Strong Convexity of Neural Networks. Advances in Neural Information Processing Systems, 32: 12:12290 1993-112999, 2019.
[41] Nedié, A. and Bertsekas, D. Convergence Rate of Incremental Subgradient Algorithms. In Uryasev, S. and Pardalos, P. M. (eds.), Stochastic Optimization: Algorithms and Applications, Applied Optimization, pp. 223-264. Springer US, Boston, MA, 2001. ISBN 978- 177-1-4757-6594-6 doi: 10.1007/978-11 1-77/9978-1--7777-594----111
[42] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.., Bai, J., and Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems, 0888026-837, 2019.
[43] Raihan, M. A. and Aamodt, T. M. Sparse Weight Activation Training. arXiv:2001.01969 [cs, stat], October 2020.
[44] Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regularized Evolution for Image Classifier Architecture Search. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 4780-4789, July 2019. ISSN 2374-3468. doi: 10.1609/aaii 333301.3300111111 333014780.
[45] Rhu, M., Gimelshein, N., Clemons, J., Zulfiqar, A., and Keckler, S. W. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design, arXiv:1602.08/24
[cs], July 2016.
[46] Rhu, M., O'Connor, M., Chatterjee, N., Pool, J., Kwon, Y., and Keckler, S. W. Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp.
78 11, February 2018. doi: 10. N0.1109/PCA.2218.00000
[47] Rogers, T. G., Johnson, D. R., O'Connor, M., and Keckler, S. W. A variable warp size archi- tecture. In 2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA), pp. 489-501, June 2015. doi: 10. D0.1145/2744499.22258 2255411..
[48] Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning representations by back- propagating errors. Nature, 323(6088):533-536 October 1986. ISSN 1476-4687. doi: 10. 1038/ 323533a0.
[49] Saito, S. Chainer-cifar10: Various CNN models including for CIFARIO with Chainer. https///ittub..oommiiii /chainer-cifar10, 10, May 2018.
[50] Simonyan, K. and Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.155 [cs], September 2014.
13

---

[51] Sun, X., Wang, N., Chen, C.-Y., Ni, J. Agrawal, A., Cui, X., Venkataramani, S., El Maghraoui, K., Srinivasan, V. V., and Gopalakrishnan, K. Ultra-low precision 4-bit training of deep neural networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F, and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1796-1807. Curran Associates, Inc., 2020.
[52] Tan, M. and Lc, Q. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In International Conference on Machine Learning, pp. 6105-6114. PMLR, May 2019.
[53] Tokui, S., Okuta, R., Akiba, T., Nitaani, Y., Ogawa, T., Saito, S., Suzuki, S., Uenishi, K., Vogel, B., and Yamazaki Vincent, H. Chainer: Deep Learning Framework for Accelerating the Research Cycle. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pp. 2002-20111 New York, NY, USA, July 2019, Association for Computing Machinery. ISBN 978-1-4503-6201-6 doi: 10.1145/3292500. 3330756.
[54] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[55] Virmaux, and Scaman, K. Lipschitz regularity of deep neural networks: Analysis and efficient estimation. Advances in Neural Information Processing Systems, 31:3835-3844 2018.
[56] Wang, Y., Xu, C.. You, S., Tao, D., and Xu, C. CNNpack:
Packing Convolutional Neural Networks in the Frequency Domain. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29, pp. 253-261. Curran Associates, Inc., 2016.
[57] Wu, S., Li, G., Chen, F., and Shi, L. Training and Inference with Integers in Deep Neural Networks. In International Conference on Learning Representations, February 2018
[58] Wu, W. Reproduce ResNet-v2 (Identity Mappings in Deep Residual Networks) with MXNet. https///ittuu..com/ tornadomeet/ResNeet January 2021.
[59] Zagoruyko, S. and Komodakis, N. Wide Residual Networks. arXiv: [cs), May 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 6
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [No] This work has the positive impact of decreasing training times, which decreases overall carbon footprint and allows for research of models which benefit society. However, depending on how these models are used there is a possibility of negative impacts such surveillance, privacy concerns, or job loss.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3, Appendix A, and the assumptions from Karimi et al. [28], Section 3.3
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section
6, Appendices C and D, and the code available at https: // /gi thub. com/ com/rdevans0/ acgc
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6 and Appendix D
14

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix due to space limitations
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Table 2. This work used an internal cluster.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the original works for all datasets, models and hyperparameters in Section 6
(b) Did you mention the license of the assets? [No] All are public use
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent
obtained from people whose data you're using/curating? [No] This is discussed in the original publications on these datasets (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] This is discussed the original publications on these datasets
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] This work does not involve human subjects
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "wHviSZ_ALP", "reviewer": "Reviewer_coLe", "summary": "This paper studies activation compression to reduce the memory consumption when training models with stochastic gradient descent. The goal is to establish an activation compression framework with convergence guarantees, while eliminate the tuning overhead from conventional search based method for compression precision selection. The contributions of this paper come in three folds:\n\n1. The authors first established a connection between the activation compression error and the convergence bound (i.e. the upper bound on training loss).\n\n2. Built on the above theory, the authors formulate the activation compression precision selection problem as a constrained optimization problem, which can accommodate typical DNN layers such as linear layers, convolution layers and normalization layers.\n\n3. Empirically, the authors combine this optimization-based activation precision selection method with various compression methods (with bounded errors at each compression precision). They show that it could attain matching accuracy (minimal acc degradation compared to no activation compression) compared to conventional grid search method for activation precision without the grid search (i.e. hyperparameter tuning) overhead.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nPrevious activation compression works typically use time-consuming hyperparameter tuning methods to select compression precision. This paper studies the connection between activation error and convergence, and explores a novel approach to automatically select activation compression precision. I think the theoretical analysis, the methods brings new thoughts and methodology to the area.\n\n\n\nQuality:\n\nMy major concern/question on the paper is about the claim that the proposed activation precision selection approach *does not need any hyperparameter tuning* to achieve minimal accuracy loss.  The basis of claiming no tuning is to uniformly use a fixed e^2 value (e^2 = 1 is used in the paper) across models and datasets. Theoretically, the limit of the loss bound in equation (13) can increase by 50% if e^2 = 1 is introduced due to activation compression, which could trigger significant loss / accuracy degradation; such a case can be instantiated using a quadratic function.  Totally understand e^2 = 1 could potentially be a good hands down option for a broad range of SOTA models. But to validate this, it needs more model/datasets (such as transformer based models for vision and NLP) using e^2=1 for activation precision while the current experiments only cover convolutional models.\n\nBesides the major above question, it would be great if the authors can resolve the following questions in the rebuttal and paper draft:\n\n1. Theorem 1 (line 141) assumes the expectation of the gradient error norm is zero. Does this means the theorem assumes there is no gradient error at all (because the assumption is only satisfied when the activation compression does not trigger any gradient change)? If this is the case, the assumption is too unrealistic. Also from the proof in the appendix A, it is unclear how this assumption is leveraged in the proof.\n\n2. For the constrained problem in equation (7), the presentation needs to be improved to clearly define the problem. From the proofs in Appendix B, I believe equation (7) is actually referring to \\Delta X* = \\min_{b} B(\\Delta X), s.t. D(\\Delta X) < e^2V^2 / 2. The current formulation is not rigorously reflecting the optimization problem.\n\n3.  (Not a major concern) In table 3, it seems the grid searched QuantZ is giving 0.7% higher accuracy on ImageNet than the no compression baseline. Is it because of random seed variation? Or is it because the no compression baseline is not well tuned to optimize validation accuracy (The typical ResNet 50 ImageNet validation top 1 accuracy is around or around 75% to 80% [1] while the baseline results in the paper is giving 72-73% top 1 accuracy)?\n\n\nClarity: I think the paper explain intuitions and general ideas very well. But technical clarity (such as terminology definition and etc.) should be improved in several places.\n\n1. In equation 3, is \\Delta x_{nchw} a function of compression precision b. If this is the case, it is unclear how this function (defined on discretized supports) is used in the continuous constrained problem in equation (7).\n\n2. In line 161, the range of \\Delta X is not properly defined if \\Delta X is a multi dimensional vector. My guess is you mean a ball centered at 0 with radius \\Delta X*.\n\n3. In line 252, there is a claim on \"all networks examined have loss changes below 2%\". It needs a bit clarification in the text because the loss change at -log|e| is definitely larger than 2% given loss changes from ~0.015 to ~0.025.\n\n\nSignificance: If the proposed methods can show strong accuracy in broader models / tasks (such as NLP or vision transformers) still with a uniform value for hyperparameter e^2, the impact can be significant to eliminate / dramatically reduce the tuning overhead on compression precision selection.\n\n\nI am willing to raise the score if the above concerns are resolved.\n\n\nReference\n\n[1] Deep Residual Learning for Image Recognition. He et. al.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the attached checklist answers the questions adequately. The overhead / limitation of the method is discussed in Section 6.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see anything improper or having ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "CgWzw8FMfG", "reviewer": "Reviewer_7589", "summary": "The paper proposes a theoretical convergence upper bound for DNN training with compressed activation. Using this bound, the compression rate is controlled over training epochs by computing the quantization error. Moreover, to reduce the complexity of computing the activation’s quantization error, the paper uses statistics of this error such as mean and max instead of computing the complex norm. The efficiency of this new approach (AC-GC) is evaluated in various benchmarks including the ImageNet dataset and compared with previous approaches such as WAGE, GIST, and JEPG-ACT. The result showed that the AC-GC compressed the activation more than other approaches while maintaining accuracy compared to FP32. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1- The idea behind the constraint (activation precision) convergence rate under the Polyak-Łojasiewicz Condition is novel and interesting. The author uses a common Lagrange multiplier approach to optimize the constraint.\n\n2- The paper is well organized. The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.\n\n3- The proposed approach is evaluated in large and variant benchmarks, which mean the proposed approach can be deployed for different models \n\nWeakness:\n\n1-  The author did not compare the new approach with previous papers such as [1] and [2], where the theatrical convergence rate is proposed for low-precision DNN parameters and activations. It would be suggested the comparison with these works is discussed in the paper.\n\n2-  The author needs to add more clarity on the motivation behind storing the activation ( not cited two frameworks ) and provide the assumptions to satisfy the Polyak-Łojasiewicz Condition in the paper. \n3- Is this the convergence rate depends on the dimension of activation? How does the dimension of activation affect the compression rate? \n\n4- The last sentence of the abstract is a bold statement, and I would suggest changing it. The AC-GC has shown valuable results for the experiments presented in the paper, and it is hard to conclude that the approach works for any model. \n\n5-  It would be suggested to compute and report the approximation error using the max and mean of activation error instead of an accurate norm.\n\nReferences: \n\n[1] Li, Z., & De Sa, C. (2018). Dimension-free bounds for low-precision training (https://papers.nips.cc/paper/2019/hash/d4cd91e80f36f8f3103617ded9128560-Abstract.html)\n\n[2] Li, Hao, et al. \"Training quantized nets: A deeper understanding.\" Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.\n\nPost Rebuttal ======\nThanks for the clarification. I think the response resolved my comments on the clarity and quality part. I raised my score from 6 to 7. I encourage authors to add their responses to my comments to the various parts of the papers to improve the quality of the paper. I also suggest that emphasize the weak assumption of convexity in the abstract or introduction.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"ethical_concerns": "There are not ethical issues with this paper", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24"}, "extra_text": {}}, {"review_id": "BNwKnnpkrws", "reviewer": "Reviewer_H5sM", "summary": "The authors bound the increase in loss in the case that activations used for backpropagation have errors. By doing so, they can set a fixed acceptable level of increase in the loss, and solve for the error level that can be tolerated for each activation used in backpropagation (and therefore deduce a level of compression). This analysis is relatively agnostic to the precise implementation of the compression. They evaluate their approach on a variety of datasets, including relatively large datasets such as ImageNet, demonstrating that their method can achieve high compression levels with relatively little tuning.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Detailed Feedback\n\nAbstract: I think the authors did a good job here. Motivated well, and clear description.\n\nIntro: The motivation is good, and well explained (as with the abstract). I think that fig1 is also nicely presented.\n\nPreliminaries:\n- Throughout the paper, you seem to use $\\phi$ for weights, which is a little unusual. Might be better to use $\\theta$ to be more consistent with other works?\n- A purely stylistic observation: It might be good to move the related work here, and motivate why what you're doing is different to that. This is my preference (but I am not going to consider this with regard to scoring)\n- L91: There's a comment here about compression not slowing down optimization too much; I note that this claim is made later too. But what evidence is there for this assertion?\n\nS3:\n- This entire section is written fairly well, with nice figures and useful intuition provided. But the one thing I would note is that I am _not_ an expert on this area, and as such I find it difficult to provide much technical criticism -- it all seems reasonable to me. Nothing stands out to me as being obviously questionable.\n\nS4:\n- This section was more difficult to follow than the last one -- it is very dense. If it's possible to provide any figures it would be very appreciated by readers.\n- It might be a good idea to discuss information theory and its relation to compression, along with lagrange multipliers in the preliminaries?\n- L199: the discussion of max pooling has been forgotten (although it is very simple).\n\nS5:\n- L210: could expand on how you collected this data?\n\nS6\n- The caption for fig5 is not useful, and you have to search for the interpretation in the text. You may want to push the explanation to the caption.\n- The results in table 2 are great. It would be appreciated if you could add some results for training time though :-)\n- The results for table 3 are less impressive, unfortunately (although they have been achieved in far less time -- you undersell that in the table). Also, what about AutoCuSZ and AutoJPEG? The results don't seem particularly impressive when compared with [1]\n- (I don't expect this for rebuttal) It'd be really great if the authors could look at providing some experiments using a transformer. This might be a fair bit of work -- such as deriving bounds for layer norm -- but would make this work more complete.\n- L287: can you provide numbers for how long compression takes?\n\nS7:\n- It would be a good idea to cite ActNN [1] that was recently published at ICML 21.\n\n### Overall Thoughts\n\nI quite like this paper overall, as there is a (seemingly) sensible theoretical analysis, and it is generic to the particular choice of compression. The problem area is extremely important, and this work will be of wide interest. I think the only real downside is that the results obtained are not favorable when compared to [1] (at least on ImageNet), although it is not possible to make a direct comparison. The authors may be able to deal with this by swapping in a different compression backend on the ImageNet experiments. I would ask the authors to address some of the concerns above -- especially the ones regarding overheads. I am sympathetic that the implementation is likely to be far from optimal, but it is important information to provide.\n\n==== Post rebuttal\n\nI'm happy with the author's response, and I have increased my score accordingly :-)\n\n[1] ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think it is fine as it is.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was a discrepancy between the overall scores initially, the authors' rebuttal letter clarified several concerns of the reviewers, which resulted in all the scores being positive. Hence, I recommend an acceptance. Please implement all the changes suggested by the reviewers in the camera-ready version.
Decision: Accept (Poster)