AC-GC: Lossy Activation Compression with Guaranteed Convergence
Year: 2022

Abstract:
Parallel hardware devices (e.g., graphics processor units) have limited highbandwidth memory capacity. This negatively impacts the training of deep neural networks (DNNs) by increasing runtime and/or decreasing accuracy when reducing model and/or batch size to fit this capacity. Lossy compression is a promising approach to tackling memory capacity constraints, but prior approaches rely on hyperparameter search to achieve a suitable trade-off between convergence and compression, negating runtime benefits. In this paper we build upon recent developments on Stochastic Gradient Descent convergence to prove an upper bound on the expected loss increase when training with compressed activation storage. We then express activation compression error in terms of this bound, allowing the compression rate to adapt to training conditions automatically. The advantage of our approach, called AC-GC, over existing lossy compression frameworks is that, given a preset allowable increase in loss, significant compression without significant increase in error can be achieved with a single training run. When combined with error-bounded methods, AC-GC achieves 15.1× compression with an average accuracy change of 0.1% on text and image datasets. AC-GC functions on any model composed of the layers analyzed and, by avoiding compression rate search, reduces overall training time by 4.6× over SuccessiveHalving.

heading: 1 Introduction
text: Stochastic Gradient Descent (SGD) has proven efficient and effective for optimizing Deep and Convolutional Neural Networks (DNNs and CNNs). However, due to deeper and automatically generated networks [20, 22, 44, 52], improvement of accuracy has caused a rapid increase in training memory requirements, which are dominated by the temporary storage of activations between the forward and backward pass of the back-propagation algorithm [45, 48]. Reducing memory consumption leads to faster training and, thus, more effective research of DNN models and applications. However, doing this by decreasing the batch size has many drawbacks. On parallel processors, such as GPUs, a small batch size can lead to poor compute saturation, and reduced training throughput [47]. Smaller batch sizes also introduce errors that impact convergence and accuracy [18]. Over 50GB of memory is required to train some networks, e.g., GPIPE [22].
Many works have examined reducing activation storage overheads. Lossy compression of activations in memory can reduce memory footprint without network modifications [6, 14, 25, 27]. Error bounded lossy compression (EBC) [27] has bounded activation error, however, it uses an empirical study to select an error target. Activations can also be offloaded to an external memory (e.g. CPU DRAM), using either an uncompressed link [31, 45] or compressed link [14, 46]. Activation compression and offloading have performance overheads from 5% to 60% [7, 14, 25, 27, 45]. Reduced precision
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
training has the side effect of reducing activation size [9, 51, 57]. Finally, restructuring networks to be reversible [17] or efficient scheduling of network layers [8] can reduce memory use.
Prior lossy and reduced precision approaches [6, 9, 14, 25, 51, 57] utilize automated searches or hand-tuning to determine compression rates, which increase training time and have the potential to select poor compression/accuracy trade-offs. Ideally, an activation compression method has
high compression and minimal decrease in trained accuracy. Tuning to achieve this is prohibitively expensive. For instance, selecting a fixed-point integer (fixpoint) compression rate for ImageNet/ResNet50 using Grid Search uses 16 training runs (Figure 1). Using SuccessiveHalving [26] can decrease training time, however even with aggressive resource allocations (e.g. 24 GPU-days, Figure 1) total training time is still high. With low resource settings, methods such as SuccessiveHalving [26] and Hyperband [36] allocate little time to some configurations, increasing the likelihood that compression
artifacts [6, 25] can be missed, resulting in poor accuracy. Additionally, when tuning hyperparameters, lossy compression makes it difficult to determine the cause of degraded accuracy. Finally, prior compression methods have an unknown impact on convergence behavior.
In this work, we present a framework for lossy Activation Compression with Guaranteed Convergence (AC-GC). To our knowledge, our work is the first to prove convergence bounds on SGD with activation compression. AC-GC involves allowing an increase in the bound on the expected loss, which we trade-off for increased compression. We formulate this as a constrained optimization problem: maximizing compression subject to a bounded increase in loss. Doing so allows using a single hyperparameter to correlate convergence bounds with the activation error, which creates compression methods that are iteration, network, and dataset agnostic. Having convergence bounds known a priori allows a user to set a tolerable error rate before training, avoiding compression rate search cost entirely. Our contributions:
• We prove convergence bounds on SGD under error bounded activation compression with weak assumptions on convexity.
• We express activation compression and convergence as a constrained optimization problem and analyze the activation error tolerance of common DNN layers within this framework.
• We combine these error bounds with fixpoint, image, and error bounded compression, to create methods with guaranteed convergence and a compression/accuracy trade-off known prior to training.

heading: 2 Preliminaries
text: DNNs are commonly used on problems involving a sum, for instance, minimize the total error on a set of training images. The loss L for such a problem takes the form
L(θ) = ∑ n f(θ,Xn) (1)
where f represents the loss of one example input Xn with weights θ.
Stochastic Gradient Descent (SGD) is typically used to optimize these finite sums, using the iteration
θ(t+1) = θ(t) − α∇θf(θ(t), Xnt) (2)
where α is the learning rate, t is the iteration, ∇θf represents the gradient of f with respect to θ, and nt is a randomly chosen training example index from the distribution over n such that E[∇θf(θ(t), Xnt)] = ∇θL(θ(t)). Figure 2a shows the computation graph for the back-propagation [48] algorithm for a single DNN layer without compression. Back-propagation is often used as it allows efficient calculation of
parameter gradients at the expense of storing activations [48]. A DNN layer is any linear or non-linear function, e.g., a convolution or ReLU activation. The functions fwd, bwd_param, bwd_act are algorithmic implementations of the layer function and the gradients w.r.t. θ and X . In the forward pass, each layer calculates an output activation Y = fwd(X) which is fed to subsequent layers. These activations are temporarily stored after use to avoid a performance penalty from recalculating them in the backward pass. To our knowledge, all frameworks opt to store activations [42, 53]. In the backward pass, parameter gradients and activation gradients are calculated using bwd_param and bwd_act. Parameter gradients are used to update the parameters (Eqn. (2)), and activation gradients are sent downward to the next layer. Depending on the layer type and its derivatives, the bwd_param and bwd_act functions may require activations to be stored. For example, it is computationally efficient to store the input X for convolution layers [42, 53]. The many layer types place a diverse set of constraints on the activation storage.
Activation compression addresses one of the most significant contributors to memory consumption in DNNs. In the forward pass, an activation can be lazily compressed after its last usage (C, Figure 2b). Eagerly compressing activations would require storing both a compressed and uncompressed copy until its last use. In the backward pass, activations are decompressed before their first use (D, Figure 2b). The backward pass begins only after the forward pass is completed for all layers, resulting in a large reuse distance for stored activations. Compression can thus be performed off the critical path in parallel with compute, with low performance overheads from 4%-30% [7, 25], provided that sufficient resources are available.
We denote the uncompressed activation as X = (xnchw) ∈ RN×C×H×W , where N , C, H , and W represent the batch size, channel, height and width, respectively. In the uncompressed backward pass, gradients are calculated from the saved activations, parameters, and gradients from the upward layer (Figure 2 with a); we write this as∇θf(θ,X) = bwd_param(X, θ,∇Y f(θ,X)). Lossy compression involves discarding some information of the activation to increase the compression rate. In information theory, this is referred to as the rate-distortion trade-off. In our model, the rate refers to the activation error, and the distortion refers to resulting impacts on gradient error and thus accuracy after training. We model lossy compression between the forward and backward pass as an independent perturbation on each value in the activation, ∆X ∈ RN×C×H×W . Thus, in the compressed backward pass, the perturbed activation is X + ∆X . We denote the approximate gradient resulting from lossy error as ∇̂θf(θ,X) := bwd_param(X + ∆X, θ,∇Y f(θ,X)), and the corresponding gradient error as ∆∇θf(θ,X) := ∇̂θf(θ,X)−∇θf(θ,X).

heading: 3 Guaranteed Convergence
text: This section details how gradient error ∆∇θf(θ,X) impacts convergence of SGD. Following this, the lossy compression error ∆X can be expressed in terms of the gradient error, and the compression rate for many methods can be determined. For example, the bitwidth b of fixpoint compression of an activation with range (−1, 1) is
b ≥ − log2 |∆xnchw|+ ... (3)
where the remaining terms are constants determined by the rounding mode, sign, etc. Any compression method with bounded activation error for a given rate can be combined with the error bounds from this work to guarantee convergence (Sections 3.2 and 4).

heading: 3.1 SGD Convergence
text: We will briefly summarize uncompressed convergence of SGD from Karimi et al. [28]. Consider training using t iterations of SGD with loss L(θ(t)), with a constant learning rate α and initial point θ(0). We assume that L has an optimal point θ(∗) and satisfies E[‖∇θf(θ,Xnt)‖2] ≤ V 2 for all θ and some V 2. We refer to V 2 as the variance. With some assumptions and problem-defined constants C1 and C2 (Appendix A), Karimi et al. [28] demonstrate that the expected error at iteration t is
E[L(θ(t))− L(θ(∗))] ≤ (1− C1α)t(L(θ(0))− L(θ(∗))) + C2αV 2 (4)
Initially when training, fast convergence occurs as (1− C1α)t approaches zero (Figure 3a). Later in training, C2αV 2 dominates, resulting in an approximately constant expected error (Figure 3b). The key observation of this result is that the final loss scales with gradient variance, E[L(θ(∞))] ∝ V 2. Many DNN classes fall under this progress bound as it uses relatively weak assumptions and does not require a convex f . DNNs using ReLU activations are Lipschitz continuous [55]. Furthermore, those with an L2 loss are piecewise strongly convex, which implies that the required Polyak-Łojasiewic condition is satisfied locally [40]. Networks this does not apply to could use another progress bound [3, 28, 41].

heading: 3.2 Compressed Convergence
text: Lossy compression trades accuracy for compression. This trade-off can be empirically observed with fixpoint compression on CIFAR10/ResNet50 (Figure 3d and Section 6). Our method functions by allowing the loss to increase by some multiplicative error (1 + e2), where e2 ≥ 0, and determining how much compression can be extracted from the change (Figure 3c). The error 1 + e2 is chosen such that the compressed loss converges to the uncompressed loss as e2 → 0. Our process for translating the loss bound into maximum activation errors is illustrated in Figure 4a. This section outlines how the loss can be bounded by bounding the gradient error or using an intermediate bounding function D(∆X) to simplify the problem. We follow this in Section 4 by deriving an activation error ∆X(∗) for each network layer that satisfies this bound. e2 becomes the sole hyperparameter in our method, and selection determines the maximum increase in loss and the compression rate.
Compressed convergence with increased loss can be viewed as an uncompressed problem with an increased gradient variance bound, (1 + e2)V 2. To determine the activation error which satisfies this gradient variance, we must express the variance in terms of a bound on the gradient error ‖∆∇θf(θ,X)‖2. Theorem 1 demonstrates that a maximum gradient error of e2V 2 satisfies the (1+e2) loss bound. There may be multiple regions where the gradient error is below e2V 2 (Figure 4b), which would require iterative solvers to determine suitable activation errors. To apply our technique in practice we introduce a convex function D(∆X) ≥ ‖∆∇θf(θ,X)‖2, which provides a flexible
proxy for the gradient error (Figure 4b). Using D(∆X) allows defining the problem so that it has a unique solution. The gradient variance bound, bounding function, and gradient error are related in Theorem 1. Theorem 1. Given f which obeys (4), and a convex function D(∆X) which bounds the gradient error from above for all X , θ, and ∆X:
‖∆∇θf(θ,X)‖2 ≤ D(∆X) (5)
then any activation error ∆X(∗) where D(∆X(∗)) ≤ e2V 2 satisfies
E[‖∇̂θf(θ,Xnt)‖2] ≤ (1 + e2)V 2 (6)
assuming that E[‖∆∇θf(θ,Xnt)‖] = 0 for all θ, with positive value e2, and variance V 2 satisfying E[‖∇θf(θ,Xnt)‖2] ≤ V 2. All expectations are taken over the training examples nt. Proof: See Appendix A.
Figure 4b demonstrates the relationship between the various bounds, as well as the motivation for the bounding function. The gradient error is derived from per-layer equations, and in many cases is highly non-convex. Due to this, local minima can cause difficulties when optimizing with the constraint ‖∆∇θf(θ,X)‖2 ≤ e2V 2 (Figure 4b). D(∆X) is defined to be convex, making the region defined by the constraint a closed region (hatched, Figure 4b). Although the original constraint could tolerate a higher activation error (and thus compression), using a bounding function provides favorable conditions for obtaining a closed-form solution for the activation error.

heading: 4 Framework for Activation Compression
text: For our evaluation of AC-GC, we develop activation error bounds for various commonly used DNN layers and apply them to several recent networks. This section summarizes bounds for common layers, and derivations and additional layer bounds are provided in Appendix B. Calculating compression error from the gradient error bound e2V 2 requires expressing and solving for the compression/convergence trade-off. We tackle this by formulating the trade-off as a constrained optimization problem: maximizing the compression subject to bounded gradient error. The problem must be solved once per layer type and can be formally defined as
∆X∗ = argmax ∆X B(∆X) s.t. D(∆X) = e2V 2 (7)
where D(∆X) ≥ ‖∆∇θf(θ,X)‖2 (8)
where B(∆X) is a continuous convex function that measures compression rate as a function of the activation error ∆X ∈ RN×C×H×W . A closed-form solution can be found for many systems of this type using the method of Lagrange multipliers. The constraint D(∆X) ≤ e2V 2 defines a convex region of potential activation errors where the convergence constraint is satisfied (hatched, Figure 4b).
However, as both the B(∆X) and D(∆X) functions are convex, the maximum value must occur along the boundary, hence the equality constraint D(∆X) = e2V 2 in (7) (∆x(∗), Figure 4b). The convexity of D(∆X) also implies that ∆X(∗) is the maximum activation error, i.e., any error (∆X)2 ≤ (∆X(∗))2 also satisfies the variance bound (6). Many compression methods use a variation of fixpoint. Hence, we select B to measure the number of bits removed from the activation when compressed with reduced precision fixpoint (9). Rounding mode and sign are constant factors that do not affect the result, however, we ignore clipping. The target compression method loosely influences the objective, hence, non-fixpoint methods may fare better with another error objective.
B(∆X) := N,C,H,W∑ n,c,h,w log |∆xnchw| (9)
To derive AC-GC error bounds for DNN layers not presented in this work, one can:
1. Derive ‖∆∇θf(θ,X)‖2 for the layer type 2. Choose a suitable convex bounding function, D(∆X) ≥ ‖∆∇θf(θ,X)‖2
3. Obtain the maximum error ∆X(∗) by solving (7) using the method of Lagrange multipliers
For the sake of brevity, we will summarize the notation, assumptions, and AC-GC error bounds for fully connected, convolution, and batch normalization layers (Table 1). We aim to locate closedform solutions with low computation overheads, although tighter bounding functions likely exist. Henceforth we omit arguments of f and use the following definitions: Batch size N , input channels C, output channels K, input and output activations X and Y , and compression error ∆X .
A) Fully Connected: Table 1A relates the error for guaranteed convergence with compression error for a fully connected layer, with weights θ = (θkc) ∈ RK×C , input activation X = (xnc) ∈ RN×C , and output activation gradient∇Y f = (∂f/∂ynk) ∈ RN×K . As the error bound (e2V 2/2) decreases, the compression must decrease to compensate. All activations for a fully connected layer have the same error tolerance.
B) Convolution: Convolution with no padding follows a similar trend to linear layers, with the addition of stride T , a filter size of R× S, and increased dimensions X = (xnchw) ∈ RN×C×H×W and ∇Y f = (∂f/∂ynkhw) ∈ RN×K×H×W . We assume an average usage of activations due to stride, as uneven usage leads to uneven compression, which would require tracking per-element compression rates. Comparing linear and convolution reveals that convolutions have a lower error tolerance due to the increased number of activations (HW ) and weights (RS).
To fully cover cases encountered in CNNs, we also derive error bounds for cases where activation errors affect multiple convolution layers (e.g., in ResNets [20]).
C) Batch Normalization: Batch normalization [24] re-normalizes the activation from per-channel standard deviation σ ∈ RC to a learned γ ∈ RC . There is a different dependence on activation error from convolution and linear layers. Instead of causing errors in the parameter gradient ∇γf exclusively, activation error propagates to the activation gradient ∇Xf and then to all subsequent layers in the network. To avoid bounding all weights in the network, we isolate the layer and bound the convergence of the batch normalization activations using
‖∆∇Xf‖2 ≤ e2V 2 (10) Arriving at a closed-form solution further requires a bound on the parameter gradient error within the layer using positive values (g2c ) ≥ (∇̂γf)2 (Appendix B). Although not observed for the networks in this work, as network parameter gradients are not directly bounded, the convergence bounds on networks with batch normalization may be violated.
Layer Normalization: This layer type is similar to batch normalization and requires a similar set of assumptions and derivations (Appendix B).
ReLU, Dropout, Max Pooling, and Summation: Summation does not require storage of any activation, and the remaining layers (Dropout, Max Pooling, and ReLU) only require a bitmask to calculate their respective gradients. For instance, ReLU requires the storage of the bitmask X ≥ 0 [14, 25], and Max Pooling requires a bitmask of the locations of maximal values. As these layers have an efficient lossless high compression method available, we do not analyze them.
Table 1: Guaranteed convergence equations for common network layers. See Appendices for full derivations and assumptions. Empty sums are over all indicies, i.e. ∑ := ∑N,C,H,W
n,c,h,w . M := NHW
QUANTITY A) FULLYCONNECTED B) CONVOLUTION C) BATCH NORMALIZATION
D(∆X) := ‖∇Y f‖2‖∆X‖2 RS
T 2 ‖∇Y f‖2‖∆X‖2
∑ 2γ2c g2c M2σ4c ∆x2nchw
∆x (∗)2 nchw =
e2V 2 2NC‖∇Y f‖2 e2V 2T 2 2RSMC‖∇Y f‖2 e2V 2Mσ4c Cγ2c g2c

heading: 5 Practical Automatic Lossy Compression
text: Using the AC-GC error bounds, we create convergence bounded compression methods with automatic compression rates, collectively referred to as AutoX. The first two methods (AutoQuant and AutoCuSZ) adapt scaled fixpoint [14] and error-bounded compression [27], which have bounded errors for a given compression rate. The errors for these methods are unbiased provided that unbiased rounding to fixpoint is used [5, 27]. The third method (AutoJPEG) uses lossy JPEG compression [14]. We bound JPEG error using an empirical error-compression relationship using activations sampled from uncompressed training of CIFAR10/ResNet50 [20]. Samples are used offline with JPEG compression to establish the compression-error relationship. This empirical compression-error relationship is used to calculate the JPEG compression levels, which approximately satisfy the error bounds in Table 1. We chain quantization and JPEG with lossless Zero Value Compression [46] to compress sparse activations better, creating AutoQuantZ and AutoJPEGZ.
As all AutoX used some form of fixpoint, we can express activation error in terms of bits. For any layer type, the relationship between bitwidth b and the convergence bound can be described as
b ≥ − log2 |∆xnchw|+ ... = − log2 |e| − log2‖∇θf‖+ ... (11) Using the results from Table 1, it can be seen that the bitwidth scales additively with the batch size as + log2(N) for convolution layers, and − log2(N) for normalization layers. Two issues with using AC-GC in a compression method are: 1) the various norms required are computationally expensive, and 2) the formulation assumes exact gradient information is available during the forward pass. We address both issues by statistically estimating activation error bounds. Instead of evaluating at every iteration, statistics are calculated at the end of every recalculation interval, specified in iterations. In the forward pass, a summary (mean or maximum) of the last ten recalculations is used when calculating the errors. This also allows approximating V 2 ≈ ‖∇θf‖2. Although some quantities are available in the forward pass, we estimate all of the activations, parameters, and gradients to avoid performing norm calculations at every iteration. Despite norms being estimated, we do not observe that the convergence bound (6) is violated for any network examined. A few training iterations can be used to verify correctness of the norm estimates (Figure 5c).

heading: 6 Evaluation
text: We examine activation compression by modifying the Chainer framework [53] to compress and decompress activations during training. We measure compression rates every 100 iterations, and otherwise perform paired compression/decompression to maintain the highest performance for our experiments. We focus our analysis on CNNs with image and text datasets, as they have large activation memory requirements, but avoid the largest networks [22, 52] due to limited resources. We create a performance implementation based off Chen et al. [7] to measure throughput.
For ImageNet [11], CIFAR10 [2] and Div2K [1], we use SGD and 0.9 momentum for VGG16 [50], ResNets (RN18 and RN50) [20], Wide ResNet (WRN) [59], and VDSR [29]. IMDB [39] and Text Copy [4] are trained using ADAM with CNN [53], RNN [53], and transformer heads [54]. All image datasets are augmented with random sizing, flip, and crop, as well as whitening and PCA for ImageNet [30], and 8× 8 cutout for CIFAR10 [12]. Learning rates, batch sizes, and epochs are 0.05, 128, 300 (CIFAR10, [49]), 0.1, 64, 105 (ImageNet, [58]), 0.1, 32, 110 (Div2K, grid search), 2.0, 64, 100 (Text Copy, [4]), and 0.001, 64, 20 (IMDB, [53]).
All forward and backward pass calculations use floating-point, using activations that have been compressed and decompressed between the two passes. Baseline refers to uncompressed training, i.e., 32-bit floating-point activations. GridQuantZ uses the same implementation of AutoQuantZ, however, it uses grid search over eight bit-widths of 2, 3, 4, 6, 8, 10, 12 and 16 bits, and then chooses the lowest with accuracy within 0.1% of the baseline. These grid points were selected to give good coverage of low and medium bit-widths, and are approximately logarithmic-spaced. SuccessiveHalving [26] and Hyperband [36] produce similar accuracy/compression to grid search but take less time. Unless otherwise stated, all experiments use e2 = 0.5, parameter estimates from the mean of a ten entry window, and a recalculation interval of 100 iterations. A value of e2 = 0.5 allows a small increase in loss over the baseline (+50%), but other values could be chosen depending on the desired error tolerance. Appendices C and D contain additional detail on hyperparameters and implementations.

heading: 6.1 Parameter Sensitivity
text: We isolate the impacts of parameter estimation and e selection by training LeNet [33] on MNIST [32], and RN18 [20] on ImageNet [11]. We train MNIST to convergence by using SGD with no momentum, a learning rate of 0.001, a batch size of 64, and 10 epochs.
The effect of e, recalculation interval, and summary method are evaluated by training MNIST/Lenet with different configurations of AutoQuant (AQ) (Figures 5a and 5b). Decreasing e (which increases bitwidth) causes the loss to increase, however, the average loss does not violate the bound in Theorem 1 (Figure 5a). In general, all networks examined with e2 = 0.5 have loss changes below 2% and validation score changes below 0.5% (Sections 6.2 and 6.3), which demonstrates that AC-GC error bounds are not violated. Both interval and summary method have a minimal impact on the training loss for MNIST (Figure 5b). As there is an insignificant change in loss, we use the mean for AC-GC as it has higher compression.
We evaluate ImageNet/ResNet18 using a dual training approach to ensure the correctness of parameter estimation (Figure 5c). This involves training using AutoQuant, while evaluating the true weight gradients ∇θf offline and comparing against their compressed counterpart ∇̂θf . The ratio is ≈ 1 for the duration of training, and does not violate the bound for e2 = 0.5 (i.e., ≤ 1.5). The mean ratio is less than one, likely due to decreased activation variance from compression to a discrete set of values. We observe similar behavior for the other layers in the network (not shown).

heading: 6.2 CIFAR10, Div2K and IMDB
text: We compare the AutoX methods with fixpoint grid search (GridQuantZ), and with prior works on lossy JPEG compression [14] (Table 2). Grid search requires oracle knowledge of the baseline, and 8× the training iterations of any other method in Table 2. Compared to GridQuantZ, AutoQuantZ uses a single run of training, and provides a similar compression rate of 7.5×. AutoCuSZ has a
compression 2.0× higher than JPEG-ACT, while maintaining accuracy to within 0.1 of the baseline on average. With a suitable error bound, CuSZ can extract significant compression from zeros and spatial information. On non-spatial data and non-image datasets (Text Copy and IMDB, Table 2) we observe that AutoCuSZ extracts similarly high compression with little accuracy change. Finally, AutoJPEGZ vs. JPEG-ACT demonstrates that using AC-GC error bounds gives higher accuracy and compression than using heuristics to select JPEG hyperparameters. In general, we find that using e2 > 0.5 decreases accuracy, leaving little reason to modify it.

heading: 6.3 ImageNet
text: On ImageNet training with ResNets (Table 2), the AutoX methods obtain a high accuracy. Our ImageNet accuracies are lower than other works as we do not use random scaling (which improves performance), and we report 1-crop accuracy. Our 10-crop accuracy for the ResNet50 baseline is 75.2%. Reduced compression rates on ResNet18 vs. ResNet50 are due to a lower sparsity in ReLU activations, which we hypothesize is caused by the different bottleneck structures of the two networks [20]. The higher-than-baseline accuracy of AutoQuantZ (Table 2) is caused by a large standard deviation for the ImageNet baseline (±0.21). On ImageNet, AutoCuSZ gives a high compression in exchange for a small decrease in accuracy, 0.15%-points. AutoQuantZ provides high accuracy, at a moderate compression rate of 0.7×, with a 1.2%-point better accuracy than JPEG-ACT. However, the primary advantage of AutoX methods is that the pre-training bound on loss increases.

heading: 6.4 Overheads
text: The AutoX methods require error bound calculation (common to all methods) and compression. Our unoptimized AutoQuantZ implementation achieves throughput of 1.66× (N = 128) vs. naive swapping to the CPU (N = 128), and 0.64× vs. uncompressed training (N = 32) (ImageNet/ ResNet50). Our primary contribution, AC-GC error bound calculation, uses 0.4% of total training time. This is negligible when compared to compression overhead, e.g., 4% for fixpoint [14, 25], 17% for CuSZ [27], 33% for ActNN-L3 [7], or 13% for hardware accelerated JPEG [14]. Unoptimized AutoQuantZ is 23% slower compared to ActNN-L3 [7]. Compression rate search time is decreased by 4.6× when compared to SuccessiveHalving (Figure 1).

heading: 7 Related Works
text: We compare AutoCuSZ with the most recent works in activation compression [6, 14, 25, 27] and reduced precision training [51, 57] (Table 3). AutoCuSZ obtains better accuracy and compression than most works, with the added benefit of not requiring a search over compression rates. BAA [6] presents a technique of recalculating activations similar to Chen et al. [8], which could be combined with any method in Table 3, including the AutoX methods. AutoCuSZ obtains higher compression and accuracy than EBC [27], demonstrating that AC-GC error bounds are better than hand-tuning in this case. WAGE [57] and ULP [51] and reduced precision training with theoretical convergence bounds [35, 37] reduce training compute through reduced precision, however, have lower accuracy due to accumulating errors in both the forward and backward pass. AutoX methods can potentially be combined with low precision training and other techniques to reduce memory requirements further.
ActNN is most similar to this work in that it invokes gradient variance to partition bits among compressed activations [7]. However, ActNN approaches the problem in an opposite manner to this work: it finds the highest accuracy for a given compression rate. The ActNN approach works well when the goal is to fit within a memory budget. Our approach allows for selecting the target loss increase a priori, and gaining information on the uncompressed accuracy from a single training run. AC-GC will also converge similarly to the uncompressed case on an unknown model, providing assurances that loss changes are due to model/hyperparameter configuration, not compression. Additionally, ActNN is specific to group-wise fixpoint compression, whereas AC-GC is quick to adapt to any lossy compression method (including group-wise fixpoint). On ImageNet/ResNets, AutoCuSZ and AutoQuantZ obtain half the accuracy change of ActNN-L3, albeit at 0.9× and 0.4× the compression, respectively.
Works targeting inference, such as precision reduction [10, 15, 23, 34], compression [13, 19, 38, 56], and sparsification [16, 43] increase memory requirements due to tracking additional state. Other works that directly address memory can be grouped into scheduling [8, 25], offloading [31, 45], and restructuring [17, 21]. Generally, these come with a performance or model-flexibility penalty. Any non-lossy method is partially orthogonal to compression and can be potentially combined with AC-GC for increased memory reduction.

heading: 8 Conclusions
text: The AC-GC automatic compression methods described in this work provide high compression rates with error trade-offs known before training. Avoiding compression rate search comes at a computational overhead of only 0.4%, many times less than tuning techniques. Provided that the assumptions of AC-GC hold, it can be further combined with any lossy compression method, layer, or network to guarantee convergence with high activation compression. Although convergence is theoretically guaranteed, some factors could impact convergence, such as floating-point errors. By lowering training costs, this framework allows for training larger models and faster exploration of the machine learning field’s landscape. Detailed derivations and implementations can be found in the supplemental material. Code is available https://github.com/rdevans0/acgc.
Funding Transparency Statement
This research was funded in part by the Computing Hardware for Emerging Intelligent Sensory Applications (COHESA) project financed under the National Sciences and Engineering Research Council of Canada (NSERC); grant number NETGP485577-15.

Review summaries:
The paper proposes a theoretical convergence upper bound for DNN training with compressed activation. Using this bound, the compression rate is controlled over training epochs by computing the quantization error. Moreover, to reduce the complexity of computing the activation’s quantization error, the paper uses statistics of this error such as mean and max instead of computing the complex norm. The efficiency of this new approach (AC-GC) is evaluated in various benchmarks including the ImageNet dataset and compared with previous approaches such as WAGE, GIST, and JEPG-ACT. The result showed that the AC-GC compressed the activation more than other approaches while maintaining accuracy compared to FP32.  

The authors bound the increase in loss in the case that activations used for backpropagation have errors. By doing so, they can set a fixed acceptable level of increase in the loss, and solve for the error level that can be tolerated for each activation used in backpropagation (and therefore deduce a level of compression). This analysis is relatively agnostic to the precise implementation of the compression. They evaluate their approach on a variety of datasets, including relatively large datasets such as ImageNet, demonstrating that their method can achieve high compression levels with relatively little tuning.

This paper studies activation compression to reduce the memory consumption when training models with stochastic gradient descent. The goal is to establish an activation compression framework with convergence guarantees, while eliminate the tuning overhead from conventional search based method for compression precision selection. The contributions of this paper come in three folds:  1. The authors first established a connection between the activation compression error and the convergence bound (i.e. the upper bound on training loss).  2. Built on the above theory, the authors formulate the activation compression precision selection problem as a constrained optimization problem, which can accommodate typical DNN layers such as linear layers, convolution layers and normalization layers.  3. Empirically, the authors combine this optimization-based activation precision selection method with various compression methods (with bounded errors at each compression precision). They show that it could attain matching accuracy (minimal acc degradation compared to no activation compression) compared to conventional grid search method for activation precision without the grid search (i.e. hyperparameter tuning) overhead.