DIVISION: Memory Efficient Training via Dual Activation Precision
Authors: Guanchu Wang
Zirui Liu
Zhimeng Jiang
Ninghao Liu
Na Zou
Xia Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: DNN training
activation compressed training
memory efficient training
frequency domain

Abstract: Activation compressed training (ACT) has been shown to be a promising way to reduce the memory cost of training deep neural networks (DNNs). However, existing work of ACT relies on searching for optimal bit-width during DNN training to reduce the quantization noise, which makes the procedure complicated and less transparent. To this end, we propose a simple and effective method to compress DNN training. Our method is motivated by an instructive observation: DNN backward propagation mainly utilizes the low-frequency component (LFC) of the activation maps, while the majority of memory is for caching the high-frequency component (HFC) during the training. This indicates the HFC of activation maps is highly redundant and compressible during DNN training, which inspires our proposed Dual Activation Precision (DIVISION). During the training, DIVISION preserves the high-precision copy of LFC and compresses the HFC into a light-weight copy with low numerical precision. This can significantly reduce the memory cost without negatively affecting the precision of backward propagation such that DIVISION maintains competitive model accuracy. Experimental results show DIVISION achieves over 10× compression of activation maps, and significantly higher training throughput than state-of-the-art ACT methods, without loss of model accuracy. The code is available at https://anonymous.4open.science/r/division-5CC0/


Under review as a conference paper at ICLR 2023
DIVISION: MEMORY EFFICIENT TRAINING VIA DUAL ACTIVATION PRECISION
Anonymous authors Paper under double-blind review
ABSTRACT
Activation compressed training (ACT) has been shown to be promising way to reduce the memory cost of training deep neural networks (DNNs). However, existing work of ACT relies on searching for optimal wit-width during DNN train- ing to reduce the quantization noise, which makes the procedure complicated and less transparent. To this end, propose a simple and effective method to compress DNN training. Our method is motivated by an instructive obser- vation: DNN backward propagation mainly utilizes the low-frequency compo- nent (LFC) of the activation maps, while the majority of memory is for caching the freq-fryy component (HFC) during the training. This indicates the HFC of activation maps is highly redundant and compressible during DNN training. which inspires our proposed Dual ActIVation PrecISIOn (DIVISION). During the training, DIVISION preserves the high-precision copy of LFC and compresses the HFC into light-weight copy with low numerical precision. This can sig- nificantly reduce the memory cost without negatively affecting the precision of backward propagation such that DIVISION maintains competitive model accu- racy. Experimental results show DIVISION achieves over 10x compression of activation maps, and significantly higher training throughput than state-of-the- art ACT methods, without loss of model accuracy. The code is available https::/anonymonn /anonyooouuus... 4oppe. sc
INTRODUCTION
Deep neural networks (DNNs) have been widely applied to real-world tasks such as language un- derstanding (Devlin et al. 2018). machine translation (Vaswani et al. 2017). visual detection and tracking (Redmon et al. 2016). With increasingly larger and deeper architectures, DNNs achieve remarkable improvement in representation learning and generalization capacity (Krizhevsky et al., 2012). Generally, training a larger model requires more memory resources cache the activation values of all intermediate layers during the back-propagattorf] For example, training a NenseNet- 121 (Huang et al.) 2017) on the ImageNet dataset (Deng et al.] 2009) requires to cache over 1.3 and training ResNet-50 (He al.) 2016) requires to cache over 4.6 billion float activation values (17GB). Some techniques have been developed to reduce the training cache of DNNs such checkpointing (Chen et al.) 2016 Gruslys ct al. 2016), mix precision training (Vanholder 2016), low bit-width training (Lin et al. Che7 al. 2020) and activation compressed training (Georgiadis) 2019 Liu et al. 2022). Among these, the activation compressed training (ACT) has emerged as a promising method due to its significant reduction of training memory and the competitive learning performance (Liu et al. 2021b).
billion float activation values (4.8GB) during back-propagation
Existing work of ACT relies on quantizing the activation maps to reduce the memory consumption of DNN training, such as BLPA (Chakrabarti & Moseley (019), TinyScript (Fu et al. (020) and ActNN (Chen et al. 2021). Although ACT could significantly reduce the training memory cost, the quantization process introduces noises in backward propagation, which makes the training suffer from undesirable degradation of accuracy (Fu et al. (020). Due to this reason, BLPA requires 4- bit ACT to ensure the convergence to optimal solution on the ImageNet dataset, which has only 6x compression rate of activation maps (Chakrabarti & Moseley 2019). Other works propose to
'The activation map of each layer is required for estimating the gradient during hackward propagation. 26x compression rate indicates the memory of cached activation maps is 1/6 of that of normal training.

---

Under review as a conference paper at ICLR 2023
search for optimal bit-width to match different samples during training, such ActNN (Chen al. 2021] and AC-GC (Evans & Aamodi (021). Although it can moderately reduce the quantization noise and achieves optimal solution under 2-bit ACT (nearly 10x compression rate), the following issues cannot be ignored. First, it is cone-consuming to search for optimal bit-width during training. Second, the framework of bit-width searching is complicated and non-transparent, which brings new challenges to follow-up studies on the ACT and its real-world applications.
In this work, we propose a simple and transparent method reduce the memory cost of DNN training. Our method is motivated by an instructive observation: DNN backward propagation mainly utilizes the fow-frequency component (LFC) of the activation maps, while the majority of memory is for the storage of high-frequency component (HFC) during the training. This indicates the HFC of activation map highly redundant and compressible during the training. Following this direction, we propose Dual Activation Precision (DIVISION), which preserves the high-precision copy of LFC and compresses the HPC into a light-weight copy with low numerical precision during the training- In this way, DIVISION can significantly reduce the memory cost. Meanwhile, it will not negatively affect the quality of backward propagation and could maintain competitive model accuracy.
Compared with existing work that integrates searching into learning (Liu et al. 2022). DIVISION has a more simplified compressor and decompressor, speeding up the procedure of ACT. More importantly. it reveals the compressible (HFC) and non-compressible factors (LFC) during DNN training, improving the transparency of ACT. Experiments are conducted to evaluate DIVISION in terms of memory cost, model accuracy, and training throughput. An overall comparison is given in Figure (a). Our proposed DIVISION consistently outperforms state-of-the-ar baseline methods in the above three aspects. The contributions of this work are summarized as follows:
We experimentally demonstrate and theoretically prove that DNN backward propagation mainly utilizes the LFC of the activation maps. The HFC is highly redundant and compressible.
We propose simple framework DIVISION to effectively reduce the memory cost of DNN train- ing via removing the redundancy in the HFC activation maps during the training. Experiments on three benchmark datasets demonstrate the effectiveness of DIVISION in terms of memory cost, model accuracy, and training throughput.
2
PRELIMINARY
2.1 NOTATIONS
Without loss of generality, we consider an L-layer deep neural network in this work. During the forward pass, for each layer (1 < 1 < L), the activation map is calculated by
H, forward(H,,1: Wt),
where Hr denotes the activation map of layer I; Ho takes a mini mani-batch of input images; W. denotes the weight of layer I; and forward(-) denotes the feed-forward operation. During the backward pass, the gradients of the loss value towards the activation maps and weights are be estimated by
(2) where VHH,, and VH, denote the gradient towards the activation map of layer 1-1 and re- spectively; Vw, denotes the gradient towards the weight of layer I; and backward denotes the backward function which takes VH, H and W,, and outputs the gradients VH, and Vw, Equation 2 indicates it is required to cache the activation maps Ho...., H₁- after the feed- forward operations for gradient estimation during backward propagation.
(H.,, Vw.] backward ( H; H,-,,
2.2 ACTIVATION COMPRESSED TRAINING
It has been proved in existing work (Chen et al. 2020 that majority of memory (nearly 90%) is for caching activation maps during the training DNNs. Following this direction. the activation compressed training (ACT) reduces the memory cost via real-time compressing the activation maps during the training. A typical ACT framework in existing work (Chakrabarti & Moseley, 2019) is shown in Figure (b). Specifically, after the feed-forward operation of cach layer I, activation map H is compressed into a representation for caching. The compression enables a significant reduc- tion of memory compared with caching the original (exact) activation maps. During the backward pass of layer ACT decompresses the cached representation into H,-1, and estimates the gradient by taking the reconstructed H1 into Equation 2 [V Vw.J= backward VH Hi.,, Wi).
We do focus on the closed from the backward function, which implemented by torch, wwtroh.....or....
2

---

Under review as a conference paper at ICLR 2023
Compression 11 Rate (nx)
SWAP
Forward poss
Bockward pess
Checkpoint
ActNN DIVISION
H Scroin W:
Iadowal Fhe H W.
PPe Feel
C(H..Y:W
batterd ( Fu, We
1o PAN
Bo.
GPU MEM
Scompree
Accuracy (%)
262
Hu
Throughout (Instance/S)
(8)
(b)
Figure 1: (a) Overall performance of DIVISION versus baseline methods. (b) Activation compressed training.
Masking
LFC-ACT
IDCT
Accuracy (16)
Memory (GB)
(thog domein)
(epotial
DCT
40-
Image Opunall donai))
Imnge (trog donair)
Hiab-pas Masiia
IDCT
20-
(deg domain)
(spanial donrale)
RN-18 DN SN
RN-18 DN-121 SN
(b)
(c)
Figure 2:
(a) Adopting DCT to estimate the low frequency component (LFC) and high frequency compo- nent (HFC) of an image. (b) Top-1 accuracy and (c) Memory cost of normal training, LFC-ACT and HFC-ACT, where RN-18, DN-121, and SN refer to the ResNet-18 18, DenseNet-122 121, and ShuffleNet-N V2, respectively.
Even though the pipeline of compression and decompression is lossy, i.c. H. * H2 for 1 < S L. has been proved ACT can limit the reconstruction error flowing back to early layers and enables the training to approach an approximately optimal solution (Chen et al. (221T.
2.3 DISCRETE COSINE TRANSFORMATION
Discrete Cosine Transformation (DCT) projects the target data from the spatial domain to the fre- quency domain via the inner-produs producrootttion of the data and collection of cosine functions with different frequency (Rao & Yip 2014). We focus on the 2D-DCT in this work, where the target data is the in- put image and activation maps of DNNs. Specifically, for 2D-matrix data H, the frequency-domain feature is estimated by H DHT(H), where H and H have the same shape of N x N, and each of the element his given by
N-IN-1
h
>> hm.n 006
N
2
(3)
where hm, 0 S m, N -1,, elements in the original matrix H. During the training of DNNs, an image or activation map has the shape of Minibatch x Channel x N. In this case, the frequency-domain feature estimated via operating 2D-DCT for each NxN matrix in each channel.
With DCT, we could extract the lov-freequencyyhhhh---eeeeeuee frequencyyyeyyeyeee component (LFC/HFC) of an image or activation map, using a pipeline of haw--aasshhihhpaae masking and inverse DCT, as shown in Figure To be concrete, the estimation of LFC and HFC is given by
H iDCT(H O M)
H iDTTH @ (INxN M)),
(5)
2014; ; M = [m, LijS N] denotes an N x N low-pass mask satisfying m,, for 1 W and m.,j = for other elements; and 1NxN - M indicates the high-pass mask. Intuitively, H has W non-zero float numbers in each channel, contrast with 22 non-zero float numbers each channel of H Generally, we have W < N in practical scenarios, e.g. W/N- 0.1 in Figure 2 (a). This indicates the HFC takes the majority of the memory cost in the caching of activation maps.
where iDCT(.) denotes the inverse DCT RRoo Yip
CONTRIBUTION OF LFC AND HFC TO BACKWARD PROPAGATION
In this section, we experimentally prove the LFC of activation maps has significantly more contri- bution DNN backward propagation than the HFC. Moreover, our theoretical result indicates the LFC enables the estimated gradient to be bounded into a tighter range around the optimal value, leading to a more accurate learned model, which is consistent with the experimental results.
3

---

Under review as a conference paper at ICLR 2023
200
100
100
175
150
60
75
BesicBlock-2
BesicBlock-1
40
BasicBlock-2
BasicBlock-1
BasicBlock-2
BasicBlock-1
50
BosicBlock-3
BasicBlock-3
BasicBlock-3
25
BesicBlack
BasicBlock
100 260
0
100
100
Aa
(a) Epoch 20.
(b) Epoch 40.
(c) Epoch 60.
Figure 3: | versus
H|MM|
in the training (epoches 20, 40, and 60) of ResNet-
18. H, takes the activation maps four BasicBlocks in ResNet- D indicates mean values; -05NN.
3.1 EXPERIMENTAL ANALYSIS
To study the individual contribution of LFC and HFC to DNN backward propagation, we design three training methods with different backward propagations: LFC-ACT takes LFC into the back- ward function as shown in Equation (6. where H is estimated by Equations (4) HFC-ACT takes
HFC into the backward function as given in Equation 7 where H is according to Equation 5
Normal training (for comparison) estimates the gradients by Equation (
[H, Vw, backward (Valdwrrrrr H/,, Wi),
LFFCACCT (6)
[H Ww, backward (V H H Wi).
HHFC-ACT (7)
We conduct the experiments on the CIFAR-IO dataset. The implementation details are given in Appendix A The top-1 accuracy and memory cost of LFC-ACT, HFC-ACT, and normal training are shown in Figure and (c), respectively. Overall, we have the following observations:
Accuracy drop: According to Figure HFC-ACT suffers from significantly more degrada- tion of accuracy than LFC-ACT. This indicates DNN backward propagation mainly utilizes the LFC of activation maps during the training.
Memory cost: According to (c), the storage of HFC requires significantly more memory than that of the LFC. i.c. The storage of HFC consumes the majority of memory.
To better understand the results of model accuracy, we theoretically prove the gradient for backward propagation is bounded into tighter range around the optimal value in LFC-ACT. This enables LFC-ACT to learn a more accurate model than HFC-ACT.
3.2 THEORETICAL ANALYSIS
We theoretically analyze the gradient estimation error of LFC-ACT and HFC-ACT which adopt Equations 66) and (7) for backward propagation, respectively. Generally, for the LFC-ACT and HFC-ACT, let V w: and V
Ww denote the estimated gradient of layer I, respectively. In this way, J/ V | and | w: -. I/r indicates the gradient estimation errors, taking the complete gradient Vw, as reference. To compare the distortion of backward propagation LFCCACT and HFC-ACT, GEB/ and GEB# denote the gradient error upper bound (GEB), respectively, i.e. || - Vw, < GEB and || w. - Vw, || < GEB/ Intuitively, higher GEB indicates less accurate backward propagation, leading to less accurate model after training. To this end, give Theorem | to compare GEB, and GEBH. where a convolutional layer is considered. The proof is given in Appendix
B A similar analysis of GEB for linear layer is provided in Appendix Theorem 1. During the backward pass of a convolutional layer I, GEB, and GEB# satisfy
= | (2---()++
H [ Ea,
Eaa. -1-- II
T5.
(8)
i=1+1
j=l
where O1,i, 31 O for < Llis L depend on the model weights before backward propugation [r; = (1-HH DDC(HTHH)
(given by Equations (24) in Appendix Bl: - | |M
and M denotes the loss-pass mask given by Equation 4
Theorem indicates the GEB difference depends on X >1 for < < T. during the training. Fol-
lowing this direction, we estimate X/ and X via X/ = ||H OMIl and X H|H during the training of ResNet-18 18 and DenseNet- 121 on the CIFAR- 10 dataset. Specifically, H, takes the
"The Frobenius norm of v matrix A is given by ||| FF
4

---

Under review as a conference paper at ICLR 2023
Layer /+] Forward
Activation map Compression
Activation map Decampression
Layer Bockwardd
GPU MEM
Layer Forward
Hi
Averagy Poolling
Hi
Quant
Dequant
Sampling
Up-
Hi
Layer Back
Hil
Forward
Backward
Figure 4: Framework of Dual Activation Precision Training.
activation maps of the BasicBlocks in ResNet- 18, and Denseblocks in DenseNet-121. The estima- tion of A and X/ is based on the checkpoint of ResNet-18 in epoches 20, 40, and 60, and visualized in Figures 3 (a)-(c), respectively; and the results of DenseNet- 121 are given in Appendix D It is consistently observed that X > X for different instances and layers. This leads to GEB; < GEB according to Theorem [I] Therefore, HFC-ACT suffers from worse distortion of backward propa- gation during the training, eventually leading to less accurate learned model than LFC-ACT.
In this section, from both experimental and theoretical perspectives, we prove the HFC of activation maps has less contribution to backward propagation than LFC. However, according to Figure 2 (c), the HFC takes the majority of memory cost during the training. This indicates the HFC is highly redundant and compressible during the training. Following this direction, we propose DIVISION compress the activation maps into dual precision representation: high-precision LFC combined with low-pre now-precision HFC. On the one hand, both LFC and low- low-precision HFC requires much less memory to cache. On the other hand, removing the redundancy HFC cannot cause much distortion of backward propagation. In this way, DIVISION enables effective compression of training memory without degradation of model accuracy.
4 DUAL ACTIVATION PRECISION TRAINING
We introduce the proposed Dual ActIVation PrecISION (DIVISION) in this section. The framework of DIVISION is shown in Figure 4 Specifically, after the feed-forward operation of each layer, DIVISION estimates the LFC and compresses the HFC into a low-precision copy such that the total memory cost is significantly decreased after the compression. Before the backward propagation of each layer, the low-precision HFC is decompressed and combined with LFC to reconstruct the activation map. The detailed compression and decompression given follows.
4.1 ACTIVATION MAP COMPRESSION
For compressing the activation map H. of layer I, DIVISION estimates the LFC H. and HFC H!" after the feed-forward operation. However, the high computational complexity of DCT prevents us from directly applying it to real-time algorithms. We thus give Theorem 2 to introduce a moving average operation that can approximate the loss-pass filter. The proof is given in Appendix E
Theorem 2. For any reul-valued function [xx) and its moving average f(x) 2B 5 s(()d, let F'w) and F(w) denote the Fourier transformation of f(x) and fxx), respectively. Generally, we have F(w)= H F(w),, where (w>) BB
BB
Remark 1. The frequency response of ((w) depends on its envelope function [BB Note that IoBl decreases with [w)] such that wB O as w + cc. Hence, H(w) is an approximate loss-pass filter.
According to Remark 1 we approximate the LFC H. into the moving average of H; Note that the average pooling operator provides efficient moving average. DIVISION adopts average pooling to estimate the LFC by H = A
(H)). The value of block-size and moving stride is unified hyper-parameter B, which controls the memory of 5 Moreover, H is cached in the
format bfloat1 for saving the memory. In our experiments, we found B = 8 can provide representative LFC for backward propagation, where the memory cost of H;; is only 0.8% of H,
To estimate the HFC, DIVISION calculates the residual value HH H -H-UUSSappling/ (H)), where the UpSampling(+) enlarges H to shape Minibatch x Channel x N x N via nearest interpolation. Then, DIVISION compress the H into low precision because it plays less important role during the backward propagation but consumes most of the memory. Specifically. DIVISION adopts Qib
For case pooling block- size stride will such that shape Minihatch Channel xixl.
5

---

Under review as a conference paper at ICLR 2023
per-channel quantization for the compression, where the bit-width Q controls the precision and memory cost of HFC after the compression. Let V denote a Q-bit integer matrix, as the low- precision representation of H The detailed procedure of compressing H into V is given by
v = Quant (H+)) = -, (H - 6,)7,
where 6y denotes the minimum element in H A = (hmax - 51) - 1) denotes the quantization
Pi Gupta et al, 2015); and 6j and A are cached in the formate of b 1oat16 for saving memory. In this way, the memory cost of (VT ōt, Mmax) is (N²Q/8 + 4) bytes per channel, in contrast with that of H, being 4N2 bytes per channel. In our experiments, we found Q 2 can provide enough representation for backward propagation, where the memory cost of V is only 8.3% of Hi.
step; max denotes the maximum element in H |o] denotes the stochastic rounding
After the compression, as the representation of H;. the tuple of (H), VH, A,, 61) is cached to the memory for reconstructing the activation maps during the backward pass.
4.2 ACTIVATION MAP DECOMPRESSION
During the backward pass, DIVISION adopts the cached tuples of {(H, VH. (1, 1 SIS
7. 1) to reconstruct the activation map layer-by-layer. Specifically, for each layer I. DIVISION dequantizes the HFC via H = A V which is the inverse process of Equation Then, the activation map is reconstructed via
H UpSampling + H
(10)
where UpSampling(.) enlarges H to shape MinibatchxChammelNNNl) via nearest interpolation. After the decompression, DIVISION frees the caching of (H) VH, Ar, ōt), and takes H, into
H Vw,] = backward H. H W.) to estimate the gradient for backward propagation. Without loss of generality, DD/3D activation maps are considered for DIVISION in Appendix F
4.3 ALGORITHM OF DIVISION
Algorithm 1
Mini-batch updating of DIVISION
Input: Mini-batch samples x and labels y.
Algorithm [ demonstrates mini-batch updat- ing of DIVISION, which includes a forward pass and backward pass. During the forward pass of each layer, DIVISION first forwards the exact activation map to the next layer (line 2); then, estimates the LFC and HFC (line 3- 4); after this, achieves the low precision copy of HFC (lines 5); finally caches the represen- tation to the memory (line 6). During the backward pass of each layer, DIVISION first decompresses the HFC (line 10); then recon- structs the activation map (line 11); after this, estimates the gradients and updates the weights of layer/ 1 (line 12); finally frees the caching of V V ( (line 13). For each mini- batch updating, the memory usage reaches the maximum value after the forward pass (caching the representation of activation maps layer-by- layer), and reduces to the minimum value after the backward pass (freeing the cache layer-by-layer). Existing work (Chen et al., 202T) estimates the memory cost of activation maps by
Output: Weight and bias (Wr, B; L..
1: for layer [ ;= 1 to do
2: H, = f(W, - B // Ho = x
3:
H. = AveraagPoollnn(HH....
4:
H H UpSamplingg[R (H)
5: Vi (H
6: Cache (H) Vi A--,
7: end for
8: Estimate the loss value and gradient VHE-
9: for layer 1 ;= to do
10:
H Dequant A A1 5;-1) Ã UpSamplingHHI + Hi
11:
12: Estimate [VH,,, Vw,] and update W.
13: Free (H) Vi &
14:
end for
Memory Cost Memory Utilization after forward Memory Utilization aff backward:
(11)
where existing deep learning tools provide API to estimate the memory utilization.
The theoretical compression rate of DIVISION given in Appendix G where general cases of convolutional neural networks and multi-layer perception are considered for the estimation. For the model architectures in our experiments, we have RResNet.50, RWRN.50.2 2 10.35.
"A fixed witth adopted for the quantization of all layers maximize the efficiency of data preeesing.
7 Per-channel quantization more efficient light than per- quantization state-of-the-at work.
8 [x]] takes the value of [x] with a probubiity of 2 [x]] and takes [=]] with probubility of [xx] -x.
9 The stochastic rounding enables the desatization-oqqaanian
pipeline to unbiased, i.e. B[V H
torch, muuda.memor all located returns memory occupied by tensors bytes.

---

Under review as a conference paper at ICLR 2023
100
100
Vermal DINSSOK
Nomaa
7018
Normal DIVISION
90
90
BMAIOON
Accuracy
Aetio
80
80
80
UE
70
70
50
CIFAR-1D
CIFAR-10D
CIFAR-10
CIFAR-100
ImageNet ResNet-50
CIFAR-10
CIFAR-10D ImageNet ImageNet
ResNet-164 ResNet-164
ResNet-18 ResNet-164
ResNet-18 DenseNet-121 ResNet-50 DenseNett1611
(a) DIV vs BLPA
(b) DIV vs AC-GC
(c) DIV vs ActNN
Figure
Top-1 accuracy (56 1 Normal training, DIVISION, BLPA (a), AC-GC (b), and ActNN (c).
EVALUATION OF DIVISION
We conduct the experiments to evaluate DIVISION by answering the following research questions. RQI: How does DIVISION perform compared with state-of-the-art baseline methods in terms the model accuracy, memory cost, and training throughput? RQ2: Does the strategy of (ual-precision compression contribute to DIVISION? RQ3: What is the effect of hyper- parrateerssss on DIVISION?
The experiment setting including the datasets, baseline methods and DNN architectures speci- fied in Appendix H The implementation details including the prrramppraeeters of DIVISION and configuration of baseline methods are given in Appendix More experiments on MLPs, vision transformers, depthwise, and pointwise convolutional layers are given in Appendix K
and
5.1 EVALUATION BY MODEL ACCURACY (RQ1)
In this section, we evaluate the training methods in terms of model accuracy on the CIFAR- 10, CIFAR-100 and ImageNet datasets. Specifically, DIVISION is compared with BLPA (Chakrabarti & Moseley 2019), AC-GC (Evans & Aamodi (021) and ActNN (Chen et al. 2021) in Figure (a)- (c), respectively, where different model architectures are considered. Here, Checkpoint and SWAP are not considered in this section because they are able to reduce the training memory without degradation of model accuracy. Overall, we have the following observations:
DIV vs Baseline Methods: Compared with normal training, DIVISION achieves almost the same top-1 validation accuracy. In contrast, the baseline methods suffer from slightly higher validation error. This indicates DIVISION provides nearly loss-less compression of DNN training.
Flexibility of DIV: DIVISION consistently achieves competitive model accuracy in the training of different architectures on different datasets. This indicates DIVISION is a flexible framework that be applied to different scenarios.
Compressibility of HFC: Note that DIVISION adopts a significantly high compression rate 12x for the HFC during the training, and achieves nearly loss-less accuracy. This result indicates the HFC of activation map is highly redundant and compressible during the training.
5.2 EVALUATION BY MEMORY COST (RQ1)
We evaluate the training methods in terms of the training memory cost on the ImageNet datasest, where the configuration of our computational infrastructure is given in Appendix R Table [ indi- cates the training memory cost and practical compression rate of DIVISION and baseline methods. Moreover, DIVISION is compared with the checkpoint strategy of Megatron-LMM (Shoeybi et al.) 2019 in Appendix Overall, we have the following observations:
DIV vs SWAP, Checkpoint & BLPA: SWAP reduces the GPU memory cost merely by transfer- ring the overhead from GPU to CPU, which non-effective if considering the memory utilization of both GPU and CPU. Checkpoint shows considerable memory overhead because it caches some key activation maps to reconstruct other activation maps during backward pass. BLPA is less effective than DIVISION because it relies
least 4-bit compression.
DIV vs AC-GC: The practical memory cost of AC-GC should be greater than the values given in Table 0 AC-GC searches the bit-width from initial maximum value, and finalizes with an optimal bit-width. Thus, the average memory cost should be greater than that in the last epoch.
DIV vs ActNN: DIVISION has approximately the storage of 2-b activation maps, DIVISION has overhead for caching the LFC; and ActNN spends almost equal overhead for storing the parameters of per-group quantization.
memory cost ActNN. Beyond the

---

Under review as a conference paper at ICLR 2023
Table 1: Memory cost, and compression ratef. Total Mem refers total memory cost of weights, optimizer, data and activation maps. Act Mem refers memory cost of activation maps. 00M refers out of memory.
Architecture
ResNet- 50
WRN-50-2
Batch-size
64
128
256
512
64
128
256
512
Normal
5.46
10.62
20.92
00M
7.52
14.23
27.68
0OM
SWAP
5.46 (1x)
10.62 (1x) 20.92 (1x)
00M
7.52
(Ix) 14.23
27.68 (1x)
0OM
Checkpoint
1.23 (4.4x) 2.16 (4.9x) 4.03 (5.2x)
7.76
1.71
(4.4x)
2.65 (5.4x) 4.51 (6.1x)
8.25
Total Mem (GB)
BLPA
1.15 (4.75) 2.01 (5.3x)
3.72 (5.6%)
7.14
1.87
(4.0%) 2.96 (4.8%)
(5.4x)
9.51
AC-GC
1.80 (3.0k) 3.31 (3.2x)
6.31 (3.3%)
12.33
2.72
(2.8x) 4.66 (3.1x) 8.53 (3.2x)
16.27
ActNN
0.81
(6.7x) 133 (7.9x)
2.39 (8.8%)
4.47
(522))
2.09 (6.8%)
3.41 (8.1x)
6.03
DIVISION
0.82
(6.7x) 1.35 (7.9%) 2.41 (8.7x)
4.52
1.5
2.12 (6.7x) 3.44
(8.0x)
6.08
Normal
5.14
10.25
20.48
OOM
6.70
13.38
26.75
00M
SWAP
(Ix) 10.25 (Ix) 20.48 (Ix)
00M
(170
13.38 (Ix) 26.75 (IX)
00M
Checkpoint
0.90 (5.7%) (5.7%) (5.7%)
7.18
(7.40) 1.80 (7.4%) 3.59 (7.5%)
7.18
Act. Mem (GB)
BLPA
0.82 (6.3x) 1.64 (6.2x) 3.28 (6.2x)
6.56
1.06 (6.3x) 2.11 (6.3%) 4.22 (6.3%)
8.44
AC-GC
1.47 (3.5x) 2.94 (3.5x) 5.88 (3.5x)
11.75
1.91 (3.5x) 3.81 (3.5x) 7.61 (3,5x)
15.20
ActNN
0.49 (10.5x)0.97 (19.63)1.94 (10.6%)
3.89
0.62 (10.8x)1.25 (10.7x)2.49 (10.7x)
4.97
DIVISION (10.5x)0.99 (10.4%)1.97 (10.4x) 3.94
0.64 (10.5x)1.27 (10.53)..52 (10.6x)
5,02
700
450
80
600
Norra
Checosant
400
Mermal
Checkpon:
DIVISION
Trroughput
500
Ac/Mi
5 300
350
60
wie
400
6
50
Fixed Duant
300
0 200
40
200
150
30
100
100
50
10
64
128
Batch-size
256
512
64
128
Batch-size
256
512
ResNet-50
DenseNet-161
ResNet-50.
(b) WRN-50-2.
(c) Ablation Results.
Training throughput of (a) Resnet-50 and (b) WRN-50-2 on the ImageNet dataset, where x indicates out of memory. Top-1 validation accuracy (%) f DIVISION, DIVISION w/o HIFC, DIVISION w/o LFC and fixed bit-width quantization on the ImageNet dataset.
Figure 6:
Act. Maps: For normal training. the caching of activation maps takes the majority of memory cost 509,, growing with the mini-batch size), which is consistent with our discussion in Section [
Compression rate: The activation map compression rate of DIVISION is consistently with the theoretical results (RResNesso,, RoRR 2 10.35, see Appendix G which not influenced by the mini-batch size. Moreover, the overall compression rate grows with the mini-batch size.
5.3 EVALUATION BY TRAINING TUROUGHIPU (RQ1)
We now evaluate the training methods in terms of the training throughput on the Imagenet dataset. Generally, the throughput indicates the running speed of training method via counting the average number data samples processed per second. The throughput is given by Mini-butch Size where Thach T
denotes the time consumption of single mini- updating. Each method is combined with the automatic mixed precision (AMP UT] to speed up the training. rigures/6 (a) and (b) show the average throughput of 20 times of mini-batch updating. Overall, we have the following observations:
Reason for Time overhead: Compared with normal training, the time overhead of DIVISION comes from the estimation of LFC and compression of HFC. In ActNN, the overhead mainly comes from the the dynamic bit-width allocation and activation map quantization. In Checkpoint, it comes from replaying the forward process of inter-media layers. In SWAP, the overhead mainly derives from the communication cost between the CPUs and GPUs.
DIV vs ActNN: DIVISION shows higher throughput than ActNN result of more simplified data compression. To be concrete, DIVISION adopts average-pooling to extract the IFC,, and fixed bit-width per-channel quantization compress the HFC. In contrast, ActNN relies searching optimal bit-width to match different samples, and per-group quantization based on the searched bit-width. ActNN has more complex processing, which leads its lower throughput.
DIV vs SWAP: P: SWAP is less efficient than ACT--based methods (DIVISION and ActNN), which indicates the CPU-GPU communication cost is larger than the cost of activation map processing.
8

---

Under review as a conference paper at ICLR 2023
5.4 EFFECT OF DUAL PRECISION STRATEGY (RQ2)
To study the effect of our proposed dual precision strategy. DIVISION is compared with three train- ing methods: DIVISION w/o HFC: Merely caching the high-precision LFC for back- prok-propagation. DIVISION w/o LFC: Merely caching the low-precision HFC for back-propagation. Fixed Quant: Compressing the activation maps using a fixed bit-width quantization. The experiments are con- ducted on the ImageNet dataset using the hyper-paramette of Fixed Quant with different bit-width are given in Appendix P The model accuracy are given in Figure 6 (c). Overall, we have the following insights:
given in Appendix [S] More experiments
LFC & Low Precision HFC:
Removing either HFC or LFC from DIVISION, the training con- verges to far lower levels of accuracy. This indicates both the LFC and low precision HFC of activation maps are necessary for leading the training to converge to an optimal solution.
Benifits of Dual Precision: The fixed bit-width quantization fails to converge to an optimal solution. This indicates the noise caused by the fixed bit-width quantization can terribly disturb the back-propagation. DIVISION solves this problem by combining high-precision LFC and fixed bit-width quantization for compressing the activation maps.
We study the effect of TUNING FOR DIVISION (RQ3)
5.5
95
B (block-size) and Q (bit- width) the accuracy and compression rate. Specifically, adopt DIVISION to train ResNet-18 on the CIFAR-10 dataset with B E {8, 12, 18) and Q E {2, 4, 8). The aceuracy versus compression rate is shown in Figure (a). Overall, we have the following insights:
Accurncy
90
B=18Q=2
95
B-12Q--2
Bad Q-2
B-8 Qud
80
Effect of Q: The accuracy is stable (consistently nearly 95%) when reducing the preeision-level of HFC (Q reduces from 8 to 2). This
(a) Effect of Hyperr
indicates DIVISION only requires approximate values of HFC dur- ing backward propagation.
AMP Acoussy
Throughhp pci
Accurscy
Effect of B: Lower-precision LFC for backward propagation leads to significant degradation of accuracy (as B grows from 8 to 18). This is because DIVISION relies on a high-precision LFC to re- construct the activation maps for backward propagation.
95
3.0
128
258
532
ceuuracyccompresioon trade-off taking B- = 8 and Q 2, where the degradation of ac- curacy is less than 0.4%. According to more empirical studies in Appendix N B = 8 and Q = can be a default setting effective for most of model architectures and datasets.
Optimal Setting: DIVISION has optimal
Batch-skee
(b) Effect of AMP.
Figure 7: (a) Top-1 Accuracy and compression rate of DI- VISION in different settings. (b) Top-1 accuracy and train- ing throughput of DIVISION w/ and w/o AMP.
Note that normal training can be accelerated by the automatic mixed precision (AMP) (Micikevicius et al. (017)) without loss of accuracy. We study whether AMP can speed up DIVISION without loss of accuracy. Specifically, we follow the setting of DIVISION B = 8, Q = 2 to train ResNet-18 on the CIFAR-10 dataset. The accuracy and training throughput of DIVISION w/ and w/o AMP are shown in Figure (bb). More experiments with different mini-bateh size are given in Appendix Q It is observed that AMP can significantly speed up the DIVISION when MiniBatch-siiee 2 256 without loss of model accuracy. This indicates DIVISION has the potential to be applied to scenarios where both time and memory are limited.
CONCLUSION
In this work, we propose simple framework of activation compressed training. Our framework is motivated by an instructive observation: DNN backward propagation mainly depends on the LFC of the activation maps, while the majority of memory is for the storage of HFC during the training. This indicates back-propagation mainly utilizes the LFC to estimate the gradient, while the HFC is highly redundant and compressible. Following this direction, our proposed DIVISION compresses the activation maps into dual precision representations: high-precision LFC and low- fow-precision HFC, according to their contributions to the back- propagation. This dual precision compression can sig- nificantly reduce the memory cost of activation maps without disturbing the training.
Different from the existing work of ACT, DIVISION is a simple and transparent framework, where the simplicity enables efficient compression and decompression; and transparency allows to un- derstand the compressible (HFC) and non-compressible factors (LFC) during DNN training. To this end, we hope work could provide some inspiration for the compression of DNN training.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ayan Chakrabarti and Benjamin Moseley. Backprop with approximate activations for memory- efficient network training. Advances in Neural Information Processing Systems, 32, 2019.
Jianfei Chen, Yu Gai, Zhewei Yao, Michael W Mahoney, and Joseph E Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. Advances in Neural Information Processing Systems, 33:883-894, 2020.
Jianfei Chen, Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael Mahoney, and Joseph Gonzalez. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning, pp. 1803-1813, PMLR, 2021.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1604::600.00/7 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.08 04805, 2018.
Paulo SR Diniz, Eduardo AB Da Silva, and Sergio L Netto. Digital signal processing: system analysis and design. Cambridge University Press, 2010.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL ics. uci. bucieedu eduunedu//n
David Evans and Tor Aamodt. Ac-ge: Lossy activation compression with guaranteed convergence. Advances in Neural Information Processing Systems, 34:27434-27448, 2021.
Fangeheng Fu, Yuzheng Hu, Yihan He, Jiawei Jiang. Yingxia Shao, Ce Zhang, and Bin Cui. Don't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript. In International Conference Machine Learning, pp. 3304-3314, PMLR, 2020.
Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7085-7095, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint
2017.
Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Mare Lanctot, and Alex Graves. Memory-efficien backpropagation through time. Advances in Neural Information Processing Systems, 29, 2016.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference machine learning, pp. 1737-1746. PMLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Roger A Hor and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 1341-1355, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. Advances in neural information processing systems, 30, 2017.
Xiaoxuan Liu, Lianmin Zheng, Dequan Wang, Yukuo Cen, Weize Chen, Xu Han, Jianfei Chen, Zhiyuan Liu, Jie Tang, Joey Gonzalez, et al. Gact: Activation compressed training for generic network architectures. In International Conference Machine Learning, pp. 14139-14152. PMLR, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021a.
Zirui Liu, Kaixiong Zhou, Fan Yang, Li Li, Rui Chen, and Xia Hu. Exact: Scalable graph neural networks training via extreme activation compression. International Conference on Learning Representations, 2021b.
Vijay Madisetti. The digital signal processing handbook. CRC press, 1997.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.033: 2017.
Zizheng Pan, Peng Chen, Haoyu He, Jing Liu, Jianfei Cai, and Bohan Zhuang. Mesa: memory- 2021.
saving training framework for transformers. arXiv preprint
K Ramamohan Rao and Ping Yip. Discrete cosine transform: algorithms, advantages, applications. Academic press, 2014.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv:1909.0803: 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
Duraisamy Sundararajan. The discrete Fourier transform: theory, algorithms and applications. World Scientific, 2001.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
Han Vanholder. Efficient inference with tensorrt. In GPU Technology Conference, volume pp. 2, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- tion processing systems, 30, 2017.
11

---

Under review as a conference paper at ICLR 2023
APPENDIX
A IMPLEMENTATION DETAILS OF SECTION 3
We give the details of the experiment in Section 3 Without loss of generality, the experiment
conducted on the CIFAR-10 10 dataset using R8sNet-18, DenseNet-122 121 and ShuffleNet-V2 During the backward propagation of normal training, the gradient of each layer is estimated by [ Vw,] = backward (Ve H W,)
(12)
For LFCCACT,, the gradient is estimated by
H Ww,] backward H, H/ Wi),
(13)
where HFC-ACT denotes the HFC H,; for HFC-ACT, the gradient is estimated by
WH V w. backward VH, H W.).
(14)
where H denotes the HFC of H, Note that Equations 13 and 14) causes the distortion of
backward propagation in LFC-ACT and HFC-ACT, respectively. The objective of this experiment investigate whether this distortion of propppropagttion may powerful enough to lead training to non-optimal solution. The
setting of the training is given in Table 2
Table 2: Hyper-parameter setting.
Architecture
ResNet-18
DenseNet-122
ShuffleNer- V2
Epoch Batch-size
100
100
100
256
256
256
Initial LR
0.1
0.1
0.1
LR scheduler
Step LR
Step LR
Step LR
Weight-decay
0.0005
0.0005
0.0005
Optimizer
SGD
SGD
SGD
SGD Momentum
0.9
0.9
0.9
Ratio of LFC (W/N)
0.3
0.3
0.5
B PROOF OF THEOREM 1
We prove Theorem I in this section.
Theorem During the buckward pass of convolutional layer 1, GEB) and GFB/ satisfy
GEB -GEB =(a [1,,1 H.l.|IHH1lll (1) -x) -
-
|
a a
- xb)
II
(15)
i=/+1
j=1
where or,i, Br, 0 for 1 1, < are given by Equation 24
X
||
M P;
X
OI H. = DCT(H;); and M denotes the loss-pass mask given by Equation (4)
Proof. For simplicity of derivation, we study the case with single input channel and output channel number. this case, H, and W, are 2-D matrix for each layer I, where 1 S l S L. The backward propagation of a convolutional layer is given by
Vz Znn
Wrot 1+1 @ (Z)()
(16)
V., = Vz, H
where denotes convolutional operation; 2, W H + by; bi denotes the bias of layer I; and W denotes to rotate W, by 180" The case of multiple input and output channels can be proved in an analogous way, which is omitted in this work.
12

---

Under review as a conference paper at ICLR 2023
According to Equation (6). have the gradient of Zi given by
2z, Vz.
Vzn W
[+1 @ (Zi) 2111 W 1- @ (Z,,,
ZZZ, Whot Or (2,) Vz. + [+1 Oo (Zi)+ Vz + What (Z) 2z
Whot 1+1 Oo (Zi),
Vzz, W 141 ④ (Zi) o (Zr) + (Vz41 Vz.41 W 1+1 @ o (Zi)..
(17)
For the activation functions ReLu(.), LeakyReL.u)), Sigmoid(.), Tanh(.) and Slfs the gradient o'()) satisfies (a"()) S in the differentable domains. Note that we have [[ H
(Ki + N₂ - 1)/// H according to Corollary |oo o' -Zizill satisfies
||o'(Z,) - o' (Z.)ll - 2 Z, S (Ki N H)|
|
(18)
where K₁ and Ni denote the size of convolutional kernel W, and activation map Hr in layer respectively. After taking Equation
18 into Equation 17
we have
Zz Vz,lrr I (K, + N₁ 1)ll{ F Wrot
1+ |lllllll/////// (2,) o' (Zi)ll
+ (Ki + Ni- - 1) V Z,+1 - Vz V2.4,
||.
Whot
ww 1ellllllo/// (Zi)
141
(Ki + N; 1) z ||l||||11111 [+1
( [
- H1-11111l1l -
+ (Kr + N - 1) Zz Vz,,,
W
[ H H m/1 2l/ Veeillm;
(19)
where ?/r and n given by
= (K: + N₁ - 1) 3, W Welle|/
(20)
7 = (K: + N₁ - 1)] (/]]| (
the value and "o depend on the model weight before backward propagation, which is constant with respect to the gradient. Iterate Equation (19 until / L where Vz. Vz, P HG,1 this way, we have
|| Vz Snallsm H H Hoille +
nil
H. H
7 j=t
(21)
2m2+1
According to Equation (6), we have the gradient of Wi given by
Vw, Vw, Vz, H Vz, H
-VZ A Vz * H + Vz * H.
Vz, H
H Vz. (A. H (Vz, H
(22)
Taking Equation 21 into Equation 22 we have
W wwlle
$ (K₁ + N₁ - 1) |F
H + (K₃ + N₁ - 1) - Vz ||. H HIlEe
S(K+NN- 4 ||
H
H
Σ
H 1-
75
im/+1
1-1
-(K+ 1) (11 | 11 | H -lll
H-1lle
73
i=++
j=1
=
(8 | H.
an
H
(23)
s=1+1
j=l
13

---

Under review as a conference paper at ICLR 2023
where
a1 (Ki + N 1)(K, + N; ////////.///////////////////
3: = (Kr+ N₁ 2)I|V,,11==
(24)
71 = (Kr + - [
K1 and Nj denote the size of convolutional kerne Wi and activation map H; in layer 1, respectively.
During the LFC-ACT and HFC-ACT trainings. the activation map of convolutional layer satisfies
H H
F
H
H
H
M) - AM - x/-
(25)
H H
| H H
(26)
Taking Equations 25 and 26
into 23 we have GEB, and GEB a convolutional layer by
1/ V.
(a. Ap
|
a
II /=!
75
- GEB
(27)
i=1+1
|Vw,
V w.llos [ 2t
H 1l
on
II 7
-
GEB
(28)
x=1+1
Given the expression of GEB/ and GEB, by Equations 27 and GEB for convolutional layer given by
28
respectively, we have the
- (ar.: -1-
| all
on
(1
A II
i=1+1
j=!
Corollary 1. For a K x K convolutional kernel and a N x square matrix H, we have the
|[ + H||F S (K + N 1)] W |||
(29)
Proof. According to the relations between convolutional operation and Discrete Fourier Transfor- mation (Sundararajan 2001), W H satisfies
FFT(W H) FFT(ZP(W)) @ FF)(ZP(H)),
(30)
where F-T(-) denotes the discrete Fourier transformation; ZP(W) denotes pero-padding W into
(K + N 1) (K N -1) 1) matrix. According the Parseval's theorem (Diniz et al. 2010). (FT(ZP(W)) and (FT(ZP(H)) and FFT(W H) satisfy
[FFTTZP(W)) (K N W)|,,
(HF))( -K H||
(31)
-K [FFT(W = (K+N- ])[ H|
Taking || |A @ A2llF S
into Equation 31.. we have
(ZF()P(W)) @ FFT(ZP(H)) < FFT (
(32)
Taking Equation 31 into Equation
32
we have
K + 1)
H
[[FF((W) O FFT(H)
< [|]
(K + N - + ||]||
14

---

Under review as a conference paper at ICLR 2023
C GRADIENT ERROR BOUND (GEB) OF A LINEAR LAYER
We give the Gradient Error upper Bound (GEB) of a linear layer and proof in this section. Theorem IB. During the backward pass of a linear layer I, GEB and GEB satisfy
-
(
X
(33)
xmI++
f-1
where 04, 11,9t, ' 0 for 1 SIS SSS L are given by Equation (4 A/
H M
p;
H
M)llF: H = H+); and M denotes the I-D loss-pass mask.
Proof. For simplicity of derivation, we consider the case MiniBatch-1. In this case, Hr a vector; and W, is a 2-D matrix, for 1 1.. L. The backward propagation of linear layer is given by
Vz z. (Wi+1 V Z. .) (2,),
(34)
Vw: Z.
where Z, - + by; and by denotes the bias of layer 1. The case of MiniBatch 2 can be proved in an analogous way, which is omitted in this work.
According to Equation (34), we have the gradient of Z₁ given by
Vz- Vz -W Vz (Z) W4+ V 2, (Z,)
=W4+1 Vz.., @2'(2,) Wi+IV 2.,, Oo + Wi41 VzH, ( Wi+ V2
(Zi,
=W VZ-, 2 (o'o(Z)) (Z.) - (Z,)) (Vzi4. Wi++ V Zz.. o'
(35)
For activation functions ReLu(-), PeakyReLuu() Sigmoid((), Tanh(.) and SoftPlus(-). the gra- dient d') satisfies o"()) - in each differentiable domain. Combined with Cauchy-Schwarz
inequality ||A1 F A ||22|| (Horn & Johnson 2012). we have
||o((z))
(Zz) ||F - [2 - Zille S
[ -H-illl
(36)
According to inequality ||A @ A2/| S A A1l|r||A2ll
(Horn & Johnson 2012). we have the upper
bound of V/ - Vz, || given by
Viz Vz,lFF
< || 1|11l|l110 V Zz ||| (Zi) - o o'(Zi)ll= - ||Wiilll// - Zz. lee.lllllo
[Wi+i|//||/ ||||| - - |
|| [ - Fz |///////l///)
(Z))lPP,
a HI-1lle + V V ll.,
(37)
where ou and 7 are given by
(38)
> = ||H+allello (Z))]]
the value aj and "n depend on the model weight before backward propagation, which are constant with respect to the gradient. Iterate Equation 37 untill L where Vi -uzllr I ay H H In such manner, have
I Ile
i-1
|| - Vz,llF S ou
Σ ac - i=/+1
II
74. j=1
(39)
According to Equation
34p. we have the gradient of W, given by
Vw: Vw: Fz Vz, H
HT - Vz H + HT V2,
H
HHJ H ( HL
(40)
15

---

Under review as a conference paper at ICLR 2023
Taking Equation (39)
into Equation 40 have
ww
Vw,llr
Z2,
||
H
H.
11 +
V/
Vz, 11.lE1 Hillle,
< V [ -
H
H
H.
75
im141
5=1
(i H|- H - H 11e
a
7
(41)
1m1+1
where B is given by
Z. ||=
31 2/Vz, le:
(42)
7 |/ (Z)))
During the LFC-ACT and HFC-ACT trainings, the activation map of a linear layer satisfies
H
H
||
[H H - H M)ll - X
(43)
||
H
| H H H
- X/.
(44)
Taking Equations (43) and (44
into
(41
have the GEB and GEB, of a linear layer given by
V/ V/,
|| 5 ozl
H 1 + B1 B1)
Σ
II 7
GEB
(45)
1=/+1
V - 11 (1lrr) (1
| |Hl 81 81) + ||اااl
Σ (=+11
a
Il j=l
7
GEB
(46)
Given the expression of GEB; and GEBH by Equations (55 for a linear layer given by
and
(46) we have the GEB difference
31)(2* C -1laa/||H 31) (2) - 1t)
||
Σ az(A
II
T5-
f=/+1
5=1
D N VERSUS X IN DENSENET- 121
A further experiment is conducted study whether Xf >> X can be guaranteed for DenseNet- 121. Specifically, the values of X and X in the training (epoches 20, 40, and 60) of DenseNet-122 are given in Tables 3 and where the low-pass M mask satisfies W/N = 0.1 and W/N = 0.2, respectively; and Af and X are estimated based on the input activation maps of the four DenseBlocks. is consistently observed that X X for W/N 0.1 and W/N 0.2 in different training epochs. This indicates our proposed Theorem Tholds without loss of generality.
E PROOF OF THEOREM 2
We give the proof of Theorem in this section.
Theorem 2. For any real-valued function f(x and its moving average f(x) 1 2 f(i)dt, let F(w) and F(w) denote the Fourier transformation Maadisetii 1997) of f(x) and f(x), respec- tively. Generally, we have F(w)- F(w),, where [H(w) B
2B
16

---

Under review as a conference paper at ICLR 2023
Table 3:
W/N 0.1.
Epoch
20
40
60
Average
A
Af 298.20
X 184.913
A 142.668
DenseBlock-1
A 218.605
A 138.069
A 104.755
1.36
X 3.245
A 1.284
A 0.687
DenseBlock-2
A 1.713
A 0.689
X 0.372
1.87
D 0.387
X 0.160
X 0.084
DenseBlock-3
A Y 0.260
A 0.086
A 0.048
1.70
A 0.062
X 0.011
X 0.006
DenseBlock-4
A H 0.009
H 0.001
H 0.001
7.56
Average
1
2.95
3.12
3.30
3.12
Table 4: W/N 0.2.
Epoch
20
40
60
Average Af/ /
X -622.672
X 225.543
Af 173.595
DenseBlock-1
2.34
A 154.214
H 97.439
A 73.828
A = 3.632
X 1.440
A = 0.774
DenseBlock-2
A -..26
A H 0..33
X 0.285
2.72
X 0.445
A 0.179
A 0.095
DenseBlock-3
2.49
X 0.202
X 0.067
A 0.037
0.062
A 0.011
A 0.006
DenseBlock-4
7.56
A H 0.009
A 0.001
A 0.001
Average A
3.58
3.78
3.97
3.78
Proof. We adopt the limit operator to reformulate f(x) into
f(x) 2B
1
x22
f(t)d==
1 lim Σ
N-1
2B f(x +
2Bn ))
lim
N-1
Σ
f
f(x +
2Bn,
(47)
2B
N
+00
N
n=0
Taking Equation 47) into the Fourier Transform of f(x),, we have
N-1
F'(w)=
f(x)e
dr
lim N00
Σ
f(x
2Bn
)
dr
N
oc
N
2Bn
= lim N
Σ
f(x +
)
ier dx
N
N
n=0
(48)
N-1
N-1
lim
1
Σ
28
f(x)e
hoxt dar = F(w) lim
ti
38
N
N
N
N+000
N
n=0
m==
F(w) 1 - e
F(w)(( Rw2B)
lim N N(1
-iw2B
where denotes the imaginary unit.
Let (w)
LwBB 122
The magnitude of H (w) is given by
w2B iisin w2B
- cosw2B)2 + 2in
w2B
H(w)
[22B|
W2B
4 sin wB + sin wB cos wB
V 4sinn w B (sinn wB + cos? wB)
(49)
W2B
W2B
sin wB
wB
17

---

Under review as a conference paper at ICLR 2023
F COMPRESSION OF ID AND 3D ACTIVATION MAPS BY DIVISION
We give more details about DIVISION considering ID, 2D and 3D activation maps in this section.
F.1 ACTIVATION MAP COMPRESSION
DIVISION adopts average pooling to estimate the LFC by H AvcragcPooling (H,,) The value of block-size and moving stride is a unified hyper-paramen parameter B, which controls the memory of H The average pooling of ID. 2D and 3D activation maps are considered as follows.
Minibatch x Channel x N AveragPPogling D Minibatch x Channel x N/B
(50)
Minibatch x Channel x N x N AveragoPoolinga Minibatch x Channel N/ B N
NimbatthhsChannekNNaNNNYY AveragePopling3D MinibatchxChannell /N (N
To estimate the HFC, DIVISION calculates the residual value H MH-UUSaampligh where the UpSampling(.) enlarges H to the shape of H, via nearest interpolation. The up sampling of 1D, 2D and 3D activation maps are considered follows,
Minibatch Channel x [N/B] UpSampling D Minibatch x Channel (51)
Minibatch x Channel x [N/B] BJ x [/ UpSampling2D
Minibatch x Channel x X
UpSampling3D
MinbbatcbxChhameeX(///// X N/B]
MinibachhsChameekkWWWaaYY
Then, DIVISION adopts Q-bit per-channel quantization for the compression, where the bit-width Q controls the precision and memory cost of HFC after the compression. Let V denote Q-bit integer matrix, as the low-precision representation of HH! The detailed procedure of compressing H! into V is given by
Quant (H+*) = A (H) - 5t)],
(52)
where 6; denotes the minimum element in H A, (hmax - 61)/(22 1) denotes the quantization step; Amax denotes the maximum element in H denotes the stochastic rounding.
After the compression, the representation of H,, the tuple of (H). V/, A1, is cached to the memory for reconstructing the activation maps during the backward pass.
F.2 ACTIVATION MAP DECOMPRESSION
During the backward pass, DIVISION adopts the cached tuples of {H) VH. A,, 81) 0 E
1- reconstruct the activation map layer-by-layer. Specifically, for each layer 1, DIVISION dequantizes the HFC via H A, v + which is the inverse process of Equation 52 Then, the activation map is reconstructed via H, = UpSamplingHHt) +H where UpSampling() +) enlarges H! to the shape of H, via nearest interpolation. The cases of ID, 2D and 3D activation maps are considered in Equation 51
After the decompression, DIVISION frees the caching of (H) VH, Ar, di), and takes H. into [V 1. Vw,1 backward (H,, H W,) estimate the gradient for backward propagation.
G THEORETICAL COMPRESSION RATE OF DIVISION
The compression rate of DIVISION is estimated in this section. A general case convolutional neural networks (CNN) and multi-layer perceptron (MIP) considered for the estimation.
18

---

Under review as a conference paper at ICLR 2023
G.1 COMPRESSION RATE OF CNN TRAINING
Without loss of generality, we estimate the compression rate for a block of convolutional layer (conv), batch normalization layer (BN) and Relu activation. Most of existing backbones purely stacks Conv-BN-Relu blocks (He et al.) 20066] Huang et al. 2017 ( et al.) 2015 Tan Le 2019; Simonyan Zisserman 2014), which makes our estimated compression rate hold practice. Generally, the compression rate defined as the memory reduction ratio after the com- pression. To be concrete, let Minibatch x Channel x N denote the shape of activaition maps for Conv-BN-Relu block; given the biock-size B and bit-width Q, DIVISION has the compression rate activation maps given by Theorem 3A.
Theorem 3A. DIVISION has average activation map compression rate for conv-BN-Rell block given by
Mem of H
9
(53)
Mem of (HL, VH. A)
min B.
where Minibatch x Channel N N the shape of activation map Hú; B denotes the block-size of LFC average pooling; and Q denotes the bit-width of HFC quantization.
Proof. For each mini-batch updating of normal training, conv-layer or BN-layer caches N f1oat33 x 4byte/float
byt activation map; Relu operator caches N²int& 1byte/intB = byte activation map. For each mini-batch updating of DIVISION, conv-layer
N² or BN-layer caches min min brrbfflotts x 2byte/bfloat16
Mii(B byte LFC; and
2N
Q. bit/byte Q N HFC; and spends 2bfloat16 2byte/bfloat16 4byte for A and 8/. Moreover, a Relu operator caches <it x Gbyle/
8
N byte activation map.
&
Therefore, the average activation map compression rate of a conv-BN-Relu block given by
4N2 x 2
9
R
(54)
2N3 (B4,
+ N +4) x
x2 3 N2
+
+ N2
min B N2
4
higher compression rate indicates more effective compression. is observed that the compression
rate grows with B and N, and decreases with Q. In our experiments, we found B=8 8 and Q =0 can provide loss-less model accuracy. In this condition, the shape of activation maps satisfies N 2 7 for ResNet-50 and WRN-50-2 on the ImageNet dataset (He et al.) 2016). According to Equation 53 we have RResNer.0, 5WRN-50.2 2 10.35.
G.2 COMPRESSION RATE OF MLP TRAINING
We estimate the compression rate for a 1 Dinear-Reill block in Theorem 3B. An MLP simply stacks multiple 1 Linear-Rell blocks, such that our estimated compression rate holds for MLP models.
Theorem 3B. DIVISION has average activation map compression rate for linear-Relu block given by
Mem H
5
(55)
Mem of (HL VH,
2
10+4++
Q
B.n
where Minibatch x N is the shape of activation map Hú; B denotes the block-size of LFC average pooling; and Q denotes the bit-width of HFC quantizzation.
Proof. For each mini-batch updating of normal training, a linear-layer caches Nfloat32 x 4byte/float 32
4Nyyyt activation map; a Relu operator caches Nint8 lbyte/int8 Nbyte activation map. For each mini-batch updating of DIVISION, a linearr hear--lyer caches min[B.Ny byte LFC; and Qbbi /
min(B.NT bfloat 6 2byte/bfloatl 2bttebbflaat1
N
Q Nbyte HFC; and spends
4byte for A and o, Moreover,
19

---

Under review as a conference paper at ICLR 2023
Relu operator caches Nbit x bbyte/bit N byte activation map. Therefore, the average activation map compression rate of a linear-Ree block given by
4N + N
R
(56)
min [B.N]
min NB
H EXPERIMENT SETTING
We give the experiment setting including the datasets, baseline methods and model architectures in this section.
Datasets. We consider CIFAR-10. 10, CIFAR-100 (Krizhevsky et al. 2009) and ImageNet (Deng al. 2009) datasets in our experiments. CIFAR-10: An image dataset with 60,000 color images in 10 different classes, where each image has 32 32 pixels. CIFAR-100: An image dataset with 60,000 color images in 100 different classes, where each image has 32x pixels. ImageNet: A large scale image dataset which has over one million color images covering 1000 categories, where each image has 224 x 224 pixels.
Baseline Methods. Normal: Caching the exact activation map for backward propagation. BLPA: A. systemic implementation of ACT by (Chakrabarti & Moseley (2019), which only supports ResNet- related architectures. AC-GC: A framework of ACT with automatic searched bit-width for the quantization of activation maps (Evans & Aamodt (021). ActNN: Activation compression training with dynamic bit-width quantization, where the bit-allocation minimizes the variance of activation maps via dynamic processing (Chen et al., (021). Checkpoint: Caching some key activation maps to reconstruct other activation maps via replaying parts of the forward pass during the backward pass (Chen et al. 2016). SWAP: Swapping the activation maps the CPU during the forward pass the memory consumption of GPU, and reload the activation maps to GPU during the backward pass (Huang et al. 2020).
DNN Architectures. For benchmarking the model accuracy, we consider ResNet-18 (top-1 ac- curacy 94.89%) and ResNet- 164 (top-1 accuracy 94.9%) the CIFAR-10 dataset; DenseNet- 121 (top-1 accuracy 79.75%) and ResNet-166 (top-1 accuracy 77.3%) on the CIFAR- 100 dataset; and ResNet-50 (top-1 accuracy 76.15%) well DenseNet-161 (top-1 accuracy 77.65% on the ImageNet dataset. Our reproduced validating accuracy is consistent with the official results of torchvision
[2] Moreover, for benchmarking the memory cost and training throughput, we consider the large models ResNet-50 and WRN-50-2 on the ImageNet dataset.
IMPLEMENTATION DETAILS ABOUT DIVISION AND BASELINE METHODS
DIVISION: DIVISION adopts block-size 8 (B = 8) and 2-bit quantization (Q = 2) to compress the activation maps of linear, convolutional and BatchNorm layers, where the theoretical compression rate is not less than 10.35x. For the operators without quantization error during backward propa- gation such pooling layers, ReLu activation, and Dropout, DIVISION follows the algorithms in Appendix Jorendixtto compress the activation maps. Other hyper-paramete settings are given in Table []
BLPA: Existing work (Chakrabarti & Moseley 2019) has shown that BLPA requires least 4-bit ACT for loss-less DNN training. We follow this setting for BLPA, where the compression rate of activation maps is not more than 8x. AC-GC: AC-GC follows existing work (Evans & Aamodt
2021 to take the multiplicative error (1 + exc-GG) = 1.5, where the searched bit-width enables AC- GC to satisfy this loss bound (training loss not more than 150% of normal training). In this setting, AC-GC finalizes the bit-with as 7.01 after the searching, which has nearly 4.57 compression rate activation maps. ActNN: ActNN adopts 2-bit ACT and dynamic programming for searching the optimal bit-width specific for each layer, and uses per-group quantization for compressing the activation map, which has approximately 10.5x compression rate of activation maps. Such experi- mentally setting is denoted as L3 strategy in the original work (Chen al. 2021), and we follow this
20

---

Under review as a conference paper at ICLR 2023
Table 5: Hyper-parameter setting.
Dataset
CIFAR-IO TO
CIFAR- 100
ImageNet
Architecture
ResNet-T8
ResNet-164
ResNet-164
DenseNet-122 121
ResNet-500
DenseNet-161
Epoch
100
100
200
100
120
120
Batch-size
256
256
256
256
256
256
Initial LR
0.1
0.1
0.15
0.1
0.1
0.1
LR scheduler
Cos LR
Cos LR
Cos LR
Cos LR
Cos LR
Cos LR
Weight-decay
0.0005
0.0005
0.0005
0.0005
0.0001
0.0001
Optimizer
SGD
SGD
SGD
SGD
SGD
SGD
SGD Momentum
0.9
0.9
0.9
0.9
0.9
0,9
Block-size B
8
8
8
8
8
Bit Bit-width Q
2
2
2
2
import torch. utils. checkpoint checkpoint import to as nn
class ResNet (nn. Module)
def (sel,, x: Tensor) -> Tensor:
x self. (x)
X sclf.bnl (x)
sclf.relu(x)
X self.maxpool (x)
X checkpoint..checkpoint (sel http://cooooottt.eee....oetooooo btt.ttteeeekk_aaerlllll x)
X checkpaint..heekppoii beeee bottllenecklaaee2,,, x)
checkpoint..checkponnt /etee
x)
checkpooint.chhskkooitt (self (seeettottt..ee.eeeeeeee www.bettctttecccccceeeeec layer4,
X self. avgeoovgpo
X
(x, 1)
X self.fc(e (x)
return
Figure 8: Implementation of checkpointed ResNet-50 and WRN-50-2.
setting in this section. Checkpoint: Checkpoint relies on the model-specific design of the check- pointed layers. We find that it provides good trade-off between memory cost and running speed to checkpoint the activation map each Botteneck block ResNet-50 and WRN-50-2 (a ResNet-50 WRN-50-2 has four Botteneck blocks). To implement the checkpointing of inter mnter--eeia layers, revised the forward function of the ResNet-50 and WRN-50-2 as Figure [ SWAP: For the SWAP method, the memory utilization considered both GPU and CPU because the activation maps stored on both GPU and CPU in SWAP.
J COMPRESSION OF POOLING LAYERS, RELU ACTIVATIONS, AND DROPOUT
DIVISION follows Algorithms 2 and 5 to compresse the activation map of a Max- Poooing layer, Average-Poolin layer, Relu activation and Dropout operator, respectively. For the pooling layers, we consider simple case kernelsize movingstride = k. General cases with different kernelsize and movingstride can be designed in analogous ways.
21

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Max-Pooling layer.
Algorithm 3
layer.
1: Function Forward (Hr-1, k. *kwarg))
I:
Function Forward (H kt sekwargs)
2: Hi, Vi-1" -Max-Pooling(HH kwargs)
2:
H, Avvg-Polingg(H--1 kwargs)
Pack Cache Vi-1 using Int8.
3:
return H,
4:
return H.
4:
5:
5:
6:
Function Backward(VH,)
6:
Load Vi-1 and VH, = -axx @ VH;
7:
Function Backward VH,
VH, 1ax6 VH:
VH.-1 Vi-1 @ VH,
VH k A
10: return VH;-1
10: return VH:-1
Algorithm 4 Relu operator.
Algorithm 5 Dropout operator.
I: Function Forward (H₃-1)
Function Forward (H;-1)
Vi-1 = sgn(HH, (g
Generate Minibatch Channel x N binary matrix V-1
3: H, = Vi-1 O Hi-1
3: following the Bernoulli distribution with dropout probability
4: Pack & Cache Vi-1 using Int8
4: Hi Vi-1 @ H
5: refurn H,
5: Pack & Cache Vi-1 using Int8.
return H,
7:
7:
8:
Function Backward(Vir
8:
Function Backward(V
9:
Load Vi-1-
9:
Load Vi-1-
10:
VHI-1 = Vi-1 VH;
10:
VH,-1 V @ VH.
return VH:-1
11: return VH.-1
K EVALUATION OF DIVISION ON MULTI-LAYER PERCEPTRONS (MLPS)
We conduct experiments on the GAS dataset (Dua Grafi 2017 (128-dimensional features, 13910 instances, 6 classification task). The classification model is 4-layer MLP (128 neuros in the input layer, neuros in the output layer, and 64 neuros in the hidden layer); The setting of DIVISION is B = 16 and Q = 2. The model accuracy and memory cost of activation map are given in Table 6 observed that DIVISION has 7.3x compression rate with only 0.07% degradation of model accuracy. This indicates the effectiveness of DIVISION on the MLP models.
Table 6: Model Accuracy the GAS dataset.
Training Normal Training DIVISION
Testing Accuracy (%)
Memory (KB)
Compression Rate
98.92
250
N/A
98.85
34.2
7.3x
L EVALUATION OF DIVISION ON VISION TRANSFORMERS
We conduct experiments to study the performance of DIVISION on Swin Transformer (Liu et al. 202 In this experiment, Swin Transformer-T is trained the ImageNet datset, where DIVI- SION is compared with Mesa (Pan et al. 2021). an ACT framework for visual transformer. The model accuracy and memory cost (with batch-size 128) given in Table 7 It is observed that DI- VISION achieves almost the same model accuracy compared with normal training. This indicates DIVISION can effectively compress the training of the vision Transformer.
Although DIVISION shows slightly lower accuracy lower than Mesa, the compression rate is signif- icantly higher than Mesa. Moreover, DIVISION can be applied general vision models, including MLPs, CNNs, and vision transformers; while Mesa explicitly designed for vision transformers.
BV I-1 reserves the locations of each kemne-wiie max-values in H
22

---

Under review as a conference paper at ICLR 2023
Table 7: Accuracy of Swin Transformer-T on the ImageNet dataset.
Method
Accuracy (%)
Memory (GB)
Compression Rate
Normal Training
81.2
11.81
N/A
DIVISION
81.0
4.19
2.8x
Mesa
81.3
5.37
2.2x
M EVALUATION OF DIVISION ON DEPTHWISE AND POINTWISE CONVOLUTIONAL LAYERS
To evaluate DIVISION on the depthwise convlution and pointwise convulution layers, we conduct experiments of MobileNet-V2 on the CIFAR-I0 and CIFAR- 1-00 datasets, where the model accuracy are given Table 8 It is observed that DIVISION achieve nearly the same accuracy compared with normal training. This indicates the effectiveness of DIVISION for the depthwise convlution and pointwise convulution layers.
Table Accuracy of MoblicNet-VV V2 on the CIFAR-10 and CIFAR-100 100 datasets. VobileNet-V2 (CIFAR-10) MoblieNet-Ve V2 (CIFAR-T00)
Method Normal Training
91.9
71.0
DIVISION
91.8
70.6
N RE-UTILIZATION OF HYPER-PARE PTPERRPARRAETER SETTINGS ACROSS DIFFERENT MODEL ARCHITECTURES AND DATASESTS
We conduct the follow-up experiments study whether the hyper-parameter setting of DIVI- SION has a consistent effect on different model architectures and datasets. Specifically, the hyper- parameters of DIVISION are selected from B {8, 18) and Q E {2,8) train ResNet- 18 and MobileNet-V2 on the CIFAR-10 10 and CIFAR-100 datasets. The model accuracy is given in Table observed B and Q have a consistent impact on different model architectures and datasets: accuracy slightly grows with Q and considerably reduces with B. This indicates can reuse the hyper-parameter setting DIVISION on CIFAR-IO to CIFAR-100 with the same model architec- ture, can reuse the setting with ResNet-18 18 and MobileNet-V2 on the same dataset.
Table 9: Accuracy of ResNet- 18 and MoblieNet-V2 V2 with different hyper-parameter settings.
B 18
2
B = 8
2
B = 8
ResNet-18 CIFAR-10
78.7
94.6
94.9
ResNet-18 CIFAR-100
73.2
76.9
77.0
MobileNet-V2 CIFAR-I0
10.0
91.7
91.0
MobileNet- V2 CIFAR- 100
62.4
70.6
71.6
O COMPARISON OF DIVISION WITH CHECKPOINT STRATEGY OF MEGATRON-LM
DIVISION is compared with the checkpointing strategy of Megatron-L.M (Shoeybi et al. 2019) According to the official guideline, Megatron-LMM checkpoints the activation map after each trans- former block. We follow this strategy to checkpoint the activation map after each transformer block in the Swin Transformer, and after each Bottleneck block in the ResNet-50. The memory cost (on the ImageNet dataset with batch-size 128) is given in Table TO It is observed DIVISION has a higher compression rate (2.8x for Swin Transformer-T and 7.9x
x for ResNet-50) than the Check- point strategy of Megatron Legatron-LM LM (2.3x for Swin Transformer-T and 2.27 for ResNet-50). This indicates the effectiveness of DIVISION over the checkpoint strategy of Megatron-L.M.
23

---

Under review as a conference paper at ICLR 2023
Table 10:
Memory of DIVISION and Checkpoint strategy of Megatron-I.M.
Memory cost (GB)
Normal Training
DIVISION
Checkpoint strategy of Megatron-LMM
Swin Transformer-T
11.81
4.19 (2.8x)
5.14 (2.3x) 4.68 (2.27x)
ResNet-50
10.62
1.35 (7.9x)
P EFFECT OF BIT-WIDTH ON THE FIXED QUANTIZATION
To demonstrate the effectiveness of dual activation precision, DIVISION (B = 8, 2= is com- pared with the fixed quantization under different bit-width. The model accuracy of ResNet-50 on the ImageNet dataset given in Table II observed the training fails to converge an optimal solution under 2-bit quantization, even though it performs favorably under 4-bit and 8-bit quantiza- tion. This result is consistent with existing work (Chen et al. 2021) (Table in (Chen al. 202T )
In contrast, DIVISION achieves 75.9% accuracy when adopting 2-bit quantization to compress the HFC. This indicates the effectiveness of dual activation precision terms of the model accuracy under low bit-width quantization.
Table 11: Model accuracy of fixed quantization under different bit-width.
Bit-width
2
4
8
Fixed Quantization
0.1
76.05
76.35
DIVISION
75.9
omit
omit
Q EFFECT OF MINI-BATCH SIZE ON THE THROUGHPUT
As supplementary of Section 5.5 follow-up experiment is conducted to study the effect of batch- size on the training throughput. The result on the CIFAR-10 10 dataset given in Table 12 It is observed: for the experiment w/o AMP, the throughput significantly grows the batch-size grows from 64 to 128, but is almost unchanged when the batch-size 2 128; for the experiment w/ AMP, it grows continuously when the batch-size grows from 64 to 256, and becomes stable when the batch-size 2 256.
Table 12: Traininig throughput versus the batch-size.
images/s w/o AMP
Batchsize=64
Batchsize=128
Batchsize=256
Batchsize=512
Batchsize=1024
1185.27
2184.36
2335.41
2394.84
2409.99
w/ AMP
1273.46
2285.87
3753.06
3981.95
4068.67
Intuitively, as the batch-size grows, the GPU can parallel process more images per second, until the GPU is 100% utilized (Goyal et al. 2017). In the experiment w/o AMP, the GPU is almost 100% utilized when batch-size 2 128; while this happens when batch-size 2 256 in the experi- ment w/ AMP. More images can be processed in the experiment w/ AMP, since employs bfloat operations in the training, in contrast with the float32 operations in the training w/o AMP, where 6 operation has nearly half of the computation cost of a float32 operation. Therefore, the training throughput significantly grows from 2086 (images/s) to 3753 (images/s) as the batch-size grows from 128 to 256 in the experiment w/ AMP, but is almost unchanged (2184 images/s vs 2335 images/s) in the same condition in the experiment w/o AMP.
R DETAILS ABOUT THE COMPUTATION INFRASTRUCTURE
The details about our physical computing infrastructure for testing the training memory cost and throughput given in Table 13
24

---

Under review as a conference paper at ICLR 2023
Table 13: Computing infrastructure for the experiments.
Device Attribute
Value
Computing infrastructure
GPU
GPU model
Nvidia- -TXia-
GPU number
GPU Memory
24567 MB
IMPLEMENTATION DETAILS OF THE EXPERIMENT IN SECTION 5.4
We give the implementation details of the experiment in Section Specifically, DIVISION w/o HFC takes block-size / = 4 for estimating LFC; DIVISION w/o LFC takes the bit-width Q Q=2 for the quantization of HFC; DIVISION combines these settings for the training; and Fixed Quant has 2-bit per-group quantization of activation maps during the backward pass of the training, where the group size of quantization is 256. Other training prper-parrmeters are given in Table 5
25


Reviews:
[{"review_id": "dc4esgwMpg", "reviewer": "Reviewer_xXpe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The presented work focuses on improving the efficiency of the training process by reducing memory cached during the backward propagation. Activation maps are split into low and high frequency components based on the observation that they don't affect the accuracy of the model equally. High precision copy of low frequency components is preserved, while the high frequency components are compressed, achieving over 10x compression of activation maps. \n", "summary_of_the_review": "The work is interesting and presented claims are clear and supported by performed experiments. Some additional experiments would help to support it even more and show that it can be used in a wide range of applications, instead only for a small subset of solutions. Overall, the work is good in my opinion. ", "strength_and_weaknesses": "The proposed method is novel and interesting. The work is well structured. The problem is clearly introduced along with other existing methods, comparing their limitations and proposing ideas for improvements.  The stated hypothesis is analyzed from the experimental and theoretical perspectives.\n\nThe main weakness of the work is a limited number of datasets and topologies the method was evaluated with. It would be interesting to show that division into low and high frequency components can be applied to other problems as well. Specifically, it would be important to evaluate other types of layers that operate at the depth dimension, i.e., 1x1 convolutions. \n\nAnother limitation is the need selection/finding of other hyper parameters, i.e., B and Q. Some additional experiments showing that either they can be reused for other applications or quickly selected would be beneficial. Otherwise, the gain in training efficiency might not be that significant, if additional trainings have to be performed to find the best configurations. \n\nSome other ideas for improvement of the method include the use of bfloat16 instead of float16 for storing LFC and the use of per tensor symmetric quantization. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bYdQjOXgZl", "reviewer": "Reviewer_R65G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of the paper observe that the high-frequency component (HFC) of activation maps possesses the majority of memory cost for training but the model performance can be preserved even in the absence of the HFC component of activation maps. Based on the observation, the paper proposes a new activation compressed training method called Dual Activation Precision (DIVISION) that keep the low-frequency component (LFC) of activation maps in high-precision while quantizing the HFC of activation maps into low-bit.", "summary_of_the_review": "It is questionable whether Section 3 has something to do with Section 4 as mentioned in Weakness 1 and 2. I would encourage the authors to revise the paper by addressing aforementioned weaknesses.", "strength_and_weaknesses": "- Strengths\n\n1. The paper introduce the concept of frequency domain to compress activation maps.\n\n2. The authors theoretically and empirically show that the LFC component of activation maps are much more critical to maintain the model performance than the HFC component of them.\n\n3. DIVISION consistently compresses activation maps with about 10x compression rate, while maintaining comparable model performance to normal training.\n\n\n\n- Weaknesses\n\n1. It is wondered whether $\\lambda_l^{L} > \\lambda_l^{H}$ in Theorem 1 can be guaranteed even when a different architecture (e.g., DenseNet-121) is utilized or the proportion of $W$ to $N$ is small (e.g., $W/N = 0.1$)\n\n2. Even if $\\lambda_l^{L} > \\lambda_l^{H}$ in Theorem 1 can be guaranteed for different architectures and the small proportion of  $W$ to $N$, it is doubtful if Theorem 1 is still valid when DIVISION is exploited instead of DCT. In other words, when applying the average pooling rather than the inverse DCT, does Theorem 1 still hold so that $\\text{GEB}_l^{L} < \\text{GEB}l^{H}$? If not, the theoretical analysis might seem to be ineffective\n\n3. When quantizing the HFC component of activation maps, the stochastic rounding is employed. Then, to obtain $V_l^{H}$ in Eq. (9), it is required to draw size($H_{l-1}$) samples, which seems to impose a significant burden on training. In addition, as $V_l^{H}$ varies depending on sampling, the performance of DIVISION might be likely to be different. Is there any performance degradation when using rounding instead of stochastic rounding?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TLD2SJhyObr", "reviewer": "Reviewer_Eerz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced an activation compressed training (ACT) framework, called DIVISION. Specifically, DIVISION is inspired by the insight that DNN backward propagation mainly utilizes the low-frequency component (LFC) of the activation maps, while the majority of memory is for the storage of the high-frequency component (HFC) during the training. In this case, the proposed method stores two memory-efficient copies of activations instead of the original activations, which are average-pooled low-frequency activations and low-bit quantized high-frequency signals. At the time of backpropagation, DIVISION recovers the original size of activations by upsampling low-frequency signals and dequantizing high-frequency signals. Extensive experiments have shown the proposed method works and achieves faster training throughput than related baselines\n", "summary_of_the_review": "Some experimental results are clearly not convincing, which leads to rejection. ", "strength_and_weaknesses": "### Strength\n\n1. This paper has a clear insight and motivation. In the experiments, the authors have shown that low-frequency components are more important for gradient computation than high-frequency components. To this end, the proposed DIVISION is well-motivated. The idea is also novel in the literature.\n2. The ablation studies are comprehensive. For example, Section 5.5 has clearly shown the effect of different hyperparameters in the proposed model.\n3. This paper is well written and easy to follow.\n\n### Weakness\n\nOverall, I appreciate the idea and novelty of this paper. However, the experiments and evaluations in this paper need major revision. Some results and observations are quite unusual. For example,\n\n1. I have experimented with ActNN (using official code) and Checkpointing (on both Vision Transformers and CNNs) to check the correctness of the experiment. In my observation, checkpointing is a strong baseline which should be faster than many ACT frameworks. However, the large gap between ActNN and Checkpointing in Figure 6 (a) is confusing. Moreover, according to the provided experimental settings (Figure 8), I compared checkpointing with ActNN (L3) based on ResNet-50 on my local RTX 3090 machine.  With a batch size of 64 and 224 $\\times$ 224 images, the results show that checkpointing is faster (310 images/s v.s. 280 images/s) than ActNN under a similar amount of memory-saving (3.2GB v.s. 3.4GB). Therefore, I keep skeptical about Figures 6 (a) and (b).\n2. In Figure 6 (c), fixed quantization achieves much worse performance, which is very unnatural. Fixed-bit quantization in ActNN performs favorably on ResNet. For example, L2 of ActNN quantizes activation into fixed 4 bits, but it still performs well. The authors need to give a detailed explanation here.\n3. In Figure 7 (b), it is counter-intuitive that using a batch size of 128 significantly slows down the throughput compared with using a batch size of 256. This figure also needs to be explained further.\n4. Despite the ablation studies, the main experiment in Table 1 shows that the memory-saving gain compared with ActNN is very minor.\n5. Apart from CNNs, how is the performance of DIVISION on Vision Transformers? And how do you compare with GACT [A] and Mesa [B]?\n\n\n### References\n[A] Liu, Xiaoxuan, et al. \"GACT: Activation compressed training for generic network architectures.\", *International Conference on Machine Learning (ICML) 2022*.\n\n[B] Pan, Zizheng, et al. \"Mesa: A Memory-saving Training Framework for Transformers.\" *arXiv preprint arXiv:2111.11124* (2021).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "02YiRAMID1u", "reviewer": "Reviewer_PUeD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for activation compressed training. The proposed method, DIVISION, takes a frequency-domain view of the activation map. They pointed out that high-frequency features are more robust to quantization than the low-frequency ones. To utilize this observation, the authors extracts low-frequency features (approximately) by average pooling, and only quantizes the high-frequency residual. They present a theorem to show only perseving low-frequency features yields tighter gradient error bound that only perserving high-frequency ones. The experiments are done for ResNets and DenseNets. Compared to the previous approach ActNN, the proposed method can achieve better accuracy and higher throughput due to its simplicity (does not need to estimate the per-tensor precision).", "summary_of_the_review": "The paper does make improvements to ACT for deep learning. The main weaknesses are the applicable range of the proposed method and the significance. ", "strength_and_weaknesses": "Strength:\n- Adopting frequency-domain features for ACT is somewhat novel.\n- The proposed approach is simple and effective.\n- The presentation is clear.\n\nWeaknesses:\n- The proposed approach might be limited to convolutional networks, or at least assuming that the feature map has the shape [N, C, H, W].\n\nThe authors can argue that the proposed method can be generalized to 1D or 3D problems with 1D/3D pooling. But as the feature statistics can differ across problems with different dimensions, whether the proposed approach is also effective for 1D/3D problems is still unclear. Moreover, I am not sure if the proposed approach can be applied to MLPs (just pure MLP, not MLP-Mixer, which are in fact 1x1 convolution), as MLPs are permutation-invariant.\n\nTo make the paper stronger, the authors might want to test on more transformer architectures, including the vision ones (Swin, etc.) and the text ones (such as BERT).\n\n- The significance might be still somewhat limited. There are some improvements over ActNN, like the accuracy of DenseNet-161 and the throughput. But I am not sure if the improvement is significant enough for an ICLR publication. The authors might also want to compare with more recent checkpointing approaches, such as Mesa and the hand-crafted checkpointing strategy for transformers (like those in Megatron-LM and DeepSpeed).\n\nOther questions:\n- Does the proof of Theorem 1 has anything to do with DCT or the frequency domain? I suspect that simply showing that |H^L - H| < |H^H - H| might be also sufficient to prove Theorem 1?\n\n- Should the window size be different for feature maps of different resolution? For example, the final 8x8 feature maps might require a smaller window size.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject