Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropagation
Authors: Ziyu Jiang
Xuxi Chen
Xueqin Huang
Xianzhi Du
Denny Zhou
Zhangyang Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Memory Efficiency
Sparsity
Prune

Abstract: Transfer learning from the model trained on large datasets to customized downstream tasks has been widely used as the pre-trained model can greatly boost the generalizability. However, the increasing sizes of pre-trained models also lead to a prohibitively large memory footprints for downstream transferring, making them unaffordable for personal devices. Previous work recognizes the bottleneck of the footprint to be the activation, and hence proposes various solutions such as injecting specific lite modules. In this work, we present a novel memory-efficient transfer framework called Back Razor, that can be plug-and-play applied to any pre-trained network without changing its architecture. The key idea of Back Razor is asymmetric sparsifying: pruning the activation stored for back-propagation, while keeping the forward activation dense. It is based on the observation that the stored activation, that dominates the memory footprint, is only needed for backpropagation. Such asymmetric pruning avoids affecting the precision of forward computation, thus making more aggressive pruning possible. Furthermore, we conduct the theoretical analysis for the convergence rate of Back Razor, showing that under mild conditions, our method retains the similar convergence rate as vanilla SGD. Extensive transfer learning experiments on both Convolutional Neural Networks and Vision Transformers with classification, dense prediction, and language modeling tasks show that Back Razor could yield up to 97% sparsity, saving 9.2x memory usage, without losing accuracy. The code is available at: https://github.com/VITA-Group/BackRazor_Neurips22.

Back Razor: Memory-Efiiiien Transfer Learning by Self-Sparsified Backpropagation
Ziyu Jiang", Xuxi Chen Xueqin Huang', Xianzhi Du Denny Zhou Zhangyang Wang
Texas A&M University 'University of Texas at Austin &Google
xuelggg xueq13)@tamu edu, exx {xianzhi, demnyzhou/@goggle.com
edu
Abstract
Transfer learning from the model trained on large datasets to customized down- stream tasks has been widely used as the pre-trained model can greatly boost the generalizability. However, the increasing sizes of pre-trained models also lead to prohibitively large memory footprints for downstream transferring, making personal devices. Previous work recognizes the bottleneck
them unaffordable of the footprint to be the activation, and hence proposes various solutions such as injecting specific lite modules. In this work, we present a novel memory- effficient transfer framework called Back Razor, that can be plug-and-play applied to any pre-trained network without changing its architecture. The key idea of Back Razor is asymmetric sparsifying: pruning the activation stored for pack-propagation. while keeping the forward activation dense. It is based on the observation that the stored activation. that dominates the memory footprint, is only needed for back- propagation. Such asymmetric pruning avoids affecting the precision of forward computation, thus making more aggressive pruning possible. Furthermore, we conduct the theoretical analysis for the convergence rate of Back Razor, showing that under mild conditions, vanilla SGD. Extensive transfer learning experiments on both Convolutional Neu- ral Networks and Vision Transformers with classification, dense prediction, and language modeling tasks show that Back Razor could yield up to 97 % sparsity, saving 9.2x memory usage, without losing accuracy. The code is available https: /etps://githutt github. com www/VITA /ttp/VVTAAA--a Satroup/Backazzo_bbege SeneotRazorNNeuiis222222
method retains the similar convergence rate
1 Introduction
Edge devices equipped with deep learning applications are becoming ubiquitous. However, only deploying pre-trained models not sufficient: these edge devices may keep collecting novel data that general pre-trained models have never seen before, and fine-tuning on them would notably improve the model's performance. A typical approach to upload these newly collected data cloud servers for fine-tuning, and download the trained model from the cloud to local devices. However, such a way has potential privacy issues due to data sensitivity and also imposes undesirable pressures on the Internet bandwidth, brought by transmitting both the collected data and trained models. These challenges motivate researchers to explore on-device learning methods, i.c., how to fine-tune pre-trained models on memory limited devices.
One of the main challenges of fine-tuning on edge devices is the memory constraints. Figure exemplifies the memory usage of training some popular networks: ResNet50 with small batch size of 16 can easily exceed the memory limitation of onboard devices. The newly emerged Vision
Equual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Transformer (ViT) would incur even more memory consumption. Recent studies show that the activations stored for back-propagation take up large proportion memory cost during training. Therefore, popular model compression techniques such weight pruning and quantization are not highly efficient reducing training memory footprints. Parameter efficient training method like Bitfit also fails to compress the activation to memory limitation (see Figure T Gradient checkpointing can reduce the training memory by storing only a subset of activations, but requires more FLOPs at back- prok-propagation due to the re-computation of discarded activations. Using smaller batch size or half-precision for training can also lower the memory cost, but they incur a decrease in the model's accuracy and training speed. [ successfully avoided saving activation via freezing the weights for convolutional neural networks and only learning a newly added the lite residual module, thus no need to store the intermediate activations. However, it comes at certain price of the transfer performance.
new stream of works studied how to directly compress the activations. They mainly leverage the quantiza- tion technique that discretizes the ac- tivation tensors so these tensors can be represented by efficient data for-
3000
Parameters
2500
Activations
Memory (Mb)
2000
mats and require less memory [2] 4).
However, the activation pruning tech-
1500
niques have been less explored. [5]
Raspberry
applied random activation masks each layer in both forward and back- ward passes to reduce latency. 6
1000
Pi3
500
R Raspberyy
1 DRAW
250
integrated weight pruning and activa- tion pruning by learning importance scores on neuron's s outputs. Moreover, they adopt the symmetric scheme that prunes the activation at both the for- ward and backward processes. Such a pruning scheme undermines the pre- cision of models' outputs, leading significantly worse performance when the pruning ratio is high.
0
ResNet50
VIT-B
Figure 1: Training memory usage and on-device memory limita- tion illustration. The training memory is calculated under batch size of 16. VIT-B (Bitfit) denotes the ViT-B model tuned with Bitfit.
In this work, we present general activation compression method called Back Razor. Inspired by the observations that the activations are only stored for back- procpproppgation, we adopt the asymmetric pruning scheme that only prunes the activation during the backward process. Specifically, after activation used by subsequent layer, our method replaces it with pruned activation which carries lower memory cost. During the backward phase, the gradients are calculated with the approximated activation, but only with small error between those with the exact activation.
Our contributions are summarized following:
We propose memory-efficient transfer learning method called Back Razor that prunes the activation for pock-propagttion. Our proposed method can be easily applied to various backbones, including convolutional neural networks (CNNs) and vision transformers (ViTs).
We provide theoretical analysis of self-sparsified back-propagation method for CNNs. Specifically, we discover that under mild assumptions, our method can share similar convergence rate as vanilla SGD.
We extensively verify the proposed Back Razor both CNNs and ViTs for multiple datasets and show that the proposed method can achieve up to 97% sparsity with saving 9.2x memory saving without losing accuracy, By further verifying GPU, the proposed method achieves 2.7x on-device memory efficiency for ViTs.
Related Works
2.1 Pruning and Sparse Training
Pruning can be operated at different levels, egg., pruning on weight, weight gradients, activations, and activation gradients. Pruning weights are most widely studied since it accelerate the inference

---

Storage
Sparsify
Forward
Backward
Figure Pipeline of the proposed Back Razor.
phase 7 08. 9
pruning on weight gradients help reduce communication cost especially in 13
distributed training [0] 11] pruning on activation gradients 12
14
15 or activation itself have
S randomly generated static sparse masks and applied them on each layer in both forward and backward passes. [6] proposed integral pruning that prunes both activation and weights by consecutively learning and applying two masks on weights and activations, However these works are adopting the symmetric pruning scheme, ie., pruning the activation at both the forward and the backward process. Pruning the activations at each layer brings errors, and they will accumulate as the forwarding proceeds. Backdrop [I6] improves the generalizability of large batch gradient descent via randomly masking parts of the backward gradient propagation. SWAT [17] empirically explores sparsifying both weights and activations for training CNNs from scratch, and the authors also discovered that pruning activations only for backprogration can effectively preserve accuracy better. While our idea shares similarity to SWAT, Back Razor dedicated to transfer learning and applicable to general backbones (both CNNs and ViTs), and we also supply the theoretical analysis for such self-sparsified backpropogation for the first time.
been less addressed since they are mostly dynamic during training.
2.2 Memory Efficient Transfer Learning
Transfer Learning from a model pre- pre-trained on large datasets to small datasets has been widely used for many applications [18 19 20 21.. In contrast to the pre-training that is always conducted on large clusters, transfer learning may happen on personal computing devices with limited memory constrains. To fit the large model to small devices, common choice is fine-tune the last several layers of
18 23 24 However, it can sub-optimal choice, especially for the downstream tasks with large distribution differences. In contrast, fine-tuning the whole can yield better accuracy in most cases 1 [23] However, can easily lead to memory outages. On the other hand, recent parameter efficient transfer learning algorithms [26] 27 28 29] only require tuning less than 1% of the parameters. But this parameter efficiency only brings a mild memory efficiency. The most relevant prior work is TInyTL [1. it recognizes the main bottleneck of memory is in activation and introduces a new trainable module that only requires down-sample activation. However, it still focuses on exact gradient calculation while employ approximate gradient calculation, which enables compressing activation without changing the model architecture. A few other works have also explored the approximate activation techniques. Most of them focus on activation quantization Recently, activation compression is extended to ViTs 32 and Graph Neural Networks
the model 22
33
3 Method
3.1 Activation is the bottleneck of memory
The forward of neural networks can be express as following
fæ: 0 fi (ff-1 (...
:
(1)
= (xf 0),
3

---

where æ and y is the input and label, respectively. L is the loss between prediction f(æ; 0) and label y calculated by loss function L. fi and 0 denote the function and parameters of th layer, respectively. The network is composed of / layers. We further denote the output of i th layer (a.k.a activation) as 2; = fi 0z) and ~ = fi (æ; 01). When conducting back-propagation, aL and al are required be calculated for each activation. 30:
Taking linear layer with 2; 2:-1 W, + by (0, [W,, b,]) as an example, the gradients required to be calculated are MI:
OL OL 0zi+1 OL 0z, 0zi+1 0z.
OL -T OL OL OL
W
OW, 2; 0z,+1 ab. 0z,+1
(2)
0z
As shown in Equation 2 the activation 2, is employed for computing one of the gradients. This creates the need of saving the activation of the forward process. Otherwise, extra FLOPs are required for re-computing the activation. The size of activation is always larger than the weight as weight is always shared across different patches or tokens. Also, the size of activation can increase linearly with the batch size. Moreover, in the transformer, the length of input tokens can also lead to a quadratic increase of the memory.
Discussion about freezing weight. TinyTL [T] pointed out that
avoid saving activation
via freeziig the weight W; as other terms do not require the participation of z;- This is true for most architectures of convolutional neural networks. However, the emergence of VIT breaks this assumption. Many operations (e.g. self-attention, Softmax, GeLU (34) in transformer would involve the activation for computing ăz," which cannot be avoided if previous layers requires updating. Therefore we adopt the more general method: directly sparsifying the activation for backpropagation.
at.
3.2 Back- proppapopanation activation self-sparsification
In contrast to the activation sparsification 5 6 that prunes the activation of both forward and backward, our proposed Back Razor leverages the back-propagation activation sparsification: only prunes the activation that is used for backward while keeping the forward activation dense as shown in Figure The normal backward process can be formally defined
0L 0L 0zi+1 aT. (2) (z.,
aL aL. 0zit1
ØL
h1
n1 (z Oz)
(3)
θz. 0z.+1 dzi
0zi+1
00;
0zi41 00;
Ōz.+1
where (1-) (zi, 0+) = 0z:11 and (0) (z, 04) = 0z.11 representing the derivatives of z;41 with (1)
U2s
80;
respect to 2, and 0g, respectively. For Back Razor, the backward process can be expressed as
AL OL (=) (Z,, O.), 0z.
aL
(0) h (Z., 0,).
Ōz.+1 h
(4)
00;
dz,+1
where ă; denotes the pruned activation z,. We inherit the mathematical forms behind h and A, from the normal backward to preserve the gradient flow, but calculate the gradients on weights and activations with the pruned layerwise activation ž, With the above backward, there is no need for saving the dense actívation z;. Instead, we store a sparse version é; that is more memory efficient.
The detailed optimization algorithm with the proposed Back Razor can be found in Algorithm [ In the forward pass, would compute with dense activation while only saving its sparse version. In the backward, we employ the sparse activation for computing the gradient.
We use simple pruning method for backward activation pruning: prune the smallest magnitude activations. The smallest magnitude values are usually playing less significant roles than those with larger magnitude. This pruning method is also widely used for many pruning works (35). In practice, we would prune all the activations to a fixed sparsity of X by setting all the values under & th smallest value zero, where k An, n is the total number of values for the activation. As the ranking can be slow for large activations, instead of ranking the whole activation, we rank the activation of each sample and use the sample-wiss wise threshold for pruning. This largely shortens the ranking time. Also, we noted that the thresholds across different samples are very close, which means it is functionally similar global ranking.
We save the sparse matrix into two parts: a bitmap that indicates the position of the non-zero elements and a smaller dense matrix that contains the values of the non-zero elements. The bitmap has the

---

Algorithm Backward and Update with Sparse Activation
Sample æ; from Initialize Zo for i = 1 to do
> Forward pass.
2, (z 0.) 2,-1 - TopK Prunc(zi-)
Sparsify 2,-1 and store it.
end for
Calculate the training loss I = L(z1.
Initialize 81 <- 0zi
OL
for = 1 to do
> Backward pass.
Calculate Sz, 8, 10, 0:)
Calculate the gradient for the next activation
Calculate OL 80,
(9) 1 (ž,, 0,) and update 0
end for
same shape as the origin sparse tensor, but it much smaller since is bool type. The memory cost of this format is E AnCCype byte, where Ctyp is the count of bytes for the employed data type.
For the previous activation pruning method, the error that comes from pruning could gradually accumulate layer by layer in the forward, causing the deviation from the pre-trained model and leading to performance degradation. In contrast, the proposed Back Razor ensures the correctness of the forward. Moreover, in the backward progress, the pruning еттоr of many layers would not accumulate: the activations of those layers are only used for computing the gradient itself (e.g. linear layer, batch/layer normalization layer). It's also worth noting that the proposed Back Razor saves the same amount of memory as activation pruning under the same pruning ratio.
Comparison with recent parameter-efficient transfer methods. Recent studies reveal that fine- tuning can be parameter efficient. For example, difiPrune (26) points out that the fine-tuning performance can match fully fine-tuning via only modifying 0.5% of the weights. However, employs Gumbel-Softmax (36] make the masked parameters differentiable, which means the dense activations are required for updating the weight. Therefore, it cannot yield memory efficiency. On the other hand, other works also reveal that fine-tuning part of the model can yield competitive performance [27] 28 29 37). Though it can lead to memory saving, we empirically verified that Back Razor can yield higher saving ratio.
3.3 Theoretical Results
Now we study the convergence of Back Razor. We simplify the notion and define L to be the training loss of convolutional neural network that takes the model's weights the input. 0, (W,),- as the flattened vector of the model's parameters at step f, and g, be the gradient calculated with all the samples in the training dataset. Furthermore, gradient calculated with one sample from the training set. We make several classical assumptions [3]]:
denote the stochastic gradient by 9., which is the
Assumption 1. Let (0) be the gradient of the objective L al point O, then for all x and y assume there exists non-negative constant vector B that
1 -
(Cy) - [C(x) +
- II -(x))(-=x)(11)
3
Assumption 2. For each tteration f, the stochastic gradient g, satisfies the following conditions:
Elg.] 9t. El(g. 9.)31 < o3, Vj E [n],
where n is the number of parameters of models and o² is constant representing the variance for coordinate j.
Assumption 3. The objective C is bounded below by L.*
These assumptions discussed in Appendix. We first prove the convergence for linear networks.
Lemma 1. For linear neural networks. the gradient of parameters can decomposed by g, = A,A, and approximated gradient after activation pruning can decomposed by g' E,A, where 2, the exact activation and z', is the pruned activations.
5

---

In linear networks, öz. activation stored for backward does not change the its gradient. Therefore in Lemma [ A, is exclusively constructed by activations' gradients, and not changed after activation pruning.
OL only depends on the model's weights after the i th layer, so pruning the
We call A, the transformation matrix for activations at step t.
Theorem (Convergence). If Assumption 1-3 hold, (p+ ( a3)) Ka < 1 where is the pruning ratio and Kg is the squared condition number of the transformation matrix, then
2(C(On) L2) 19.1l - (1+ a.3) Ka
(p+a8) 0², I- aB)
EI
(5)
where the T is the number of iterations, 3 = |1 Oo is the initial weights and 0² ード= o3.
The Eqn. consists of two terms: the first term will converge to zero as the number of iterations T goes to infinity, The second term will not vanish, but its value be controlled by using larger batch size. Intuitively, proves that our algorithm reach to the neighborhood of a stationary point, and the radius of the neighborhood is bounded by the gradient variance.
The key factor for theoretical convergence is (p K aB can be reduced by using smaller learning rate. The theoretical convergence will not be guaranteed when the transformation matrix is ill-conditioned.
Remark.
We next discuss the convergence for non-linear networks, and focus on analyzing convolutional neural networks with common CONV-EN-ReLU structure [15]. The activation pruning happens after the convolutional layer, where the gradient on weights and activations can be calculated as [15]
aL 0z,
aL
ØL
aL
W.
0zi+1
00;
0z,+1 2,.
(6)
which suggests the gradient 2; will not be affected if we the pruned activation ž, to obtain the derivatives on 0 when warding through the convolutional layer. The activations are neither stored nor pruned for BN and ReLU operations, the Lemma [ networks. So under the same assumptions, we achieve similar convergence situation for CNNs. We present more details in Appendix.
holds for convolutional neural
We lastly discuss the Transformer structures. Unfortunately, the self-attention module does not meet the Lipchitz condition (39], therefore the above theoretical analysis will not apply out-of-the-box to them. However we empirically observe that Back Razor also applicable on ViTs [40] with decent performance (refer to Section 4.3) A more rigorous theoretical framework customized for transformers left as our future work.
Experiment
4.1 Settings
Following the common practice 26 41
we employ supervised pre-trained models on ImageNet
the starting point of transfer learning. Specifically, employ ImageNet- 1K and Imagenet-22K for CNNs and ViTs, respectively. For downstream fine-tuning, we consider eight datasets: Pets 43
Aireraft 444) CIFARIO, CIFAR 100 45). Flowers (46). Cars [7). CUB 48 and Food 44 The
default image resolution is 224 x 224 following
Our experiments are implemented with Pytorch [50] and conducted 1080 Ti V100 GPUs. To
measure the training memory, by default we report the theoretical memory usage following [[]. We also report the actual memory usage measured the GPU at Section 4.3
4.2 Back Razor with Convolutional Neural Networks
We choose ProxylessNAS-Motie
(52 as the CNN backbone following TinyTL [. worth
noting that we keep the origin setting of ProxylesNAA--Mone as in TinyTL given Back Razor is applicable for any architecture. We also follow the most training settings of TinyTL TL for fair comparison: The fine-tuning epochs and batch size is set as 50 epochs
instead of modifying the architecture

---

Table 1:
The comparison between Back Razor with the previous methods on different datasets with CNN. All the reported results are the topl accuracy (%). FT-Last denotes
the linear classifier (the last fully connected layer). FT-Norm+LLast denotes fine-tuning the batch normalization layers plus the linear classifier 37 S1 TinyTI. @ 320 denotes training and inference with a larger resolution of 320. The Back Razor @ 990% denotes the Back Razor with a sparsity ratio of 90%. The memory footprint of the training at a batch size of 8 (test in CIFARIO0) is also reported in the second column. Some experiments are conducted five times with different random seeds. The mean and
variance are reported.
Method
Train Memory
Pers
Aireraft
CIPARIO
CIFAR100
Flowers
Cars
CUB
Food
FI-Last FT-Norm-Last
31MB
91.3
44.9
85.9
68.8
90.1
50.9
73.3
68.7
192M
94,8
80.2
76.3
77.0
FT-Full
366MB
93.0100005
88.210021
97.1110.22
84.110.07
97.011..17
91.020006
80.9-00,3
83.820.10
Ti 0
37MB
91.8
75.4
95.9
$1.4
95.5
85.0
77.1
79.7
320
65MB
92.9
82.3
96.1
81,5
96.8
88.8
81.0
82.9
Razerdt00% (ours)
42MB
92,9++0.1
87.6:40.11
96.9+0.077
83.3-+0:05
96.9=00334
90.1-+0.10
81.0-0.24
83:2=0007
Back Rxzer055% (ours)
41MB
93.2
85.7
96.8
82.7
96.6
87.2
79.5
83.1
Back Razere97%% (nurs)
40MB
93.0
85.2
96.6
82.6
95.9
88.
78.8
82.3
Back Razer695%% (ours)
39MB
92.5
80.7
96.1
80.2
95.0
85.6
76.9
80.4
and 8, respectively. The model is optimized with adam [53] optimizer and cosine learning learning rate schedule [54]. The initial learning rate is tuned for each dataset. We freeze all the parameters of Batch Normalization layers (55) as they occupy more memory than convolutional layers while having much fewer parameters to update. By freezing them, we could free the occupied memory with less influence on the fine-tuning performance. We also use the origin activation for ReLU6, which is memory efficient (equal to the memory cost of a bitmap with the same shape). The Back Razor is then applied to convolutional layers and the fully-connected classification head.
Comparison Results: In T we compare Back Razor with previous transfer learning methods on CNNs. Specifically, it includes i) FT-Last: fine-tuning the last fully connected layer of the network, which is also referred as linear classification head. ii) FT-Norm+Last fine-tuning the batch normalization layers plus the classification head iii) FT-Full: fine-tuning the full model. Also, we include two different variants of the previous state-of-the-ar (SOTA) method I: TinyTI. and TinyTL @ 320 (TinyTL @ 320 employs a larger resolution image of 320).
Compared with FT-Full, Back Razor@ use 8.7x times smaller memory while achieving com- parable performance. The variance for Back @ 90% with respect to different random seeds is also comparable with FT-Full and they are both small. In contrast, FT-Norm+ takes 4.6x time more memory while being [0. 10.% 19.5%, 2. 1%, 3.1 3.1% 2.6%. 12.2% 4.7%, 6.2%] worse than Back Razor @ 90% in [Pets, Aireraft, CI,AR1O, CIFAR 100, Flowers, Cars, CUB, Food], respectively. Though FT-Last does yield slightly less training memory (31MB v.s. 42MB), the performance largely degrades: the accuracy is lower by [1.6%, 42.7%. 11.0%, 14.5% 6.8%, 39.2%, 7.7%, 14.5%) than Back Razor @ 90% in [Pets, Aircraft, CIFAR1O, CIFAR 100, Flowers, Cars, CUB, Food], respectively.
Compared with the previous Statc-of-The-AA method TL. Back Razor 90% is comparable in terms of the training memory usage (37MB v.s 42MB). Mcanwhile, it yields higher accuracy of [1.1%, 12.2%, 1.0%, 1.9%, 1.4%, 5.1 %, 3.9%, 3.5%] at [Pets, Aircraft, CIFAR10, 10, CIFAR 100, Flowers, Cars, CUB, Food], respectively. When compared with TinyTL @ 320, which employs a larger resolution, Back Razor@ 90% saves 1.5 times of the memory while still yielding an improvement of [0.0%, 5.3%, 0.8%, 1.8%, 0.1%, 1.3%, 0.0%, 0.3%] [Pets, Aircraft, CIFARIO, CIFARIO0, Flowers, Cars, CUB, Food], respectively.
It's also worth noting that Back Razor method can even achieve more aggressive sparsity. For instance, the performance of Back Razor is [92.9%, 93.2%, 93.0%. 92.5%] for pruning ratio of [90%, 95%. 97%, 99%] at Pets, respectively. The performance maintains the same level even with the pruning ratio of 97% and only marginally drops at the pruning ratio of 99%. A similar trend is also observed in other datasets. However, the high pruning ratio does not bring further memory reduction since the main bottleneck of memory is no longer the accumulated activations. Instead, it is the model parameters (of 11 11.6MB) and one large forward activation (of 24.5MB).
Can we prune at forward pass? We further compare the performance difference between activation pruning (prune both forward and backward) with Back Razor (only prune backward) in Figure 3 We implement the activation pruning via directly applying the mask of the Back Razor to the forward pass. It can be observed that the model can casily collapse (the model can hardly converge and only
7

---

90
140
95
140
Prune Both Forward & Backward
Prune Both Forward & Backward
85
Only Prune
120
90
Only
120
a Memory
Memory Usgge
Accuracy
80
100
Accuracy
85
100
75
80
70
60
60
10
10
5
5
0
0
0.5
0.6
0.7
0.8
0.9
1
0.5
0.6
0.7
0.3
0.9
1
Pruning Ratio
Pruning Ratio
(a) Aircraft
44
(b) Pet
43
Figure 3: Comparison between pruning both forward and backward with only pruning backward under different pruning ratios on (a) Aircraft. (b) Pet. For both figures, the y axis in the left is the top 1 accuracy (%) while the axis in the right is the memory usage, and x-axis is the pruning ratio. The line is coupled with y axis that is with the same color. The yellow line in both figures denotes the memory usage in different pruning ratios. Note that methods share the line their memory consumbtion is the same under the same pruning ratio.
have accuracy below 5%) for activation pruning even with mild pruning ratio of 70%. In contrast, for Back Razor, the performance can be at a high level even with an aggressive pruning ratio of 99% ( the performance is even higher than activation prune at sparsity of 50%).. This demonstrates that the backward activation can more sparse than that the forward.
Convergence speed in practice: As shown in Figure the convergence speed in transfer experiments comparable with each other. In the early stage of the training, the convergence speed of Back Razor is even faster. Even the last stage, FT-Full is only marginally better than Back Razor.
Comparison with SWAT: We further compare with SWAT [17] which leverages the similar insight the sparsity of backward activation. Although [17] was originally developed for training from scratch, here apply it to the identical fair setting of transfer learning. As shown in Table 2 while similar memory footprint is employed for Back Razor @ 90% and SWAT@ 80%, the proposed BackRazor can yield higher performance by 0.6% and 2.6% CIFAR10 and CIFAR 100, respectively.
4.3 Back Razor with Vision Transformer
In this section, we study Back Razor on ViT [40]. The patch size by default set as 16. We employ the standard SGD optimizer with cosine learning rate decay for finetuning. The training steps are fixed as 20k and the initial learning rate tuned for each dataset. We employ a larger batch size of 128 following the common practice for accelerating training 40 56 32).
Back Razora FT-Full
Back Razora 0.9
FT-Full
Training Loss
Training Loss
0.7
1.5
0.6
Epochs
Epochs
(a) CIFAR10 45
(b) 100 45
Figure 4: The training loss convergence speed comparison between Back Razor@0:, and FT-Full. The convergence speed of them is comparable.

---

Table Comparison between BackRazor and SWAT [77 on CIFARIO and CIFARI00 with Resnet-
18. The memory footprint of training at a batch size of 128 is reported in the second column. We reproduce SWAT with the official code under fine-tuning setring for fair comparison.
Method
Train Memory
CIFAR10
CIFAR100
FT-Full
164.4MB
96.5
82.0
SWAT@50%
42.7MB
95.8
79.6
SWAT @90%
31.7MB
95.3
77.7
Back Razor @ 90% (ours)
41.7MB
96.4
82.2
Table The comparison between Back Razor with the previous methods on different datasets with VIT-B/16. All the reported results are the topl accuracy (%). The memory footprint of the training at a batch size of 128 (compute in CIFARI00) is reported in the second column. Back Razor @ 80% +
Mesa denotes combining the BackRazor with Mesa.
Method
Train Mensor
Pets
Aircraft
CIFARIO
CIPARI00
Flowers
Cars
CUB
Food
FT-Last
525MB
91.6
44.4
96.8
86.5
99.3
57.4
85.6
86.3
FT-Flll
19235MB
94.1
78.8
99.0
93.1
99.5
85.5
85.5
90.3
Ball
9460MB
94.0
74.2
98.8
92.3
99.5
80.6
85.7
89.7
5442MB
77.2
99.4
(ouns)
77.3
Razord9905 (ours)
3912MB
75.7
09.4
Back Ra2 (ours)
3496MB
02.4
74.3
98.8
90.0
99.4
81.5
86.1
88.7
Back Racor 80% + Mesu (ouns)
2937MB
93.5
76.9
99.0
92.7
99.4
81.1
86.5
90.5
Comparison Results: In Table compare Back Razor with previous transfer learning methods on VIT, including FT-Last and FT-Full same as defined in Section 4.3 We further compare two
parameter efficient transfer learning method from the literature: Bitfit (77. and the SOTA ViT transfer
leanring method, Mesa 27 that is based on backpropagation quantization.
Compared with FT-full, Back Razor@80%) 80% yields comparable performance across the datasets while achieving 3.9x memory saving. Though FT-Last employs smaller train memory than Back Razor, the accuracy drops by [2.2%, 32.9%, 2.1%, 6.4%, 0. 1.1% 26.6%, 0.9%. 4.2%] in [Pets, Aircraft, CIFARIO 10, CIFAR 100, Flow- ers, Cars, CUB, Food], respectively, compared to Back Razor 80%.
Table 4: Comparison of the on-device memory usage on GPU under the batch size of 1 28 (tested in CIFARI00). We do not report Bitfit here cannot yield hardware memory efficiency on GPU with Pytorch.
Method
On-device Memory
FT-Last
1449MB
Bitfit achieves competitive performance via tun- ing less than 0.1% 1% of the parameters. However, can only reduce the memory saving 2.0x memory given it requires saving some large ac- tivations full (e.g. attention map). In contrast, Back Razor can reduce the size of any activa- tions. While Mesa has similar memory usage and performance compared with Back Razor, we note that its methodology is orthogonal with Back Razor. By combining Meta and Back Razor together, can achieve 7.5x memory saving with comparable performance to FT-Full.
FT-Full
22631MB
Mesa 32
10425MB
Back Razor @ 80%
8501MB
Back Razor 90%
7604MB
Back Razor @ 95%
7189MB
Back Razor 80% Mesa
6640MB
When employing larger pruning ratio for BackRazor in ViTs, more memory footprint savings can be achieved little performance degradation. Especially, for CIFARIO 10, the performance of BackRazor only drops by 0.2% compared to FT-Full at pruning ratio of 95%.
4.4 Back Razor with More Tasks
In this section, we further exploring applying the proposed BackRazor to more downstream tasks.

---

Table BackRazor for Pascal VOC segmentation task with DeepLabV3-MobileNe [57 58 memory footprint of the training at a batch size of 8 is reported in the second row.
The
Pruning ratio
0% (FT-Full)
10%
20%
50%
70%.
90%
95%
97%
Mem)ry(MBB
8864
8295
7467
4983
3327
1671
1257
1092
Accuracy(%)
70.9
71.0
70.9
70.6
70.5
70.3
69.4
68.5
Table BackRazor for RTE from GLUE Benchmark [59 with BERT base on-device memory footprints of training at a batch size of 8 are reported.
The theoretical and
Method
Theoretical Memory (MB)
Actual GPU Memory (MB)
Accuracy
FT-Full
4174
7002
70.4
Back 990%
2187
4858
69.7
Semantic Segmentation: As shown in Table 5 when applying semantic segmentation tasks, BackRazor can yield 5.3x memory efficiency at a pruning ratio of 90% with only a marginal accuracy drop of 0.6% compared to the fully fine-tuning baseline.
Language modeling: As demonstrated in Table when applying Back Razor on the language model, Back Razor achieve significant memory efficiency improvement pruning ratio of 90% with only a marginal accuracy drop of 0.6%.
5 Conclusion
In this work, propose Back Razor, memory-efficient transfer learning method, which is also the first asymmetric sparse transfer method that only sparsifies the back-propagation activations while keeping the forward. Through extensive experiments in both Convolutional Neural Networks and Vision Transformer across multiple datasets, we demonstrated the proposed Back Razor can achieve state-of-the-arr memory efficiency with an aggressive pruning ratio. Moreover, it can be combined with the propagation quantization method and yield better memory efficiency. As limitations, Back Razor has only been tested on limited range of tasks, and its theoretical underpinning is so far restricted to CNN backbones despite its empirical success in ViTs, calling for continued efforts.
Acknowledgment
Z. Wang is in part supported by the National Science Foundation under Grant IIS-2212176, and Google TensorFlow Model Garden Award.
References
[1] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce activations, not trainable parameters for efficient on-device learning. arXiv preprint 1622 2020.
[2] Ayan Chakrabarti and Benjamin Moseley, Backprop with approximate activations memory-efficient network training. Advances in Neural Information Processing Systems, 32, 2019.
[3] Jianfei Chen, I Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael Mahoney, and Joseph Gonzalez. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning, pages 1803-1813. PMLR, 2021.
[4] R David Evans, Lufci Liu, and Tor M Aamodt, Jpeg-act: accelerating deep learning via transform based lossy compression. In 2020 ACM/IEEE 47th Annual International Symposium Computer Architecture (ISCA), pages 860-873. IEEE, 2020.
[5] Arash Ardakani, Carlo Condo, and Warren J. Gross. Activation pruning of deep convolutional neural networks. In 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 1325-13295 2017.
10

---

[6] Qing Yang, Wei Wen, Zuoguan Wang, Yiran Chen, and Hai Li. Integral pruning on activations and weights for efficient neural networks, 2019.
[7] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint
2015.
[8] Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William Dally, Exploring the regularity of sparse structure in convolutional neural networks, arXiv preprint arXiv: 1705.0822,,,, 2017.
[9] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[10] Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXiv preprint Xrv: 100.::1:00000000 75021, 2017.
[11] Haobo Sun. Yingxia Shao, Jiawei Jiang, Bin Cui, Kai Lei, Yu Xu, and Jiang Wang. Sparse gradient com- pression for distributed sgd. In International Conference on Database Systems for Advanced Applications. pages 139-155. Springer, 2019.
[12] Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. International Conference on Machine Learning. pages 3299-3308. PMLR, 2017.
[13] Bingzhen Wei, Xu Sun, Xuancheng Ren, and Jingjing Xu. Minimal effort back propagation for convolu- tional neural networks, arXiv preprint arXiv:1709.05804 2017.
[14] Zhiyuan Zhang, Pengeheng Yang, Xuancheng Ren, Qi Su, and Xu Sun. Memorized sparse backpropagation. Neurocomputing, 415:397-407. 2020.
[15] Xucheng Ye, Pengcheng Dai, Junyu Luo, Xin Guo, Yingjie Qi, Jianlei Yang, and Yiran Chen. Accelerating cnn training by pruning activation gradients. In European Conference on Computer Vision, pages 322-338. Springer, 2020.
[16] Siavash Golkar and Kyle Cranmer. arXiv:1806.013:3 2018.
Backdrop:
Stochastic backpropagation. arXiv preprint
[17] Md Aamir Raihan and Tor Aamodt. Sparse weight activation training. Advances in Neural Information Processing Systems, 33:156 16625-16688, 2020.
[18] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
[19] Jacob Devlin. Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.0448 04805, 2018.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729 9738, 2020.
[21] Kaiming Hc, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 4918-4927, 2019.
[22] Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint 405.333,
[23] Chuang Gan. Naiyan Wang, Yi Yang, Dit- Yeung. and Alex G Hauptmann. Devnet: A deep event network for multimedia event detection and evidence recounting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2568-2577 2577, 2015.
[24] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf. an astounding baseline for recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition workshops, pages 806-813, 2014.
[25] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. arXiv preprint 22:.:22200999 2022.
[26] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff' pruning. arXiv preprint arXiv:2012.07663, 2020.

---

[27] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameler-efficiet fine-tuning for transformer-based masked language-models. arXiv preprini w7Xi::2166.0019, 2021.
[28] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank
hypercomplex adapter layers, arXiv preprint arXiv: 100.04477, 2021.
[29] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint 107 13586, 2021.
[30] Fangeheng Fu, Yuzheng Hu, Yihan He, Jiawei Jiang, Yingxia Shao, Ce Zhang, and Bin Cui. Don't ton't waste your bits! Squeeze activations and gradients for deep neural networks via TinyScript. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 199 of Proceedings of Machine Learning Research, pages 3304-3314, PMLR, 13-18 Jul 2020.
[3]] R David Evans and Tor Aamodt. Ac-ge: Lossy activation compression with guaranteed convergence. Advances in Neural Information Processing Systems, 34, 2021.
[32] Zizheng Pan, Peng Chen, Haoyu He, Jing Liu, Jianfei Cai, and Bohan Zhuang. Mesa: A memory-saving training framework for transformers. arXiv preprint arXiv:2111.11224, 2021.
[33] Zirui Liu, Kaixiong Zhou, Fan Yang, Li Li, Rui Chen, and Xia Hu. Exact: Scalable graph neural networks training via extreme activation compression. In International Conference on Learning Representations, 2021.
[34] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv: 1ee..10066880 08415, 2016.
[35] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03660 2018.
[36] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzaion with gumbel-softmax. arXiv preprint 16Xi::661..000 1.0114444 2016.
[37] Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew Howard. K. for the price of I: Parameter efficient multi-task and transfer learning. In International Conference on Learning Representations, 2019.
[38] Léon Bottou et al. Online learning and stochastic approximations. On-line learning in neural networks, 17(9):142, 1998.
[39] Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562-5571. PMLR, 2021.
[40] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2020.
[41] Simon Korblithh Jonathon Shlens, and Quoe V Lc. Do better imagenet models transfer better? In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2661-2671, 2019.
[42] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categoriza- tion and domain-specific transfer learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4 4109-4118 18, 2018.
[43] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 3498-331 3505. IEBE, 2012.
[44] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-graines visual classification of aireraft, arXiv preprint urXiv:1306.5151. 2013.
[45] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[46] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
12

---

[47] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[48] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[49] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Compuder Vision, 2014.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Autiga, et al. Pytorch: An imperative style, high-perrfrmance deep learning library. Advances in neural information processing systems, 32, 2019.
[51] Jonathan Franklc, David J Schwab, and Ari S Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. arXiv preprint arXiv:2003.00152, 2020.
[52] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 2018.
[53] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.60 2014.
[54] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint 1:08vv6608.03900 2016.
[55] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[56] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[57] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706. 05587, 2017.
[58] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilencts: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704,0484111111 2017.
[59] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint mrXiv:180.00746/ 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasig new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] The paper itself comes with a license.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A] We do not curate/relcase new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "_EtR1RuPIz2", "reviewer": "Reviewer_RTcC", "summary": "To reduce memory use of training a transfer learning model, the paper proposes to prune activations stored for back-propagation, instead of traditionally pruning activations stored for both forward-propagation and back-propagation. Experiments on CNN and ViT show that the proposed method achieves comparable accuracy with more aggressive memory reduction compared to competitive methods TinyTL, Bitfit, and Mesa.", "questions": "1) Is it possible to provide peak memory use during training?\n2) Have you tried structural sparsification strategies?\n3) Is it possible to qualitatively or quantitatively compared with the traditional method gradient checkpointing in the experiments?\n", "limitations": "As I mentioned above, a major limitation is the adopted unstructural sparsification strategy which incurs discrepancy between theoretical memory use and practical memory use.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n1) In Section 1 and Section 3.1, the paper provides clear explanations on why the paper focuses on compressing activations stored for back-propagation. \n2) In Section 3.2, the proposed techniques are clearly written and reasonable. Theoretical analysis on convergence is also provided in Section 3.3.\n3) Experiments on CNN and ViT are convincing. For ViT, the paper provides comparison between theoretical memory use and practical memory use in Table 2 and Table 3, respectively. \n\nWeaknesses.\n1) I think the authors should provide peak memory use in the experiments. Since the method prunes activations stored for back-propagation, I agree with the authors that the theoretical memory use and averaged memory use are both decreased. However, as the activations are not sparsified during forward-propagation, I am not sure whether peak memory use is decreased compared to previous methods. If the peak memory use is not decreased, the proposed method actually requires computing devices with the same memory capacity as previous methods.\n2) A potential problem of the method is discrepancy between theoretical memory use and practical memory use, as shown in Table 2 and Table 3. The problem is incurred by the adopted unstructural sparsification strategy, pruning the smallest magnitude activations, as introduced in Section 3.2. According to memory working mechanism, memory is accessed in column-wise or row-wise manner. A column/row is accessed even if there is only one non-zero element. Therefore, unstructural sparsification cannot effectively practical memory use.\n3) I think the traditional method gradient checkpointing is very related to this paper. Is it possible to qualitatively or quantitatively compared with this method in the experiments?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ECi-8e8JXdW", "reviewer": "Reviewer_agR8", "summary": "* This paper considers the scenario that one want to not only deploy pre-trained models but also continue train/finetune these models on local devices. It proposes a novel method called Black Razor that reduces the training/finetuning memory of neural networks, by pruning/compressing the activation during back-propagation. BlackRazor is able to achieve 96% sparsity, saving 9.2x memory without losing accuracy. The paper also conducts a theoretical analysis of BlackRazor's convergence. ", "questions": "* Typo in Figure 2: “Storeage”\n* What is the memory difference between activations in forward pass and backward pass? How can pruning activations in backpropagation reduce memory by more than 2x?\n* Have you tried on language transformers?", "limitations": "N'/'A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The proposed method seems to have great results: successfully reducing much training memory without much loss of accuracy.\n* The proposed method seems to be simple and generalizable. \n\nWeakness:\n* One weakness of this paper is the novelty of the proposed method. The method is to conduct magnitude pruning on activations for back-propagation.  This seems to be more like an engineering trick rather than a science / learning paper.\n* This paper will be more solid if it’s validated on more downstream tasks. \n* The theoretical results are a bit confusing: for example, in algorithm 1, it prunes activations during forward pass, yet in Lemma 1, it states that “so pruning the activation during backward does not change its gradient”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bhr3wSg9HS", "reviewer": "Reviewer_3voy", "summary": "Back Razor is a memory-efficient algorithm for transfer learning on edge devices, which works by using sparse activations during the backward pass of neural network training. The paper supports its claims with experimental validation on four image classification datasets and two architectures (CNN and ViT), as well as some theoretical convergence results.\n", "questions": "The main suggestion to the authors is to conduct a substantially more thorough experimental validation of the method.", "limitations": "The authors correctly pointed out that the paper only looks at image classification tasks, and also addressed the limitations of their theory’s application to ViTs. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe proposed method is elegant, simple to understand, and easy to implement. If the early experimental results hold up, this method can substantially help with neural network training on edge devices. Experimental evaluation looks quite promising.\n\nThe placement of the algorithm in the context of transfer learning on edge devices is appropriate and compelling. Overall, the paper flows well.\n\n## Weaknesses\n\n### [major issue] Experiments [This was addressed to my satisfaction during the discussion period]\nThe authors only show experiments for vision tasks using two architectures (a CNN optimized for mobile performance and ViT). Besides the basic baselines of full finetuning / FC finetuning / FC+Norm, the paper only benchmarks on one other method in the case of CNNs and two methods in the case of ViTs; it is not clear from the paper that these methods are SOTA for each of the tasks considered. In almost all cases, the experiments are presumably run only once, as generally no confidence intervals were given.\n\nThe biggest concern is that the experimental coverage is insufficient to draw strong conclusions. In particular, TinyTL, which is used as a benchmark, provides results for nine datasets of which BackRazor is only tested on four for CNN, and three for ViT. There is also no investigation of how well the method performs with different batch sizes and optimizers, or for other categories of tasks, such as segmentation or language understanding. In the case of CNNs, it would also be useful to see a comparison to other methods that use reduced precision (i.e. quantization); for ViT this was provided.\n\nFinally, it would be useful to see the actual on-device memory usage in the case of the CNN architecture.\n\n### [minor issue] Theory [This was addressed to my satisfaction during the discussion period]\nThe theory gives a convergence result for a linear network, and then some argumentation is made to extend this result to CNNs. However, Theorem 1 states that the algorithm converges to a neighborhood of a stationary point, and the argument that this neighborhood can be reduced by increasing the batch size does not work in this memory-constrained setting. Thus, it is not clear how well the theory applies, even though it is also experimentally verified that the convergence rates are similar, hence this is overall a minor issue.\n\n### [moderate issue] Writing [This was addressed to my satisfaction during the discussion period]\nWhile the flow of the paper is good, important details are often not made as clear as they could be, or are hard to find. For example, the TopK operator used to prune the activations should be clearly indicated in the algorithm. The theoretical assumptions, and the grounds for them are not explained, leaving the reader to consult another paper for context. Finally, the transition in line 129 of moving from an arbitrary activation $z_i$ to the final activation $z_l$ is somewhat confusing, since none of the rest of the paper focuses on the last layer specifically.\n\nSome sentences make overly broad claims, for instance the very first statement of the abstract calls transfer learning the unqualified _de facto_ choice, which is not generally true. Likewise, in line 101, the claim “transfer learning always happens on personal computing devices” is unsubstantiated.\n\nThe proof in the appendix contains numerous typos in the math notation, for instance:\nIn line 4, $ \\tilde{z}’_{k} = \\text{diag}(m)(\\tilde{z}$ should have been  $ \\tilde{z}’_{k} = \\text{diag}(m)(\\tilde{z}_{k}$ \t \nIn line 6, $\\tilde{g}_k$ should have been $\\tilde{g}_k$\nIn the last formula of line 5, $\\beta_i$ should have been the boldface $\\beta$. Also, some of the gradients in that equation are incorrectly marked as the full gradient $g$ rather than $\\tilde{g}$.\nThe $\\kappa$s in line 13 should have been $K_a$.\t\n\nOther typos are likewise numerous, although they generally do not adversely affect understanding:\n\nLine 40 of -> or\nLine 99 - missing “the” before “de facto”\nLine 102 constrain -> constraints\nLine 124 by -> of\nLine 130 FLOPs should be capitalized\nLine 157 actions -> activations\nLine 166 value -> values\nLine 179 weight -> weights\nLine 201 Assumption -> assumptions\nLine 231 the word “based” shouldn’t be there\nLine 267 Missing verb after “even”\nLine 288 employ -> employs\nLine 313 illustrate -> illustrates\nLine 329 backpropogation -> backpropagation\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "26d9oNy3aQY", "reviewer": "Reviewer_i5p7", "summary": "The paper proposes a simple and memory-efficient method called Back Razor that prunes activations only for back-propagation, presents a theoretical analysis that the convergence rate of Back Razor can be similar to that of SGD, and shows the effectiveness of Back Razor on CNN and Vision Transformer.", "questions": "In Figure 3, it would be better if the memory footprint of training as well as accuracy is plotted when comparing pruning both forward and backward with pruning only backward.", "limitations": "As mentioned in the paper, the authors plan to apply Back Razor to several downstream tasks.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n(1) The paper is well structured and the main idea is clearly explained.\n\n(2) The extensive experiments demonstrate the efficacy of Back Razor to some extent.\n\nWeaknesses\n\n(1) Proof-reading is required due to some errata and awkward expressions.\n\n(2) More experiments on large models such as ViT-L/16 seem to be needed to verify whether Back Razor is really working well on large pre-trained models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept