Activation Map Compression through Tensor Decomposition for Deep Learning
Authors: Le-Trung Nguyen
Aël Quélennec
Enzo Tartaglione
Samuel Tardieu
Van-Tam Nguyen
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Learning
Computer Vision
Compression
Primary area: optimization_for_deep_networks
TL;DR: In this paper we demonstrate the relevance of applying tensor decomposition methods to compress activation maps and allow on-device learning.

Abstract: Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with a massive call for their unification into a common framework called Edge AI. While on-device inference is a well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive computational and memory costs compared to the extreme resource constraints of embedded devices. Drawing on tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in considerable memory savings while preserving the features essential for learning, and also offers theoretical guarantees to convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-art solutions, in terms of the trade-off between generalization and memory footprint.

Activation Map Compression through Tensor Decomposition for Deep Learning
Le-Trung Nguyen
AEl Quélennec
Enzo Tartaglione
Samuel Tardieu
Van-Tam Nguyen
LTCI, Télécom Paris, Institut Polytechnique de Paris {name.
Abstract
Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with massive call for their unification into a common frame- work called Edge AI. While on-device inference is well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive com- putational and memory costs compared to the extreme resource constraints of embedded devices. Drawing tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in consid- erable memory savings while preserving the features essential for learning, and also offers theoretical guarantees convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-an solutions, in terms of the trade-off between generalization and memory footprint
Introduction
Recent advances in Deep Learning have enabled Deep Neural networks to be used as an efficient solution for a wide variety of
8C
Auc
Mamey
A
0A
cases, including computer vision [21 39 30). speech recog-
nition 18, 31 and natural language processing [45] 42). Much of this performance improvement is linked to the exponential increase in the number of parameters the neural architectures. According Sevilla et al., the release of AlphaGoo [47] in late 2015 marks the advent of a new era, which they call the "Large Scale Era", in reference to the computational cost of training doubling every 8 to 17 months [38]. At the root of exponential growth in neural network size is the improvement in hardware capabilities, particularly those designed for large-scale parallel computing, such GPUs and TPUs [I]. While this trend demon- strates the strength of neural networks as a powerful generaliza- tion tool in many fields, it goes the opposite direction when it comes to environmental concerns [41). making the deployment of newer architectures increasingly difficult. This is particularly true for edge devices such as mobile phones and embedded sys-
A.
W;
Merters
ac
A
a.A.
A
Fervad
Beckward
Can
Difkazet
We compress the activa- tions that will be later employed for backpropagation.
Figure
CCode:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tems, which cannot afford the high computing memory costs. To address these challenges, three interconnected factors must be taken into account: power consumption, memory usage, and latency.
When considering larger neural networks with more layers and nodes, reducing their storage and computational cost becomes essential, especially for certain real-time applications such edge computing. In addition, recent years have seen significant advances in the fields of intelligent edge and embedded intelligence, creating unprecedented opportunities for researchers to address the fundamental challenges of deploying deep learning systems on edge devices with limited resources (e.g. memory, CPU, power, bandwidth). Efficient deep learning methods can have significant impact on distributed systems and embedded devices for artificial intelligence. Since training is supposed to take place in the cloud, most research on model compression and acceleration specifically focused on inference 77]. There is. however, an emerging area of research concerning on- on-device training, which represents a decisive advance in the field of artificial intelligence, with considerable implications for a variety of practical situations [17]. Models trained offline on dataset built at one point in time tend to fall victim to data drift when deployed "in the wild" [36]. Its combination with online learning strategies has the potential to enable continuous model improvement after deployment [14), thus adapting the model predictions to observed evolutions in the data distribution. We can illustrate this with the example of sensors in autonomous vehicles, where deep learning models must correctly classify vehicles with new designs never seen in the training set. Other advantages of on-device learning include security and privacy. By processing data locally, sensitive information remains more secure and less susceptible to data breach, a major concern in applications such healthcare.
The main challenge limiting the feasibility of on-device learning lies in the computational demands of the backward pass, as gradient computation and parameter updates are significantly more resource- intensive than the forward pass (Appendix [A.I). On embedded devices, memory and computation limitations act strict budgets that must not be exceeded. Some approaches address the memory constraints by exploring alternatives to traditional backpropagation, including unsupervised learning for image segmentation [51], the Forward- -orrardd algorithm [16), and PEPITA [34] While promis- ing, these methods typically fall short of backpropagation based techniques in terms of performance. A pioneering effort by Lin et al., demonstrated that fine-tur mreettning deep neural network within 256 kB of memory is fcasible by selectively updating a sub-network. achieving a good results [25]. In a complementary approach, Yang et al. proposed reducing the number of unique elements in the gradient map through patch-based compression of the input and gradients of a given layer with respect to the output, thereby lowering memory costs and speeding up the learning process (53).
Inspired by tensor decomposition methods, we propose a method that compresses activation maps, reducing the memory demands for backpropagation while maintaining the deep neural network's s generalization capability(Fig. [ Our approach adaptively captures the majority of tensor variance, offering guaranteed accuracy in the gradient estimation. The key contributions of our work are as follows:
We propose to exploit powerful low-rank approximation algorithms to compress activation maps, enabling efficient on-device learning with controlled information loss (Sec. 3.2 and
Sec. 3.3).
We provide a theoretical foundation for our method, along with an error analysis demon- strating that high compression ratios are achievable with limited performance degradation
(Sec.
We extensively explore diverse experimental landscape, demonstrating the generalization capacity of our proposed algorithm (Sec.
2 Related Works
Tensor Decomposition. Model compression and acceleration is a well-developed and ongoing area of research in the deep learning community. It can be divided into two major areas, namely hardware and algorithm design. In the case of algorithmic compression, the five main components that stand out are efficient and compact model design, data quantization, network sparsification, knowledge distillation, and tensor decomposition 7. All these approaches aim to reduce the space occupied by network parameters. Also known as low-rank approximation, tensor decomposition has emerged robust solution due to its combination of grounded theoretical findings and its

---

practicality in terms of hardware implementation [49]. Originating in the field of systems theory
and signal processing (28). low-rank approximation has attracted growing interest in the deep
learning community. Early examples were limited to the application of singular value decomposition (SVD) used to compress fully connected layers [50]. More recent work includes the application of
generalized Kronecker product decomposition (GKPD) to both linear and convolutional layers [13].
the introduction of semi-tensor product (STP) to improve compression ratios with reduced loss [56
or even more recently the acceleration of inference and backpropagation in vision transformers [52]
Evolutions in this domain can be summed up as improvements in compression ratios, latency, and power consumption, with limited performance loss and diversification of architectures considered.
Activation Map Compression. The term "activations" here refers to the outputs of each layer after the non-linearity has been applied, which are then fed to the next layer. While model compression generally aims at reducing the storage space occupied by parameters (which typically translates into network acceleration during inference), key observation is that activations occupy much more space than parameters in memory during backward pass, as they are required to compute weight derivatives [2]. This state of facts motivates new line of research, whose main objective is to compress activation maps using methods inspired by the literature on weight compression. For example, we mention the use of quantization [9], sparsification
22,, a combination of both with the addition of entropy encoding [12,, or even the application of wavelet transform in combination with quantization [10]. With the exception of Eliassen et al. 's work which accelerates training runtime, most of these works focus on accelerating inference, in a similar way to traditional model compression.
On-device Learning. Today's typical pipeline for on-device AI consists of designing resource- efficient deep neural networks [24,, training them offline on target data, compressing them, and deploying them for inference on the resource-constrained environment. Alongside the rapid commer- cial expansion of the IoT field and the ubiquitous presence of embedded devices in our daily lives, Al at the edge has naturally attracted a great deal of interest. However, recent research has shown that real-world data often deviated from training data, resulting in poor predictive performance [40]. In such a context, continual learning proved to be a strong candidate for ensuring ongoing adaptation after deployment. Similar to the human learning process, it provides models with the ability to adapt to new incoming tasks, ideally without falling into the trap of catastrophic forgetting (i.e.,
degrading performance older tasks in favor of new ones) [11 33 [9] In this respect, the very low memory and computing resources of extreme edge devices are an obstacle to the high cost of backpropagation. Lin et al. have shown that this challenge can be overcome by selectively fine-tuning a sub-graph of pre-trained networks on general dataset (25). Their work started a line of research showing the possibility of improvement through careful selection of channels to be updated at training time [23 35 However, none of these works addresses the computational cost of training neuron, which is related to both the cost of loading the necessary weights and activations into RAM.
Gradient Filtering. In their work, Yang et al. demonstrate that it is possible to significantly reduce the memory and computational cost of full layer learning using method called gradient filtering, which reduces the number of unique elements in the gradient map [53] In such a frame and with the same specific objective, we propose compress the activation maps by using tensor decomposition to minimize the memory cost. We perform this while providing a strong guarantee on the signal loss: unlike gradient filtering, we adaptively size the decomposed tensor to guarantee minimal information loss. In the next section, we will illustrate our proposed approach.
3 Method
In this section, we motivate our compression proposal by exposing the major memory bottleneck of backpropagation (Sec. (.T].. We then present our contribution which is the compression of activation maps through two well-known low-rank decomposition strategies, namely Singular Values Decomposition (SVD) and its tensor counterpart Higher Order SVD (HOSVD) (Sec. In Sec. 333 we explore the induced changes to backpropagation calculation and in Sec. 3.4 memory and computational complexity, alongside an evaluation of the error introduced. Our ultimate goal here is to reduce the memory footprint of backpropagation (BP).
Sec.3.4 we study the resulting
3

---

3.1 The Memory Bottleneck of Backpropagation
Following the formalism introduced in 2 we consider simple convolutional neural network (CNN)
represented as sequence of n convolutional layers (excluding the bias for simplicity):
F(X)= (Cw,. CW.-,
o Cw3, Cw,, (xX),
(1)
where W; E RC'xCxDDD represents the filter parameters of the 7th layer, with kernel size of Dx D. This layer receives a C-channel input and produces an output with C' channels. For this layer, we denote the input and output activation tensors as A, E BxCxH** W and A1+1 E BRC' xH'xW" respectively. Note that and W denote the height and width of each element of the input, while H' and W denote the height and width of each element of the output, with B as the minibatch size. Oc The loss C is computed at the output of the network and backpropagated until the 7hh layer as 8Aai1 At this stage, the filter parameters are updated thanks to the computation of ac and the loss is aW;
propagated to the previous layer as aA:"
ac The computation of these terms follows the chain rule:
0C Oc
Oc 0Ai+1 conv Aii 0Ai+1
Ow; 0Ai+1 OW;
Oc
Dc
0A
ac
[aA4+1' rot (WW))
convfull
0A: 8Ait1 0A:
where conv(-) is the traditional convolution operation, convolving the kernel
with the input
3A.11 Oc
A;; while convian(.) is the convolutional operation that naturally maps the input TA.41 with the same dimensions as A; by using the 180 rotated kernel W;-
to an output
From (2). it is clear that computing the weight derivatives requires to load input activation A, and (3) shows that the weights W; must be loaded into memory to calculate activation derivatives. We obtain the same conclusions for linear layers and provide the demonstration in Appendix A.2
To save memory, two possibilities naturally emerge: either compressing the weights compressing the activation. Weight compression is an extensively explored matter for network acceleration and we do not intend to further this area of research in this work. This leaves with activation compression, which still a new domain of exploration, but shows great potential for prospectively enabling on-device backpropagation. In such regard, thanks to its strong theoretical grounding, tensor decomposition stands as a promising approach.
3.2 Tensor Decomposition
We will present here first the general Singular Value Decomposition (SVD) approach, which is then extended to a multidimensional variant (HOSVD), instrumental for our purposes.
SVD. Given a matrix A E MxN, applying SVD to A consists in a factorization of the form:
A UEVT. U MXM.
ΣE R MxN,
VERNNN
where Σ is a rectangular diagonal matrix composed of r singular values Ste(1rr] with r the rank of A. From this, we can deduce the amount of overall variance of explained by the jth pair of SVD vectors as of / / 57.
Let assume the singular values in Σ are ordered in descending order, 8; 8j. Vi j. Given desired threshold of cumulated explained variance e E [0. 1). it becomes easy to find the "truncation ~K threshold" which is the minimal K E [1,r] such that Li=1 of 2 E. We can then approximate A by only selecting the first columns of U and V and the first singular values from Σ, according to:
Ā U(K [<K) Vie)=
U(KY E MRA
Σ(K) R
V(K) NXK_
Historically, SVD was the first example of tensor decomposition applied to neural network compres- sion and acceleration (55] but applied to the model's 's parameters [49]. We take the SVD decomposition to the activation maps as one possible baseline to compare with more complex low-rank compression solutions.
In the case of convolutional layer since SVD is designed for matrix decomposition, we will simply reshape the activation A, as matrix A, of dimensions B (CHW). Given a desired level of ex- plained variance E, we obtain the decomposition described in 5). We then store in memory the terms

---

U(K) x (()) and ViTi meaning that instead of storing spuce (BCHW) unique elements in memory, only storing apece (B + CHW) unique elements. Regarding linear layers, activations M x N matrices, and applying SVD is much more straightforward, leading to the storage cost of spaee! (M + N)] instead of space NM elements. The larger the explained variance, the closer Á will be A, intuitively allowing for better estimation when performing backpropagation. However, this also means a larger K which results in a larger memory occupation. The goal is then to find an efficient trade-off between the desired explained variance and compression rates.
HOSVD. By construction, SVD is designed for matrix decomposition. It was demonstrated that the reshaping operation tensors introduced structure information distortion, leading to sub-optimal performance [49]. Other methods more suited for tensor decomposition such as Canonical Polyydii [3] were naturally introduced to tackle this issue. Given a n -order tensor T E R&M XM2 XM its Tucker decomposition corresponds to:
(CP) decomposition (20] or Tucker decomposition
T Sx U X2 ((2) X3. x, U(n).
where S E xlaxxLxx--LL, is the core tensor which can be viewed as a compressed version of T, UG) E M,,,, are the factor matrices and their columns correspond to the principal components over the jth mode. The i-mode product "x," ",," of a n²h -ordrr tensor G E IR PR x x and matrix B E RRXP. is a thth -order tensor R E TR Pix Pix x....PP-- Pxxxxxx.41 x...Xx which can be expressed as:
Pr
PII....I-11.PP 1.
9 x; 0x Q=
Σ
Pp..Pa....pp
p-=1
In such a setup, Higher-Order SVD is specific of Tucker decomposition where the factor matrices (+) are orthogonal 44]. As follow-up SVD, we propose to compress activation tensors through HOSVD, with the intuition that each dimension encodes a different variance, potentially providing enhanced compression rates for equivalent performance and vice-versa. Similarly to traditional SVD, we can truncate S and each (T(T) along each mode given desired level of explained variance resulting in:
Ý= x1 (!I)) X2-.. X
U(a) (K.) = T,
(8)
where U.2) (K2) E RR,, corresponds to the Kj first columns of UG) and S RK.....KK, is the truncated version of S. In the following section, we focus on HOSVD decomposition, more precisely on the alterations induced to the backpropagation graph, the resulting compression and speedup ratio, and the error bound. An equivalent analysis can be made for SVD.
3.3 Backpropagation with Compressed Activations
Prior works 13 18] already demonstrated the possibility of performing backpropagation operations in the decomposed space without relying on the recomposition of compressed tensors. In their work, Kim al. compress a 4th -odde kernel tensor W C Dx limited to mode and 2:
through Tucker decomposition
W W (L)) x2 (2),
W E R a C'LL,
T RCL2
Then, is shown that the output is calculated through
convix Ix (2) conv DE D conv 1x (U (X)))
(10)
Our problem is similar although with some fundamental differences linked the need reconstruct gradients. A convolution operator is still required to compute the weight derivatives as introduced in (2) we apply a specific case of Tucker one of the two components. namely the activation A,,
Following the same reasoning, derive that the computation of d2 in (2) becomes:
aw;
ac aw;
convix1 conv.. convix1 (conViX)
(6.)())) _(kk)) convix1 (Bt
BC (B U(k)) )]
(22)
(Ka)
(11)
23 and (6)
where conv, is a 2D convolution with a specific kernel size as defined
(K,)
the vertically padded version of (76) as defined in 19 Demonstration details are available in (K,)
5

---

101
102
10
10P
R
Rs
107
100
104
10
10:
10
10
0.00
11.25
1.30
0.70
1.00
Ki K2 Kg
K1
K1
Kg Ks K4
(a)
(b)
(c)
Figure 2: For a single convolutional layer with minibatch size B, (a) and (b) illustrate the predicted changes in compression rate Rc and speedup ratios Rş as functions of Kj. when comparing HOSVD with vanilla training, respectively, (c) shows the evolution of the SNR with retained variance s.
Appendix A.3
This way, compute the approximated weight derivative without reconstructing the activation, through the successive computation simpler convolutions.
Fig. T illustrates our method. During training, the forward pass proceeds as usual, with one key modification to memory management. Instead of keeping complete activation maps in memory, we store only their principal components, which are derived from the decomposition process, During the backward pass, the principal components are retrieved from memory and used for calculations as described in (TT).
3.4 Complexity and Error Analyis
Computational Speedup and Space Complexity. For simplicity we assume here that A and A; have the same shape and that the truncation thresholds across all modes are equal for HOSVD, i... K2 K₂ = K3 K4. Considering different thresholding values, we can predict the relative improvement of HOSVD to vanilla training in both space complexity (30) and computational speedup for backward pass. As illustrated Fig. 2a and Fig. 2b our method more effective for both space complexity and latency for the backward pass when the truncation threshold decreases and the activation size increases. More details are provided Appendix|A.A
We hypothesize that the first components along each dimension variance, implying that with relatively low values of Kj, we can achieve good training performance. We later empirically confirm this assumption in variety of experimental setups (Sec.
enough encode most of the
ac Signal to Noise Ratio. For simplicity, we rewrite this paragraph JW, as AW, A, as and BC Jh... as AV. Regarding the error introduced by truncating I in order to retain the components corresponding to a given level of explained variance E [0, 1J. we demonstrate that the energy contained in the resulting gradient is equal to the energy contained in Ż. We note I, AW and AY as the input activation, weight derivative, and output activation derivative in the frequency domain; Iu, v). the spectrum value frequency (u, v). Applying the discrete Fourier transformation Í gives us i ET, which we use to compute the signal to noise ratio SN Ry:
E(u,o) 7[, v]2 SNR; E(Tu, - 8/ ol) (1 8)²
1
(12)
Similarly, in the frequency domain, as the convolutional operation becomes regular multiplication, AW becomes AW = TAY = eTAY. Y. As for (12). we obtain SNR = (1 (1-6)-² = SN SNN A visual representation provided in Fig. 2c Analytically, this confirms the idea that the closer is to 1, the larger the energy is transferred from the compressed input to the weight derivatives (for = 0.8 we have SN RAW 25).
Additionally, in our setup since we are only compressing the activations and given and 3 the introduced error only transfers to the weight derivative for each layer. The activation derivatives that are propagated from one layer to another are exact computations they do not involve the

---

1st dimension
2nd dimension
1.0
1.0
Explained Varianced
Explained Varianced
0.8
K2
12
0.S)
32
K (()
24
0.6
0.4
51
101
128
201
401
576
K
Kg
Figure 3: Explained variance for the first two dimensions of the activation map in the 4th last layer when fine-tuning the last four layers of MCUNet using HOSVD on CIFAR-10 10, following setup A.
activations. This means that the error introduced when compressing the activations at each layer does not accumulate through the network.
Experiments
In this section, describe the experiments conducted to support the claims presented in Sec. First, we introduce the setups used for our experiments (Sec.4..) then, we analyze the energy distribution the different dimensions of HOSVD, providing an overview of the typical values of K (Sec. 4.2 finally, we test our algorithm in different setups, state-of-the-an aar architectures and datasets to evaluate the tradeoff between accuracy and memory footprint (Sec. 4.3). Experiments were performed using NVIDIA RTX 3090Ti and the source code uses PyTorch 1.13.1.
4.1 Experimental setup
To validate approach, we perform variety of computer vision experiments split across two tasks, classification and segmentation. Classification. We explore types of fine-tuning policies:
Full fine-tuning (referred to as "setup A"): Following conventional fine-tuning trends, we
load models pre-trained on ImageNet [2] and we fine-tune them variety of downstream datasets (C1FAR-10, CIFAR-100, CUB [46]. Flowers
32 and Pets (54)
Half fine-tuning (referred as "setup B")) Following Yang et al. approach, each classifica- tion dataset Net,geNet, CIFAR-10/100) is split into two non-i.i.d. partitions of equal size using the FedAvg [29] method. The partitions are then split as follows: 80% for training and 20% for validation. The first partition is used for pretraining, and the second partition is used for finetuning.
Semantic Segmentation. Similarly to the half fine-tuning method, we reproduce Yang et al. seg- mentation setup: we fine-tune models pretrained on Cityscapes G by MMSegmentation [5]. Here there is only one downstream dataset which is Pascal-VOC12 3 Experimental details for both tasks
thyper-parameters. policy, etc.) are provided Appendix B.2
Memory Logging. For HOSVD and SVD, instead of focusing on compressing the tensor based on rank, control compression through the desired amount of retained information. As consequence, we cannot explicitly control below, we will always include two columns displaying peak and average memory, along with their standard deviations.
the memory usage of the principal components. In the results presented
4.2 Explained Variance Evolution
In this section, conduct experiments to fine-tune the last four convolutional layers of MCUNet 24 following Setup A, and CIFAR-10 as the downstream dataset. Using HOSVD with e set to 0.8, Fig shows the explained variance retained across dimension j as function of Kj, where j {1,2)
corresponds to the two largest dimensions of the activation map. We define K! (==x) as the number of principal components required to retain at least fraction z of the explained variance.
7

---

90
Topl Validation Accuracy (%)
85
80
HOSVD
SVD E =
76
Gradient Filter R7
19. 1% higher in accuracy
Gradient Filter R4
Gradient Filter R2
65
Vanilla training
60
18.87 smaller in memory
55
50
10
10a
10a
10a
10a
10
Peak Activation Memory (kB)
Figure 5: Performance curves of an MCUNet pre-trained ImageNet and finetuned on CIFAR-10
with different activation compression strategies.
Top1 Vahdation Accuracy (%)
82.5
Topl Valiiation Aocuaee (5)
We observe that along the largest dimensions (batch size and number of channels), less than 20% the components capture more than 80% of the explained variance, confirming the as- sumption proposed in Sec. 3344 Additionally, the curves observed present logarithmic behav- ior, hinting at the possibility of reaching high explained variance with little loss regarding the compression ratio. This is especially important as the SN R transmitted to the weight deriva- tives increases quadratically to the explained variance (Fig. 2cp. The results for the other lay- are presented in Appendix [B.3]
Poak mem (MD)
80.0
77.5
100
75.0
mm (01
70.0
10
0.70
0.74
0.78
0.82
0.86
0.90
0.94
0.98
Figure Behavior of topl validation accuracy and peak memory when applying HOSVD with different explained variance thresholds e when finetuning the last four convolutional layers of an MCUNet model using the CIFAR- dataset on setup A.
Fig. 4 illustrates the results of perform- ing HOSVD with different explained variance thresholds e. The results are averaged over three different random seeds. For smaller than 0.8, we observe that as it increases, we achieve sig- nificant gains in accuracy, along with impressive compression ratios. However, when exceeds 0.8, the accuracy growth starts slowing down. Above the 0.9 threshold, the exponential growth of peak memory results in worsened tradeoff between accuracy and compression ratio. Therefore, in subsequent experiments presented this paper, we will use values of 0.8 and 0.9.
4.3 Main Results
MCUNet on classification with setup A. In this experiment, we fine-tune on CIFAR-10 an MCUNet pre-trained on ImageNet, with the number of finetuned layers increasing from 1 to 42 (all layers). We compare vanilla training, and gradient filtering with patch sizes of 2, 4, and 7, SVD, and HOSVD with an explained variance threshold of 0.8. For each method, we compare the effect of fine-tuning different model depths. Fig. plpresents the performance curves for our experiments, with the X-axis representing activation memory in kiloBytes (kB) on logarithmic scale and the Y-axis representing the highest validation accuracy. Each marker indicates the number of convolutional layers finetuned. Y-axis
For example, on the yellow curve representing the HOSVD method, the marker closest shows the result when finetuning the last convolutional layer, the next marker represents finetuning the last two convolutional layers, and so on. The most effective method is the one whose performance curve trends towards the upper-left corner of the plot.

---

Table 1: Experimental results on ImageNet-Ik. "#Layers" refers to the number of fine-tuned convolutional layers (counted from the end of the model). Activation memory consumption is shown MegaBytes (MB).
MobileNetV2
ResNet18
Method
Method
ayyyers
Apc
Mem (MR)
Mean Mom (MR)
alayers
Acc
Peak Mem (MB)
Mean Mem (MR) |
Vanilla training
All
62.6
24,0
165T81
15.51
[6S1&11000 15.30=0000
Vanilla training
All
663
72.8
$32.88
12.25
552.88 2.2.882200 0100
12.255=00000
30.63
0.00
Gradient Filter R2
62.6
5,00
5,00 0.00
Gradient
2
68.7
4.00
4.00
65.2
9.38
9.58 + 0.00
Filler R2
4
69.3
7:00
7.00 0.00
SVD
7.58
1.71
0.8)
65.2
14.76
14.61 = 0.09
(c 0.8)
71.1
19.98
19.72 = 0.28
SVD
62.3
8.97
891 + 0.18
SVD
69.7
9.86
9.77 0.13
(.99)
65.5
20.35
20.20 0.07
te 0.9)
71.3
24.81
24.66 0.17
HOSVD
61.1
0.15
0.13
HOSVD
69.2
0.7
091 0.08
(e 01.8)
63.9
0.73
0.68 a 0.03
[c ===0.. 0.8)
70.5
2.89
2.74 0.12
HOSVD
61.8
0.43
0.43 0.00
HOSVD
09.5
2773
2.65 0.10
0.9)
64.8
1.76 0.08
71.1
7.96
7.66 0.21
Method
MCUNet
Method
ResNet34
MLayers
Acc
Peak Mem (MB)
Mean Mem (MB)
Maayers
Acc
Peak Mem (MB)
Mean Mem (MB)
Vanilla training
627
75.7%"
632.98
13.78 20000
Vanilla
8076
839.04
12.25
"12.25
X39
64.7
19.52
19.52 + 0.00
training
72.2
24.50
24.50 0..00
4.50
4:00
R2
64.4
6.38
6.38 0.00
Filter R2
70.9
8.00
8.00 0.00
SVD
02.0
742
7.11=0.12
SVD
099.2
6.70
644
0.8)
SVD
62.1
10.32
10.26 0.08
SVD
69.4
9.10
8.96 2 0.23
(s - 0.99
64.6
14.39
14.26 0.13 0.13
(c 0.9)
4
72.0
19.11
18.83 + 0.37
HOSVD
HOSVD
68.7
0.30
0.02
0.8)
63.9
0.88
0.78 0.07
71.1
1,11
1.02 0.07
HOSVD
62.0
1.32
1.27 0.06
HOSVD
69.2
0.71
0.65 # 0.08
0.9)
64.4
2.52
2 0.15
0.9)
71.9
3.24
3,09 0.13
We observe that as the number of finetuned layers increases, the gradient filtering accuracy increases up to a certain point, and then deteriorates, whereas the accuracy for SVD, HOSVD, and vanilla training keeps improving with additional layers, along a similar trend. Intuitively, gradient filtering might propagate errors through the layers during training, while our HOSVD method keeps the error introduced on each individual layer confined to that specific layer as demonstrated in Sec. 3.4 Moreover, we observe that for identical depths, SVD accuracies consistently exceed HOSVD ones. We hypothesize that this due to HOSVD performing SVD across all modes of the tensor: potentially loses information in all modes, whereas SVD only loses information in one mode. Compared to SVD, HOSVD significantly reduces memory, given equivalent accuracy levels (up to 18.87 times). Similarly, for equivalent memory usage, HOSVD yields greatly improved accuracy compared to SVD (up to 19. 11)). When compared to methods such as gradient filtering and vanilla training, given an equivalent memory budget, HOSVD yields significantly higher accuracies. Notably, fine-tuning all layers with HOSVD requires much less memory than fine-tuning only the last layer with vanilla training, which is consistent with the theoretical compression ratio shown in Fig. Additional experiments with setup A can be found in Appendix B.5
ImageNet Classification with setup B. Table presents the classification performance and memory consumption for MobileNetV2 37.. ResNet 18. and ResNet34 [15] models using various methods, including vanilla training, gradient filtering, HOSVD, and SVD on the ImageNet dataset. We observe in most cases that, for the same depth, SVD and HOSVD are competitive with the gradient filtering method in terms of performance while reaching much lower activation memory with HOSVD.
Segmentation. Table 2 reports the segmentation performance and memory consumption for variety of architectures as presented in Yang et al.. In general, we observe that increasing the level of explained variance from 0.8 0.9 substantially increases the performance with little trade-off on retained activation memory. This further confirms that most of the explained variance is contained in the first few components, allowing for efficient generalization with high compression rates.
5 Conclusion
In this work, we have addressed one of the main obstacles to on-device training. Inspired by traditional low-rank optimization approaches, we propose to compress activation maps by tensor decomposition, using HOSVD as a supporting example (Sec. 3.2) We demonstrate that the compression error introduced is bounded and confined to each individual layer considered, validating our approach

---

Table Experimental results for semantic segmentation. mloU the mean Intersection over Union,
and MAcc is the micro averaged accuracy.
Method
PSPNet
mloU
mAcce
Peak Mem
Mean
(MB)
mLAcc
Pesk Mer
Meen Mem
Vanilta minnnng
55"
All
10
53.17
90735
54.97
51.77
67.18
69.46
123.0
920 76
352.00
128700 s 0.00
920.7K 0/00
trining
Vanilia
All
10
9622
48.97
45.62
36.51
62.11
53.35
2623.40
604.00
2602 0.00
10470 :: 0.00
604.00 0.00
Gradien Filter
5
39.54
51.59
3,00
0,00
Gradieza Filter
5
35.73
45.78
6.50
4,50
51.2
65.01
22.00
22.00
10
44.80
$7.38
37.75
57.75 0.00
SVD
5
38,97
50.04
90.40
2,50
SVD
5
34.98
44.48
58.50
47.77
(c
10
52.03
00:44
238.20
22250:4..7
(c
10
44.73
56.53
377.65
304.24 6.93
SVD
90 94
58
SVD
78.65
52.7
2.5.
482.00
475.17
BOSVD
29.03
0.30 6.44
HOSVD 0.9)
0.07
0.07
0.0)
8.20
6.16 0.62
Method
DLV3
Method
DLV3-M 3]
MLayers
mlnU
mAkec
Peak Mem (MBB
Mean Mem (MB)
fLayers
minU
Peak Mem (MBB)
Meon Mem (MB)
Inining
Vusilla
All
55.04
3673
44
32.95
09:07
23.03
1128.00
338.00
560.00
1128 8.000
336320 E
300.00 0.00
Vusilla uaining
All
10
38.38
55.87
47.91
39.51
01.57
89.47
2758.00 01
240.00
620.00
2788.1 0.00
24000 0.00
620.00 = 0.00
Geadher
20.47
20
83.47
6562
SVD
SVD
40.47
291
291.40
204.50
200.69
(c 0.9)
472.70
406.52
(6 0.9)
493.85
5.77
HOSVD
56.52
50.14
2,66
2.02+4000
HOSVV
35.85
45.64
0.68
0.55 0.00
6 = 0.89
10
50.11
63.14
1.93
1.48
65 = 0.89
10
42.68
54.17
1.04
0,96 a 0.00
HOSVD
40.19
52.3
12.64
12.35 0.000
HOSVD ( 0.00
57.06
47.23
4,56
4.16
( 0.00
10
52.26
68.8
9.33
7,24-00000
10
45.75
$9.61
5.52
4.79 a
Method
FCN 27
Method
UPerNei [88
Mayar
mlall f
mAoe
(MR)
Mon (MR)
4aaym
mlat! 1
mAoo
Pork Mom (MR)
(MR)
=:
All
5
139,000
1:9000000000
1436.00
1436.0
Gradient Filter
27.24
18,00
Gradien Filier
46.79
33.00
0.00
36.91
50.14
120.00
120.00
36.50
SVD
26.62
58.43
196 90
491441
SVD
5
46.96
59.46
804.55
764 17.28
(e = 0.89
$4.60
45.71
116 70
268.37 13.76
6 = 0.89
10
47.00
61.00
$40.70
839.13 1
SVD
31.92
42.55
251.50 50
245-40.81.00 1:74
SVD
47.22
59.77
1073. 38
1056.52 14,92
( - 0.d)
10
43.04
$4.53
406 30
906. 146114-4433 4.89
(6 - 0.90
10
48.3
61.16
1136.90 90
1119.8 11.49
HOSVD (6
5
26.34
36.14
1.43
1.26:20.08
HOSVD (4
5
45.28
57.80
1.35
1.52 0.02
10
30.91
43.19
3.77
2.54
10
46.44
58.93
1.68
1.63 4 0.03
BOSVD (11)
46
41.12
0.39
IKSVD
5
59.29
412
4.59
39.89
1.55
7.81
through extensive experimentations. This work paves the way for line of research combining the accumulated knowledge on tensor decomposition strategies and the recent field of on-device learning.
Acknowledgements
Part of this work was funded by Hi!PARIS Center on Data Analytics and Artificial Intelligence, by the European Union's HORIZON Research and Innovation Programme under grant agreement No 101 101120657, project ENFIELD (European Lighthouse to Manifest Trustworthy and Green Al) and by French National Research Agency ANNR22--PFFT-0003 and ANR-22-PEFT-ONE as part of France 2030, the NF-NAI project and NE-FITNESS project.
10

---

References
[1] Toru Baji. Evolution of the gpu device widely used in ai and massive parallel processing. In 2018 IEEE 2nd Electron devices technology and manufacturing conference (EDTM), pages 7-9. IEEE, 2018.
[2] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33:1 1285- 1 11297, 2020.
[3] Liang-Chiee Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
[4] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprini arXiv:1710.09280 2017.
[5] MMSegmentation Contributors. Mmsegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.
[6] Marius Cordts, Mohamed Omran, Sehastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223 2016.
[7] Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485- 532, 2020.
[8] Li Deng, Geoffrey Hinton, and Brian Kingshury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 5599-8603. IEBE, 2013.
[9] Sebastian Eliassen and Raghavendra Selvan. Activation compression of graph neural networks using block-wise quantization with improved variance minimization. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7430-7434, 2024.
[10] Shahaf E. Finder, Yair Zohav, Maor Ashkenazi, and Eran Treister. Wavelet feature maps compression for image-to-image cnns. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 20592-20606. Curran Associates, Inc., 2022.
[11] Robert M. French. Catastrophic interference in connectionist networks:
Can it be predicted, can be prevented? In Proceedings of the 6th International Conference on Neural Information Processing Systems. Morgan Kaufmann Publishers Inc., 1993.
[12] Georgios Georgiadis. Accelerating convolutional neural networks via activation map compres- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7085-7095, 2019.
[13] Marawan Gamal Abdel Hameed, Marzieh S Tahaei, Ali Mosleh, and Vahid Partovi Nia. Con- volutional neural network compression through generalized kronecker product decomposition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 771-779, 2022.
[14] Tyler L Hayes and Christopher Kanan. Online continual learning for embedded devices. arXiv preprint arXiv:2203. 10681, 2022.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] Geoffrey Hinton.
The haward-forwart algorithm: Some preliminary investigations. arXiv preprint 133:4:2212.13345, 2022.

---

[17] Ozlem Durmaz Incel and Sevda Ozge Bursa. On-device deep learning for mobile and wearable sensing applications: review. IEEE Sensors Journal, 2023.
[18] Yong- Deok-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprini arXiv: /5/ 1. 1.05300, 2015.
[19] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis
drei A. Rusu, Kieran Milan, John Quan. Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 1 114, 2017.
[20] Tamara G Kolda and Brett Bader. Tensor decompositions and applications. SIAM review,
5 51(3):455-500 2009.
[21] Alex Krizhevsky, Πya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[22] Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting acti- vation sparsity for fast inference on deep neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5533-5543 PMLR, 13-18 18 Jul 2020.
[23] Young D Kwon, Rui Li, Stylianos I Venieris, Jagmohan Chauhan, Nicholas D Lane, and Cecilia Mascolo. Tinytrain: Deep neural network training at the extreme edge. arXiv preprint wrXiv:230.009988 2023.
[24] Lin, Wei-Ming Chen, Yujun Lin, john cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning on iot devices. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11711-11722. Curran Associates, Inc., 2020.
[25] Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems, 35, 2022.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[28] Ivan Markovsky. Structured low-rank approximation and its applications. Automatica, (4(tmmmticc,4444)89111 909, 2008
[29] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication-eficiinn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[30] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3523-3542) 2021.
[31] Ali Bou Nassif, Ismail Shahin, Imtinan Attili, Mohammad Azzeh, and Khaled Shaalan. Speech recognition using deep neural networks: A systematic review. IEEE access, 7:19143-19165, 2019.
[32] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
12

---

[33] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. Neural Netw. 113, 2019.
[34] Danilo Pietro Pau and Fabrizio Maria Aymone. Suitability of forward-forwart and pepita learning mlcommons-tiny benchmarks. In 2023 IEEE International Conference on Omni- layer Intelligent Systems (COINS), 2023.
[35] Aèl Quélennec, Enzo Tartaglione, Pavio Mozharovskyi, and Van-Tam Nguyen. Towards on- device learning on the edge: Ways to select neurons to update under a budget constraint. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 685-694, 2024.
[36] Berkman Sahiner, Weijie Chen, Ravi K Samala, and Nicholas Petrick. Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology. 96(1150):20228878 2023.
[37] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[38] Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2022.
[39] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXi:: 1556, 2014.
[40] Gabriele Spadaro, Riccardo Renzulli, Andrea Bragagnolo, Jhony H Giraldo, Attilio Fiandrotti, Marco Grangetto, and Enzo Tartaglione. Shannon strikes again! entropy-based pruning in deep neural networks for transfer learning under extreme memory and computation budgets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[41] Emma Strubell, Ananya Ganesh, and Andrew McCallum, Energy and policy considerations for deep learning in nip. arXiv preprint arXiv: 1906.02330 2019.
[42] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nip pipeline. arXiv preprint
2019.
[43] Ledyard R Tucker. Some mathematical notes three-mode factor analysis. Psychometrika, 31(3):279-31, 1966.
[44] M Alex O Vasilescu and Demetri Terzopoulos. Multilinear subspace analysis of image ensem- bles. In 2003 JEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 2, pages 11-93, IEEE, 2003.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in newral information processing systems, 30, 2017.
[46] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. caltech-ucsd birds-200-2011 dataset, 2011.
The
[47] Yue Wang. Jun Jason Zhang. Xinhu Zheng, Xiao Wang, Yong Yuan, Xiaoxiao Dai, Jie Zhang. and Liuqing Yang. Where does alphago go: From church-turing thesis to alphago thesis and beyond. IEEE/CAA Journal of Automatica Sinica, 3(2): 2016.
[48] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418 434, 2018
[49] Ou Xinwei, Chen Zhangxin, Zhu Ce, and Liu Yipeng. Low rank optimization for efficient deep learning: Making a balance between compact architecture and fast training. Journal of Systems Engineering and Electronics, 2023.
13

---

[50] Jian Xue, Jinyu Li. and Yifan Gong. Restructuring of deep neural network acoustic models with 2013.
singular value decomposition. Interspeech, pages
[51] Junhuan Yang, Yi Sheng, Yuzhou Zhang, Weiwen Jiang, and Lei Yang. On-device unsupervised image segmentation. arXiv preprint S3.I::2303./255533 2023.
[52] Yuedong Yang, Hung- Yueh Chiang, Guihong Li, Diana Marculescu, and Radu Marculescu. Efficient low-rank backpropagation for vision transformer adaptation. In Proceedings of the 37th International Conference Neural Information Processing Systems, pages 14725-14736, 2023.
[53] Yuedong Yang. Guihong Li, and Radu Marculescu. Efficient on-device training via gradi- filtering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3811 1-120,0 2023.
[54] Hui Zhang, Shenglong Zhou, Geoffrey Ye Li, and Naihua Xiu. 0/1 deep neural networks via block coordinate descent. arXiv preprint NiKiv:2206.093799 2022.
[55] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions pattern analysis and machine intelligence, 38(10):1943-195 2015.
[56] Hengling Zhao, Yipeng Liu, Xiaolin Huang, and Ce Zhu. Semi-tensor product-based tensorde- composition for neural network compression. arXiv preprint /2X0 2021.
[57] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jía. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2899 2017.
14

---

A Additional Theoretical Details
A.1 Forward pass Vs. Backward pass
We consider the same notations introduced in Sec. 3.1 Regarding the training of a convolutional layer:
There is convolution operation in each forward pass with computational complexity of FLOOP PPLOPS(D"CC" BH'')
In each backward pass, there two convolution operations, corresponding formu- las 2 and (3) and one weight update operation, with computational complexities of and (FLOPs(D*CC") respectively.
FLOOPS (D&CC BH'W"), OFLOPS (D
BHW) (HCC
From this, we can calculate the ratio of computational complexity between forward and backward pass RFLOPS as follows:
D''C'BH'W'
RELOP: D'CC'BHW' D'CC'BHW DZCC"
(13)
It is evident that RELOPs is always smaller than one, indicating that the backward pass is always more computationally intensive than the forward pass.
A.2 Backpropagation Derivatives in Linear Layers
With the same notations introduced in Sec. 3.1 but in the case of a linear neural network represented as sequence n linear layers:
F(X) = (Fw,, Fw,-, o... Fw, Fw,)
(14)
where W, E Rxx corresponds parameters matrix of the 7h layer. This layer receives an I-feature
input A; E RBx1 with minibatch size B to produce an O-feature output Ait1 E RBx0 The
derivatives in that setup become:
Oc
OC
dAi+1
OC JA.1I OW;
A.
(15)
OW.
0Ai+1
OC
dC
0Ai+1
OC
W.T
(16)
0A,
dA
0A,
0A,+1
The same conclusions as for convolutional layers can naturally be derived from these equations for linear layers, namely that activations and weights occupy most of the space in memory when performing backpropagation.
A.3 Details of Backpropagation with Decomposed Activation Tensors
Since the convolution operation involves other variables such stride, dilation, and groups, provide more precise description by rewriting in ( ac 8W, AW xxxxxCxDx A, as
I E xB and ac JA... AY E X which gives us:
15

---

H W
AWey.c.kl
Io,o, N. Df
b=1 11=1= =1===
where:
h = N' Stride + k x Dilation,
w' x Stride + I x Dilation,
Groups
Nout
C'
N Groups
(17)
C = Nout + +
Cg = Nin + +,,
è' E [1, Noarh,
C E [1, Nin)
E [1, Groups],
and E DI,
I the padded imput.
In our case, before being saved in memory, I is decomposed and compressed through truncated HOSVD as presented in
K
K2
Kg
K
Sz, da
x xU (Kibon
xU
xU( (Ks),,ng
xU (Ka)
(18)
(Ka)og
Ki
K
Therefore, in the backward pass, its padded version be restored using the following formula:
K1
K2
K2
K
Lo,cg... w. A1
S Say
xU
(Ki)a.
xU
(Kaleg A2
(Ka)a.
(Ks)
(4)
(19)
Rrg
Aog
where, (18)) and U(k,) are U(2 and U respectively, Then, substitute 199
with vertical padding (top and bottom edges only). into which gives us, after reordering and grouping:
(Kg)
Kg
(1)
AVo.e, M ou U (1)
(Ki)umia
(20)
Kg
(22)
kk k2.kiikk
St, ks (Ks)nnss
(21)
k==1
Kg
((3) k2.hi
7 Kz, h,k
(Koloorka
(22)
k==1
H'
W
K
((1)
Z h 2(1)
(23)
ci k2,k,
W=
K₂
z(4)
AW
(24)
(K2)ey
1==1
Computing 20 and corresponds to performing convolutions while H' x W' convolution, resulting (1I.
23 is
16

---

A.4 Details of Overhead, Computational Speedup and Space Complexity
Following the notations introduced in the main paper, we propose in this section analitically study the overhead introduced in the forward pass when performing HOSVD compression, the speedup of efficient weight derivative computation and the required space store the compressed activations.
Overhead. During the forward pass, the over- head of activation compression is the compu- tational complexity of performing the tensor decomposition, which can be calculated fol- lows:
10th
Forward pass FLOPs
10
The computational complexity of SVD for ma- trix of size m x n with m 2 n is (m HOSVD essentially performs SVD on each mode of the tensor. During the forward pass, for each convolution layer, given an activation map of size B x C' x H W, HOSVD in- volves performing SVD on four matrices of sizes B x CHW, C x BHW, H x BCW, and W BCH. Therefore, the computational complexity for decomposition in the forward pass is:
109
109
10a
HOSVD
Vauilla teaining
16
64
128
B= C
W C'
Figure Predicted evolution of FLOPs for the forward pass of both vanilla training and HOSVD.
FLOP Soverhead OLOOPS max(B,CHWY min(B,CHW) BHW)) x min(C, BHW)+
max(H, BCW)) min(H, BCW) + max(W, BCH)² min(W, BCH)
(25)
Additionally, the amount of FLOPs that vanilla training requires to perform the forward pass is:
FLOPsvanilla D2CC'BHW
(26)
From this, the total FLOPs that HOSVD requires is:
FLOPSHOSVD Omors(
max(B, CHW)² min(B, B,CHWCCCWHHH max(C, BHW)² x min(C,BBW))
max(H, BCW) BCW)) min(H, BCW) max(W, BCHY² min(W,BCH)) D'CC'
(27)
Based on these calculations, we represent in Fig. the evolution of FLOPs in the forward pass for both vanilla training and HOSVD, showing that our method results in an increased latency in the forward pass.
Speedup. The key difference between HOSVD and Vanilla BP lies in computing AW. For Vanilla BP, the cost of computing AW is (D'CC(D Whereas our method involves:
Computing (1)) OFLOPS K,C'H''B),,
Computing Z OFLOPS K1K2HK,K3),
Computing (3 OFLOPS K,K2HKKK, K2KHHWK)),
Computing (4) OFLOPS H'W'' K1KGHK + K2HHHW H'K,DDH'W)).
Therefore, the complexity for computing AW with our efficient reconstruction is:
K2 (K K', /AT.:///SS...........C//C/C////.
(28)
We thus deduce the speedup ratio Rs for the backward pass, between vanilla training and HOSVD (results represented Fig. 2b
D2CC' BH'W' W.t.SK/WWWWW.................
Rg K.C'HW
(29)
Space Complexity. For vanilla training, storing A, correspond to the storage of Q (BCHW) elements.
17

---

For HOSVD, instead of storing the entire tensor, we store its truncated principal components: S, U(I) u(2) U(a) (Ki)' (Ka)) (Ks)' and U (Ka)' which corresponds to a total of space (K KK2K3K4 BK1
CK2 + H + WK)) elements.
Thus, obtain the storage complexity ratio Rc follows (results represented in Fig. 2a
BCHW
Rc KiK2KaKa + BK + CK2 + H + WK.
(30)
B Additional Experimental Details
B.1 Variance of Different Runs
We conducted classification experiments using MCUNet with setup A, fine-tuning on 100,R-10, and segmentation experiments using the Deep checkpoint provided by (53) fine-tuning on the augmented VOC12 dataset. Both experiments employed three different random seeds (233, 234, and 235). The results, presented in Table and Table 4 indicate no significant variation in performance across the different random seeds. Therefore, for subsequent experiments, we used a single random seed, 233.
Table 3: Classification with MCUNet on different random seeds.
#Laayes Vanilla training Gradient Filter R2 Gradient Filter R4 Gradient Filter R7 SVD 0.8)
HOSVD (c = 0.8)
71.8240.30
70.7520.19
87.68=0001
70.42=0.18
66.84220188
63,46:10.09
83.15.10.19
82.19:10.11
89.54.2003
80.98-0.06
80.79.10.04
77.22:10.03
Table 4: Segmentation with DeepLab on different random seeds.
Method
#Layers
mIoU
mAec
38.62::0.01
50.41:0..06
HOSVD (c = 0.8)
10
50.33:10.05
63.39
5
39.711:0.00
51.39-00007
SVD (8 = 0.8)
10
52.47-+0.04
66.09++0.01
B.2 Detailed Experimental Setup
similar finetuning policy [53 Specifically, we finetune the checkpoints for 90 epochs with L.2 gradient clipping with a threshold of 2.0. We SGD with weight decay of 10-4 and momentum of 0.9. The data is randomly resized, randomly flipped, normalized, and divided into batches of 64 elements. We use cross-entropy as the loss function. The difference from their setup lies in the initial learning rate value after the warm-up epochs: in our case, the learning rate increases linearly over 4 warm-up epochs up to 0.005 (while in [53] it is 0.05). After this, similarly their approach, the learning rate decays according to the cosine annealing method in the following epochs.
ImageNet Classification. In this setup,
Other dataset finetuning. We the same set of hyperparameters described in 5 for both setup A and setup B. For training, we use cross- cross-entropy loss with the SGD optimizer. The learning rate starts at 0.05 and decays according to the cosine annealing method. Momentum is set to 0, and weight decay is set to x 10-4 We apply L2 gradient clipping with a threshold of 2.0.
For setup B, batch normalization layers are fused with convolutional layers before training, a common technique for accelerating inference. We set the batch size to 128 and normalized the data before feeding it to the model.
Semantic Segmentation. Following the policy outlined in [53] we utilized the pretrained and calibrated checkpoints provided by the authors for our experiments. The models were pretrained on Cityscapes using MMSegmentation, then we finetuned them on the augmented Pascal- VOC12 dataset. The optimizer's 's learning rate starts at 0.01 and decays according to the cosine annealing schedule during training. Additionally, we set the weight decay to 5 10-1 and momentum to 0.9. The batch size is set to 8. For data augmentation, we randomly crop, flip, apply photometric distortions, and normalize the images. Cross-entropy used as the loss function. Models are finetuned for 20, 000 steps.
18

---

B.3 Additional Explained Variance Evolution Results
1d dimecusion
2nd dimeensio
3 dimension
4th dimenston
Explinet Vatanced
Varionndd
Explaned Vanuned
Explined Vasiamedd
0,588
0.00
w
0.08
0.58
0.06
0.50
10 100
Ki
K2
(a) Last Layer.
1d dimensson
2n dimension
3rd dimension
4th dirnension
Explined Vari setanoe
Explanne Variuned
Explaied Vanined
Explamed Vanmodd
3,8-
0.8
-67
=30
0.8-
Rig
=
0.6
Ag
A2
m00
0.5
0.0
128
(b) nd
Last Layer.
1st dimension
2nd dimension
3rd dimension
4th dimension
1.0
1.0
1.0
Eaglined Varinzed
Explined Variened
Explaied Varanoed
Boolieed Varaneed
0.0
K
1
-7
0,8
R2
=50
0.0
Ri
=1
06-
0.7
=22
Rd
-2
570
Ki
K2
Kg
Ki
(c) 3dd
Last Layer.
dimennsio
dimension
3rd dlmmensio
dimeension
Exylaine oolaaned Vatiazzd
1.0
Exgl aneel Vacisare
1.0
Explaned Vartzzed
1.0
Exglamed Varizzed
L.0
0.8-
0,8
Retental
-6
R
S0
0.8
B.0
R:
0.6
0.4
51
100
122
201
676
Xi
Kg
K2
KG
(d) 4th Last Layer.
Figure 7: K2 and variance for each of the last four layers of MCUNet when fine-tuning them using HOSVD on CIFAR-10 following setup A.
presents the explained variance as a function of K, at the jth dimension of the four fine-tuned layers of MCUNet. We observe that in the first two dimensions, which are also the largest, more than 80% of the tensor's energy is concentrated within the first 30% of the principal components.
As anticipated in Sec. 4.2 Fig.
With the same experiment, Fig. shows that K2 tends to gradually increase after each training epoch, then peaks and begins to decline. Intuitively, at the beginning, the activation map energy is mainly concentrated in the few first components. As training progresses, this energy starts to spread out to other components, leading to an increase in the value of Ky. Over time, the model learns how to efficiently condense the energy, leading to gradual decrease in K₁- This instability in K₃ throughout the training process is why we had to log memory as described in Section
19

---

1s dimersion
2nd dimension
3dd dimesssion
elimension
1.00
2.0
100
1,00
<2 1.00
12 1.55
1.00
42 1.00
0.95
1.0
0.95
0.95
21
41
50
21 Epoch
4:
50
21
30
21
41
50
Epoch
Epoch
Epoch
(a) Last Layer.
dimecasion
2dd dimensioo
3rd dimension
dimension
20
0
91
50
41
20
50
Epech
Epoch
Epoch
Epoch
(b) 2nd Last Layer.
1st dimenssion
2 dimrensio
3rd dinenssoon
4 dinenssion
40
20
21
03
21
41
20
21 Epodh
41
21 Egoch
41
Rood:
Epoch
(c) 3rd Last Layer.
1 ditnension
2nd dimnemsion
3rd dimenssion
4th dimenssio
40
21
21 Rpoch
21
21
41
Rood:
Fpodh
(d) 4th Last Layer.
Figure 8: Behavior of K2 during training for each of the last four layers of an MCUNet when fine-tuning them using HOSVD CIFAR-10 dataset following setup A with two different values of e.
B.4 Processing Time Results
This section presents latency comparisons between vanilla training and HOSVD, We conducted the experiments using MCUNet on a single batch of CIFAR-10 10 dataset with setup A and for one epoch only.
Fig. 9 compares the execution time of the forward, backward, and total training processes between vanilla training and HOSVD. It can be observed that the backward processing time of HOSVD is tens of times lower than vanilla training, but the forward time is up to thousand times higher. As result, when considering the total training time (the sum of forward and backward), HOSVD is on average 4.29 times slower than vanilla training. This outcome is entirely expected, as the decomposition
process introduces overhead, predicted and described in Fig. 6 and Appendix A.4 A.4
20

---

10f
10
Vazile Loaiiti 3OSVD
Bockowarl Tame (s)
Total Time (o)
10
Porward Time
10
10-1
10-4
X. in JvWrrage
22.18
# Layerrs
Layyrrs
Leggers
(a)
(b)
(c)
Figure 9: (a), (b), and (c) represent the time in seconds taken by the algorithm to respectively perform Forward, Backward, and the total training process, when fine-tuning MCUNet batch of CIFAR-10 10 with Setup across to 10 layers.
Table 5: More results with setup A.
Ave
AcTT
ANeT
belligg
$7.43
13.03 1/0
17.23
20.4
$7.43
47.23 ccc
17.43
1742.000
476
67.40
17.0..00
1023
100100000
127741000
1223
351
1750
230
C5 4.31
13.4
03.3
00.0
6.16
26.5
03:40:11
06.1
Varda
10.3
18.00
18:00:20 3.40
-41
40.0
3120
1020
1000
31251001
10531000
125
1200
03:4
1044
404
40.5
10.5
Vacia
41:35
00.28:20 3.40
01.25
1509
01.25
11351000
15.
01.25
01.5
1225
1023
100000000
107740000
1000
1200
130
13
13.3
50.0
1.80
11.
18:0--22
17.15
3811
17.6
Vacia
06-1
43/0
48.00
3.40
03.1
20/0
03.5
00.00
100
405
00.03
0000.003
125
25.3
1640
1029
10770000
1009
1030
1300
139200
100
030
13
0.1
1.27
00.
1.96
130
10.95
40.
11.
14.3%
Varta
121
10.3
12:9..15 1.00
16.3
10.22
004
12.3
0225
1000
1225
1235.00
162
05.35
1.4
1015
2670
30071000
191
112
102
B.5 Additional Classification Results
In addition to the main experiments, we also performed many classification experiments with setup A various datasets, the results shown in Table 5 Especially, when fine-tuning SwinT on CUB200, Pets, and CIFAR-100 100, applying HOSVD to the last four layers (maintaining the rest of the architecture frozen) consumes less memory than fully fine-tuning the last two layers. This happens because the variance of the activation maps is concentrated in few components: for the same e, K will be smaller.
C Limitation
While HOSVD is effective in compressing the forward signal in DNNs for gradient estimation during backpropagation, it is just one among possible choices when dealing with tensor decomposition. Indeed, more variants of tensor decomposition methods such as GKPD or STP could have been used and evaluated and could have led to larger memory and computational gains. Our work opens the door to further ex explorations in such direction.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We develop the theoretical aspects of tensor decomposition applied to activa- tions Sec. (construction, practical implementation, complexity, and error bounding). In Sec. 4 we provide experimental results on wide variety of popular tasks, architectures, and datasets, while also exploring the combined effect various hyper- pyper-parameters.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the work are discussed in the dedicated section of the pa- per(Sec. C
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Sec. 3.1 we develop the formulas necessary to perform backpropagation in convolutional network using the chain rule. In Sec. 3.2 we provide explicit definition of the tools used this papers. The complete proof for Sec. 3.3 is provided in the supplemental ma- terials(Sec. A..3). In Sec. 3.4 we evaluate the resulting memory required, the computational complexity of our method, and the error introduced in the gradient computation.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Algorithmic details are developped throughout Code to reproduce results is included in the supplemental materials, alongside instructions how it. The experimental setup is described briefly in Sec. and more exhaustive explicit details are provided in the supplemental materials B.2
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: In footnote on the first page of the paper, the exact code used to collect the experimental results is provided in an open GitHub repository, along with instructions on how run the code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Such details are discussed in Sec. and developed further in the corresponding section of the supplementary material (Sec. B.2
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
22

---

Answer: [Yes]
Justification: Given the computational cost of the experiments, we have performed all our experiments (even reproducing those from the literature) under the same conditions, but we did not provide confidence intervals for all of them. In the supplementary material we show some results providing confidence intervals, and observing a very small variance between the results obtained from different seeds.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
We detail the material used in Sec. and the time of execution in the supple- mentary materials.
Justification:
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Research conducted in this paper is conform with NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our method is meant to enable on-device learning and we discuss in the
introduction about possible impacts of such solution in our lives (increased privacy and security, reduced energy consumption and network overload, ...) (Sec. D. Nevertheless, specific section not devoted given that this work very general and not specific to some application.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The proposed approach is generalist compression algorithm tailored for on-device learning, thus, does not fall in that category.
Justification:
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the existing datasets, models, and code used presented in Sec. and credit is given to the respective creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All the details about the new assets introduced are provided in the paper. At submission time the source code is not provided, but we commit to release it (alongside a detailed documentation) upon acceptance of the paper.
14. Crowdsourcing and Research with Human Subjects
23

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
24


Reviews:
[{"review_id": "VGgMEprmtt", "reviewer": "Reviewer_f1cQ", "summary": "Tensor low-rank decomposition is used to compress the backpropagation process of full connection and convolution in neural networks. The main process is to decompose input activation tensors into Tucker structures using HOSVD algorithm and truncate subtensors at each mode. Experimental result find an efficient trade-off between the desired explained variance and compression rates.", "questions": "(1) In formula 12, how to use the discrete Fourier transform to get $\\tilde{I} =\\varepsilon I$? What does $uI$in the denominator mean?\n\n(2) In formula 16, please give the definitions of $d, h’,w’$. \n\n(3) The methods mentioned in the related work are not compared with the proposed methods.", "limitations": "If the contraction between matrix and tensor is replaced by 1*1 convolution, then the proposed method should also be compared with CP decomposition, tensor train decomposition and other related methods.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) Compute the approximated weight derivatives without reconstructing the activations, through the successive computation of simpler convolutions.\n\n(2) The experiments on trade-off between the desired explained variance and compression rates are sufficient.\n\n(3) Experimental results obtained on mainstream architectures and tasks show that it has a Pareto advantage over other comparison schemes in terms of the trade-off between accuracy and memory footprint.", "weaknesses": "(1) There are a lot of confusing expressions on the formula.\n\n(2) In formula 8, How to distinguish $U^{(k_j)}$s when the size retained after truncation on two modes is the same?\n\n(3) The symbol $]$ in formula 11 is in the wrong position, and $\\frac{\\partial A_{i+1}}{\\partial W_i}$ should be $\\frac{\\partial L}{\\partial A_{i+1}}$?\n\n(4) In Sec3.4 $\\frac{\\partial L}{\\partial A_{i+1}}=\\Delta Y$, but in appendix A.2 it expressed as \\Delta O.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1: Confusing formulas]**\nThanks for your feedback, we are taking into account your comments to improve the readability. \n\n\n**[W2: Distinguishing $U^{(k_j)}$s]**\nWe acknowledge the notation issue, taking action to fix it. Specifically, we will change the notation for factor matrices introduced in eq (6) from $U^{(j)}$ to $U_j$. The truncated version will thus become $U_j^{(k_j)}$. This will allow for distinction of the $U^{(k_j)}$ after truncation.\n\n**[W3: Mistakes in formula 11]**\nThank you for your remarks, they are both correct. We will update the paper accordingly.\n\n**[W4: Inconsistent notation]**\nWe agree on the remark, we will correct it in the appendix to ensure consistent notations in the final version of the paper.\n\n**[Q1: Fourier transform formula 12]**\nIn formula 12, $vI$ in the denominator should be replaced by $\\varepsilon I$ (sorry for our typo). By construction, $\\tilde{I}$ contains the $k$ principal components of $I$ such that it explains for a percentage $\\varepsilon$ of the variance of $I$. By switching to the discrete frequency domain, $I$ and $\\tilde{I}$ become power spectrums where $\\tilde{I}$ is composed of $k$ components originating from $I$. The sum of these components amount to a fraction of the power of the original signal which by construction is $\\varepsilon$, thus $\\tilde{I}=\\varepsilon I$.\n\n**[Q2: Definition of $d,h',w'$]**\nWe are sorry, these definitions were missing in the paper. We have $d\\in [1, D]$, $h'\\in [1, H']$ and $w'\\in [1, W']$ where $D$ is the weight kernel dimension, $H'$ and $W'$ are the output activation height and width respectively.\n\n**[Q3: Comparison with related works methods]**\nPlease refer to answer #1 of *general answers*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkOSUEWlIG", "reviewer": "Reviewer_HEKd", "summary": "[Editing to reflect my score increase from 6 to 7 after the author discussion phase.]\n\nThe authors tackle the problem of memory consumption due to needing to keep realized activation tensors available between their use in the forwards pass and backwards propagation in training neural networks.  They propose the use of tensor decomposition, specifically SVD and higher-order SVD, to compress activation maps and show that the compressed activations can be used in backpropagation without first decompressing.  An error analysis is performed, and experimental results confirm the hypothesis that significant compression can be applied while maintaining significant explained variance.  To validate the method's success on real workloads, the authors apply it to image classification and semantic segmentation tasks, fine-tuning pre-trained models for both tasks, as well as training from scratch for image classification.", "questions": "1. Is there a reason that results for fine-tuning all layers were omitted?\n2. Why is SVD superior to HOSVD in some results (line 288)?\n3. Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)?\n4. What impact does the compression process have on the training speed?", "limitations": "The authors have addressed the limitations (in the appendix).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality**\nI believe this is a novel application of tensor decomposition, a well-known technique previously applied to weight compression.  Adaptively sizing the decomposed tensor to maximize retained information in the activation maps is an advancement over past work.\n\n**Quality**\nThe authors have performed a reasonable set of experiments to show the technique works as intended: model quality when activations are subject to tensor decomposition for backpropagation remain competitive for image classification tasks and semantic segmentation tasks.  Further, the authors have reported the memory consumption, proving that it is reduced significantly enough to satisfy constraints of edge devices in many cases.  I greatly appreciate the time spent in the background, as well as the seemingly offhand note that any error induced by these decompositions will be limited to that layer's weight gradients and does not accumulate deeper into the network.\n\n**Clarity**\nI generally found the manuscript easy to follow: the overall organization was great, and the method itself was explained adequately.\n\n**Significance**\nThe intended use case aside, I think even datacenter users may be interested in this technique, as many of today's latest networks are simply huge.  Reducing the storage costs for activations between forward and backwards passes could reduce the parallelism needed, and in turn the overhead of communications in distributed training.  I'd encourage the authors to consider this angle and apply the technique to, for example, large language models.", "weaknesses": "**Originality**\nOnly one piece of past work comes to mind that was omitted: Rhu et al., \"Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks\" (HPCA 2018) compresses activation maps, using their existing sparsity, to reduce the cost of offloading in order to accelerate training.  While novel hardware was designed to keep up with compute throughput, if compression is the only goal, then this could simply be performed in software (like the submission's tensor decomposition).\n\n**Quality**\nA missing piece is the cost of compression: what impact does this have on the training speed?  This is related to the claim on line 59 that \"gradient computation and parameter updating are considerably more expensive than the forward pass,\" which isn't obvious to me.  At a high level, each of the three calculations (forward propagation, gradient calculation, weight update calculation) are roughly equivalent in cost, and applying the weight update is very simple.  Increasing the cost of any one of the phases would seem to be as painful as any of the others.  Another claim that stood out, given the stated goal of \"reducing the memory required for backpropagation\" (line 72) is that the \"main challenge limiting the feasibility of on-device learning lies in the computational cost of backpropagation\" (line 58).  I'd also question the claim that \"we can assume that the networks considered are already [weight] compressed\" (line 159) - while it's extensively studied, it's not a solved problem.  It's fine to not tackle this particular problem, but stating that it's assumed to not be an issue is misleading.\n\nSome of the results deserve some extra discussion.  Do the authors have a hypothesis about why SVD is superior to HOSVD (line 288)?  Why does MobileNetV2's Vanilla BP results get worse when including 4 layers compared to only 2 (Table 2)?\n\nI have not seen CIFAR treated as a \"downstream task\" of ImageNet-1k before; it's typically just a simpler data set used, recently, as a proving ground before testing on larger data sets.  Is it common to fine-tune a network trained on ImageNet-1k on CIFAR10/100?\n\nGiven the significant memory savings of the technique, I'd have liked to see how it performs when fine-tuning all layers, not just a subset.  Is there a reason this result wasn't gathered?\n\n**Clarity**\nI've had to make an assumption about the behavior of the method: during forward propagation, I assume the *uncompressed* activations are used, and only compressed when writing to memory to be used later in the backwards pass.  (If this is not the case, then errors *will* compound.)\n\nFigure 2 shows speedups and space gains, but without indicating what, exactly, is speeding up or seeing a reduction in memory space.  I assume, but am not sure, that it is for a single weight gradient calculation (of a size noted in the figure's legend and axes).\n\nThe wording of Figure 3's caption is confusing: I do not understand the meaning behind \"Mbv2 fine-tuned on Cifar10 at initialization.\"  \n\nNone of the networks used in the experimental results have references attached.  For example, MCUNet is prominently used in Figure 4, but I couldn't find either a description of this network or a reference to it in the text.  It looks likely that it is in reference [24], but this only has a passing citation in the related work section.\n\nThere are some typos:\n\n- Line 223: \"derives\" -> \"derivatives\"\n\n- Line 234: the range of explained variance uses two open brackets.\n\n- Reference [48] is listed with the incorrect year: 2024 should be 2023.\n\n\n**Significance**\nThere are several ways that the results would be more compelling:\n\n- Evaluation on more (difficult) tasks, like language modeling or object detection.\n\n- Application to more recent or larger networks.\n\n- More closely matching the densely trained or fine-tuned baselines.  As this is initial work in this area, the results shouldn't be dismissed, but there's still a non-trivial gap between the tensor-decomposed results and what the networks are capable of learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1 - Originality: Missing reference]** In their work Rhu et al. exploit inherent activation sparsity for compression, resulting in low memory usage and accelerated training. This is a very relevant paper with respect to our research and we will cite it in our paper, thank you for pointing it out. It is worth noting that we cite in our related works section Kurtz et al. work on activation sparsity [22]. In their work they mention Rhu et al. work as a reference and build upon their findings to induce improved activation sparsity.\n\n**[W2.1 - Quality & Q4: Cost of compression]** Please refer to answer #3 of *general answers*.\n\n**[W2.2 - Quality: Backpropagation vs Forward complexity]** Thank you for your feedback. \n\nConsider convolutional layer $i$ with:\n- An activation map $\\mathcal{A}_i \\in \\mathbb{R}^{B\\times C\\times H\\times W}$, \n- A weight tensor $\\mathcal{W}_i \\in \\mathbb{R}^{C'\\times C\\times D\\times D}$, \n- Produces an output $\\mathcal{A}_{i+1} \\in \\mathbb{R}^{B\\times C'\\times H'\\times W'}$. \n\nDuring the training process of a convolutional deep learning model:\n- There is one convolution operation in each forward pass with the computational complexity is $\\mathcal{O}_{\\text{time}}(D^2CC'BH'W')$.\n- While, in each backward pass, there are two convolution operations, corresponding to formulas (2) and (3) in our paper and one weight update operation, having computational complexities of:\n    - $\\mathcal{O}_{\\text{time}}(D^2CC'BH'W')$\n    - $\\mathcal{O}_{\\text{time}}(D^2CC'BHW)$\n    - $\\mathcal{O}_{\\text{time}}(D^2CC')$\n\n$\\Rightarrow$ Therefore, we can calculate the computational complexity ratio between the forward and backward passes. This ratio is always less than 1, indicating that the backward pass is more complex. This supports our claims, and we will include this information in our supplementary materials for clarity.\n\n**[W2.3 - Quality: Weight compression SOTA]** We understand how our claim might be misleading and we will change line 158-159 to: “Weight compression is an extensively explored matter for network acceleration and we do not intend to further this area of research in this work.”\n\n**[W3.1 - Quality & Q2: SVD vs HOSVD]**\nPlease refer to answer #2 of *general answers*.\n\n**[W3.2 - Quality & Q3: MbV2 Vanilla BP result]** In our paper, we report results extracted from the paper Efficient On-device Training via Gradient Filtering to compare with our method. However, we have recently reproduced their experiments using the same setup and obtained different, more reasonable results. Specifically, for Vanilla BP with 2 layers, the accuracy is now 62.6%, while it is 65.8% for 4 layers.\n\n**[W3.3 - Quality: C10/100 as downstream tasks]** We used CIFAR10/100 as downstream tasks in a similar fashion to what has been introduced in [49] as they also feature these datasets in their results. As an additional note, Han et al. use CIFAR10/100 as downstream tasks in their works [2] and [25]. \n\n**[W3.4 - Quality & Q1: Fine-tuning all layers]** We will provide the complete set of results and will include them in the final version of the paper. The full results are presented in Figure 1 and Table 1 in the PDF rebuttal file.\n\n**[W4.1 - Clarity: Method Clarity]** You are correct. This can be inferred from Figure 1 of our paper. However we agree that it has not been made sufficiently clear in the description of the method. In Section 3.3 \"Backpropagation with Compressed Activation,\" we will include the following paragraph: \"Figure 1 illustrates our method. At training time, forward pass proceeds as usual with the only difference that instead of storing full activation maps in memory, we propose to store their principal components, which are the products of the decomposition process. During the backward pass, the principal components are retrieved from memory and used for calculations as described in formulas (18) to (22).”.\n\n**[W4.2 - Clarity: Figure 2 clarification]** Yes, it is for a single layer. We will make the graph easier to understand by adding more detailed descriptions.\n\n**[W4.3 - Clarity: Clarity in figure 3]** In this figure, four components are essential: the fine-tuning dataset, the network considered, the layer within the network from which we extract the explained variance curves, and the epoch at which we extract the curves. The “at initialization” means that these plots are extracted at epoch 0. We will change the caption of the figure to better reflect that aspect.\n\n**[W4.4 - Clarity: Networks references]** We use the same networks as in [49]. We will add proper referencing to all the networks considered in this paper.\n\n**[W4.5 - Clarity: There are some typos]** Thank you for your remark, we will fix them in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tNLgj19lWp", "reviewer": "Reviewer_dqZf", "summary": "This paper proposes a method to compress activation maps in deep neural networks using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and Higher Order SVD (HOSVD). The goal is to reduce memory requirements during backpropagation, enabling on-device learning for resource-constrained environments. The authors provide theoretical analysis of their method's impact on memory usage, computational complexity, and error bounds. They demonstrate the effectiveness of their approach through experiments on various tasks, architectures, and datasets.", "questions": "- How does the proposed method perform on more diverse and complex neural network architectures, such as transformers?\n- What are the practical challenges and considerations when deploying this method on real-world edge devices with strict latency and power constraints?\n- How is the appropriate rank for decomposition selected, and what guidelines can be provided for tuning this parameter effectively?", "limitations": "- Specific to Certain Architectures: The experiments focus mainly on specific neural network architectures and tasks. It is unclear how well the method generalizes to other types of models and applications.\n- Dependence on Hyperparameters: The effectiveness of the compression is highly dependent on the chosen hyperparameters for the decomposition, which may require extensive tuning.\n- Resource Requirements for Decomposition: While the method reduces memory usage during training, the initial decomposition itself can be computationally intensive and may require powerful hardware.\n- Practical Usability Concerns: Despite significant memory reduction, the method results in considerable accuracy drops, raising concerns about its practical usability. Vanilla backpropagation, while using more memory, generally maintains higher performance and might be a more realistic approach for actual deployment.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novelty: The use of tensor decomposition for compressing activation maps is a novel approach that addresses a significant bottleneck in neural network training.\n- Efficiency: The method significantly reduces memory usage during backpropagation, which is crucial for deploying deep learning models on resource-constrained edge devices.\n- Theoretical Support: The paper provides  theoretical background for the proposed method, including error analysis and guarantees of minimal information loss.", "weaknesses": "- Severe performance degradation relative to memory reduction\n- Complexity of Implementation: Implementing tensor decomposition techniques like HOSVD can be complex and may require specialized knowledge, potentially limiting its adoption.\n- Lack of Detailed Rank Selection: The paper does not provide sufficient explanation on how the appropriate rank for decomposition is selected, which is crucial for balancing memory reduction and accuracy.\n- Inconsistent and contradictory results: SVD frequently outperforms HOSVD, contradicting the authors' hypothesis and lacking adequate explanation.\n- Limited comparison with other activation map compression techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1: Severe performance degradation relative to memory reduction]**\nWe are focusing on performing direct training on edge devices with extremely limited memory, so it is worth considering trading off accuracy for memory savings at an incredible rate.\n\n**[W2: Implementation complexity]**\nWe have the implementation, and we commit to publish it as open source upon acceptance of the paper.\n\n**[W3 & Q3: Lack of detailed rank selection]**\nPlease refer to answer #4 of *general answers*.\n\n**[W4: SVD vs HOSVD]**\nPlease refer to answer #2 of *general answers*.\n\n**[W5: Activation map compression comparison]**\nPlease refer to answer #1 of *general answers*.\n\n**[Q1: Alternative architectures]**\nWe provide results on SwinT architecture in table 5 in supplementary materials sec B.2.1. For every downstream task, HOSVD allows for incredible compression rates (up to 40 times less memory than vanilla backpropagation on 2 layers and 20 times less than SVD). Regarding accuracy, HOSVD is competitive with SVD and depending on the dataset, can be slightly below vanilla BP for the same number of fine-tuned layers. These results are on par with the results obtained with the CNN architectures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UwsCuCkoiJ", "reviewer": "Reviewer_WgAz", "summary": "The paper addresses the challenge of on-device training for deep learning models, particularly focusing on the memory bottleneck caused by the storage of activation maps during backpropagation. The authors propose a method to compress activation maps using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and its tensor variant, High-Order Singular Value Decomposition (HOSVD).", "questions": "1. LOMO[1] fused gradient computation and parameter update in one step to minimize the size of gradient tensors. Compared to this approach, what are the advantages of your method?\n2. Would using quantization methods instead of compression methods produce better results?\n\n\n[1] Lv, Kai, et al. \"Full parameter fine-tuning for large language models with limited resources.\" arXiv preprint arXiv:2306.09782 (2023).", "limitations": "None", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Using tensor decomposition for activation map compression to address a bottleneck in on-device training.\n2. The paper provides a solid theoretical foundation, including error analysis and guarantees for convergence.", "weaknesses": "1. Using SVD or tensor decomposition to compress activation maps incurs significant computational overhead. This approach of trading computational complexity for space complexity is questionable, especially considering that the computational capacity of embedded devices is usually also limited. Moreover, this compression may introduce additional information loss, potentially affecting the model's performance.\n2. For SGD, recalculating activations instead of storing them is a viable method to save memory. Given that both methods trade computation time for storage space, why use low-rank compression which may potentially affect model performance?\n3. With limited computational resources, why not search for a more appropriate model architecture using NAS?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1.1: Compression cost]** Please refer to answer #3 of *General answers*.\n\n**[W1.2: Introduced compression loss]** Please refer to answer #4 of *General answers*.\n\n**[W2: Comparison with activation checkpointing]**  As evidenced in Figure 1 of the PDF rebuttal file, the required memory to store the compressed activations of all the layers of an MCUNet is far below the full activation memory of the last layer of that same network. This is especially meaningfull as that specific layer is one of the least costly in terms of activation memory compared to other layers in the network. In that sense, although activation checkpointing results in an equivalent accuracy to that of vanilla BP, its memory consumption is about one to two orders of magnitude above low-rank compression applied to all the layers. This truly underlines the idea that the accuracy tradeoff of low-rank compression is viable in the case of training on edge devices with extremely limited memory.\n\n**[W3 & Q2: Comparison with NAS and quantization]** These research directions are orthogonal to what we propose. We intend to study the effect of activation compression through tensor decomposition in a vacuum. Future research will lead us to combine this method with other compression strategies such as quantization or memory-efficient networks to further validate its generalizability. Furthermore, MCUNet, a model that we use for our experiments, was designed through NAS [24].\n\n**[Q1: Comparison with LOMO]**  As exposed in eq. (2) of our paper, activations are necessary to compute weight derivatives and they must be stored in CPU memory during the forward pass, meaning that our method could still be relevant in conjunction with LOMO to further reduce CPU and GPU memory usage. LOMO combines gradient computation and weight update in a single step instead of computing all gradients then updating all weights which allows for reduced GPU memory usage. In that sense, LOMO and low-rank activation compression adress different aspects of BP acceleration and memory reduction which means that they could be combined in an unified framework for improved savings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First of all, we would like to thank you and express our appreciation for the time and effort you have invested in reviewing our work. We are especially grateful for the highlights regarding the novelty of the proposed method (HEKd, dqZf), the theoretical groundings (HEKd, dqZf, WgAz) and the experimental results showing its efficiency (f1cQ, HEKd). Below, we have provided detailed responses to your questions and concerns, together with the PDF Rebuttal file containing additional figures and a table. We look forward to further feedback during the discussion period. Thank you once again for your valuable input.

**Note:**
- Wi: Weakness number i
- Qi: Question number i

# General answers

**#1: Related works comparison (answer for Q3 of f1cQ and W5 of dqzf)**

Regarding tensor decomposition, methods introduced in the related works section either focus on weight compression for accelerated inference and lighter networks [13, 50, 52], or on gradient compression for accelerated backpropagation [48] whereas our approach compresses activation maps for efficient and lightweight backpropagation, thus making them not directly comparable.
In a similar fashion most current Activation Map Compression strategies focus on accelerated inference [10, 12, 22]. [9] apply their method for large GNN training. We considered this setup to be too far remote from our own experimental setup and we leave comparison with this work for future research.
In the context of on-device learning, we compared our technique to the state-of-the-art method, Gradient Filtering [49]. Other methods introduced in this paragraph focus on efficient subgraph selection [23, 25, 33] which is orthogonal to our proposition and thus compatible for joint implementation. We leave this research area for future work.

**#2: Discussion on results (answer for W3.1 - Q2 of HEKd and W4 of dqZf)**

 - **Discussion**: SVD is only slightly better than HOSVD in terms of accuracy but is significantly worse in terms of memory usage. Specifically, Figure 4 in our paper shows the Pareto curves for SVD, HOSVD, and Gradient Filter. It is clear that HOSVD has a significantly better Pareto curve. To be clearer, we will update Figure 4 by Figure 1 in the PDF Rebuttal file showing the full pareto curve of HOSVD and SVD.
- **Explanation**: HOSVD essentially performs SVD on each mode of a tensor. Specifically, applying HOSVD to a tensor $\mathcal{A} \in \mathbb{R}^{B \times C \times H \times W}$ involves performing SVD on four unfolded versions of $\mathcal{A}$ with sizes $B \times CHW$, $C \times BHW$, $H \times BCW$, and $W \times BCH$, respectively. In contrast, SVD is applied only to the first mode, which is a tensor of size $B \times CHW$ (lines 180 - 181 in our paper). Thus, with SVD, the explained variance threshold is applied to retain an $\varepsilon$ fraction of the information in the first mode, **without affecting other modes**, meaning that the three remaining modes correspond to the raw uncompressed activation. In comparison, HOSVD allows us to retain an $\varepsilon$ fraction of the information **across all modes**. This explains why, in the same context, SVD generally offers higher accuracy but is more memory-intensive, while HOSVD is less accurate but more memory-efficient.

**#3 Compression overhead analysis (Answer for W2.1 - Q4 of HEKd and W1.1 of WgAz)**

In section A3 in supplementary, we will add description for overhead as follows:

"
**A.3 Details of Overhead, Complexity and Speed-up Computations**

Following the notation in our paper, we can compute the overhead, space complexity and the speed-up of HOSVD.

**Overhead:** During the forward pass, while Vanilla BP does not perform decomposition, HOSVD does. Therefore, the overhead of the training process is the computational complexity of HOSVD, which can be calculated as follows:

The computational complexity of SVD for a matrix of size $m \times n$ with $m \geq n$ is $\mathcal{O}_{\text{time}}(m^2n)$. HOSVD essentially performs SVD on each mode of the tensor. During the forward pass, at each convolutional layer, given the activation map of size $BCHW$, HOSVD involves performing SVD on four matrices of sizes $B\times CHW$, $C\times BHW$, $H \times BCW$, and $W\times BCH$. Therefore, the computational complexity for decomposition in the forward pass is:

$$ \begin{equation*}\mathcal{O}_{\text{time}}\left( \max(B, CHW)^2 \times \min(B, CHW) + \max(C, BHW)^2 \times \min(C, BHW) + \max(H, BCW)^2 \times \min(H, BCW) + \max(W, BCH)^2 \times \min(W, BCH)  \right) \end{equation*}$$

**Space complexity:** (refer to $complexity$ - formula (23) in our paper)

**Speedup:** (refer to $speedup$ - formula (24) in our paper)
"

Additionally, in Figure 2 in our paper we will add another sub-figure showing the predicted overhead as shown in Figure 3c of the PDF Rebuttal file.

**#4: Details on rank selection (answer for W3 - Q3 of dqZf and W1.2 of WgAz)**

We do not focus on rank but rather on how much information is retained after projecting tensors into a lower rank space, and we do that by manipulating the explained variance threshold $\varepsilon$, which is 0.8 or 0.9 in our experiments. We **are the first method** allowing for explicit control over the information loss. In lines 185 - 187, we wrote "The larger the explained variance, the closer $\tilde{A}$ will be to $A$, intuitively allowing for better estimation when performing backpropagation." We use the explained variance threshold $\varepsilon$ to control how much information can be retained after compression. This value goes from 0 (loss of all information) to 1 (retention of all information). Moreover, as shown in Figure 2c in our paper and Figure 2d in the PDF Rebuttal file , as $\varepsilon$ is closer to 1, $SNR$ becomes bigger resulting in improved performance.

Meta-review: The authors submitted rebuttals, and all the reviewers here acknowledged them and had discussions with the authors.
I requested clarifications about their comments, and some of them responded.
Variance of the ratings was somewhat high as one of the reviewers (Reviewer HEKd) raised their score to 7, following multiple discussions with the authors. As they suggested, this is an interesting tensor decomposition research with great clarity and writing in general. They also value originality and its potential impact and interest of data center users.

I should clarify that I decided to put my focus on Reviewers HEKd, WgAz, and dqZf as I found none of the weaknesses claimed by Reviewer f1cQ's concrete, but more like questions. The reviewer argued that they had found authors' rebuttal satisfactory, but didn't provide any justifications (including their initial review) to support the rating.

Reviewer WgAz gave the lowest rating among the four reviewers, and key weaknesses they claimed in the initial review and follow-up are 
1.  "why not search for a more appropriate model architecture using NAS?"
2.  "Rebuttal was not convincing because the experiments are on small datasets with small models. [1] is more efficient and doesn't affect model performance ". 

For 1, I saw that the reviewer pushed NAS through interactions, but NAS is a resource-intensive approach. This work discusses on-device learning, and I believe that not using NAS cannot be a weakness.

For 2, I agree with the reviewer to the point in general, but again this work is about on-device learning. I also checked [1], but it doesn't discuss model performance (accuracy) but FLOPs, latency, and memory consumption.

Reviewer dqZf pointed out multiple weaknesses of this work, and the authors rebuttal addressed their concerns. But the reviewer didn't explain rationale of the rating 4. I requested a follow-up from the reviewer, but the reviewer did not answer my question.
Having reviewed the authors rebuttal, I found the rebuttal satisfactory. E.g. (1), "Severe performance degradation relative to memory reduction" should be observed as part of trade-off discussions, which also shows better trade-off by the proposed method than those of the baselines. E.g. (2), "Limited comparison with other activation map compression techniques" <- The authors provided additional experimental results using Swin Transformer model (a significantly different architecture) in the supplementary material, which seems like a reasonable choice to me. Overall, it seems to me that the reviewer is looking for reasons to reject.

Considering all these points, I found my position literally on on fence, but towards acceptance. My recommendation is based on the fact that the review by Reviewer HEKd shared very concrete suggestions and relevant concerns, and the authors successfully addressed the concerns through the discussions, thus I would like to recommend acceptance for this work.


[Suggestions]
While my recommendation is acceptance, I still highly recommend that the authors take into account all review comments and address their concerns in the camera-ready (e.g., clarifying motivation of this work). 
- Replace "AI model" with "deep learning model". AI is not the right term in this scientific paper, but just a distraction.
- My recommendation is partly conditioned on the authors' code release.
Decision: Accept (poster)