Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
Authors: Le-Trung Nguyen
Aël Quélennec
Van-Tam Nguyen
Enzo Tartaglione
Venue: ICML 2025 poster (2025)
Keywords: Deep Learning
Computer Vision
Compression
Low rank
Primary area: deep_learning->algorithms
TL;DR: In this paper, we introduce a novel shortcut method that reduces the activation map to a low-rank space, enabling the practical implementation of on-device learning.

Abstract: On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to $120.09\times$ compared to vanilla training, while also reducing overall training FLOPs up to $1.86\times$ when evaluated on traditional benchmarks.

Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device
Learning
Le-Trung Nguyen
Aël Quélennec'
Van-Tam Nguyen
Enzo Tartaglione
Abstract
B
Before training
On-device learning has emerged promising direction for AI development, particularly be- cause of its potential to reduce latency issues and mitigate privacy risks associated with device- server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still repre- sent major challenges for its deployment. Draw- ing previous studies on low-rank decompo- sition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to 120.09x compared vanilla training, while also reducing overall training FLOPs up 1.86x when evaluated on traditional benchmarks. The code is available at https://itthu..cmm/ TrungNeuyen/ICMM122555553
Ranks Selection
R.
A
9
Memory
ac a.A.
ac
Layer
Forward
Backward
Low-rank
ASI
Low-rank calculation
Figure I. Trajectory of activation maps during single training step of a deep learning model using ASI.
1. Introduction
nication between data and servers, and thereby enhancing user data security. This method proves valuable in diverse scenarios, particularly in environments without internet con- nectivity, such as autonomous vehicles (Dhar et al., 2021). It can also enable smartphones to learn from local user data, allowing personalized adaptation to individual usage pat- terns (Hard al., 2018).
Recent significant advances deep learning have allowed artificial intelligence to penetrate deeply into human activi- ties, enhancing the quality of life for users more than ever before. Deep learning has found applications in wide variety of fields, such as healthcare through tumor image de- tection (Hu et al., 2018) or finance through quantitative trad- ing (Ozbayoglu et al., 2020). Most notable are applications of large language models (LLMs), such as GPT (Radford al., 2018) or Gemini (Gemini Team et al., 2023).
Along with the substantial increase in the number of param- eters in deep learning models comes an exponential rise in resource demand to train and exploit such models. During training, the main algorithmic bottleneck with respect to energy and memory backpropagation. Although it has demonstrated its effectiveness in terms of performance, sur- passing many other methods such as Forward-Forwarl (Hin- ton, 2022) and PEPITA (Pau & Aymone, 2023), it is not yet "resource- "resource-friendly"" algorithm. Specifically, for each trained layer, backpropagation requires storing in memory the cor- responding activation maps during the forward pass (Gomez et al., 2017), which are then used to compute weight deriva- tives during the backward pass. As the number of model
In that regard, one of the current prominent research branches is on-device learning, where models are fine-tuned directly on edge devices, eliminating the need for commu-
'LTCI, Télécom Paris. Institut Polytechnique de Le-Trung Nguyen Enzo Tartaglione
Paris, France. Correspondence ce-trung.nguuyen
(etelecom-paateeee sareecoom-rris.... fr> <enzo Cenzoo.arttallonne@eeee @telecom-
-tllccom-paris.ff>> frris.fr>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond Low-rank Decomposition
parameters increases, the memory required store these pa- rameters and activations during training grows accordingly.
activation perplexity that accounts for the impact of the low-rank decomposition during pook-propagttion.
The increase in resource demands leads to longer training times and higher demands for data centers, resulting in greater carbon footprint emissions (Strubell et al., 2020; Thompson et al., 2020; Patterson et al., 2021). This critical issue demands immediate attention. The techniques used to train models directly on devices offer a promising solution, as their inherently low energy consumption helps address these challenges (Alajlan & Ibrahim, 2022).
We introduce the use of single subspace iteration as a replacement for traditional compression methods, aim- ing to balance accuracy and computational efficiency (Sec. 3.4), translating into a consistent reduction in computation time (Sec. 3.5).
We demonstrate the effectiveness of our method through various experiments, including ImageNet- kk, and performing real measurements on a resource- limited device like a Raspberry Pi (Sec. 4).
Various research directions have emerged to overcome back- propagation inefficiencies and enable learning directly on devices. A pioneering study, initiated by Lin el al., demon- strated that fine-tuning a predefined subnetwork of deep model under a strict memory constraint of just 256KB while maintaining acceptable performance is entirely feasible (Lin et al., 2022). In similar fashion, Quélennec et al. (2024) propose to dynamically fine-tune the most suitable subnet- work at each training step rather than relying on a fixed one, allowing for improved accuracy while maintaining mem- ory usage under constraint. Extending beyond the scope of on-device learning, many other studies have been proposed to optimize the training process by reducing the number of trained parameters, notably LoRA (Hu et al., 2021) and its variants. While these approaches significantly reduce the number of parameters that need updating during training, they all neglect activation memory constraints. To solve this issue, Nguyen et al. (2024) applies low-rank strategies to compress activation maps for greatly reduced mem- ory footprint with a limited performance drop. Although promising, the actual on-device feasibility of their method is compromised by the prohibitive computational cost of per- forming the activation decomposition hetween each training epoch, as well as their lack of control over memory con- straints.
2. Related Works
In this section, we will first present some recent works that propose low-rank decomposition (Sec. 2.1). Although conceptualized for reducing the dimensionality of the pa- rameters representation, these set methods and tools enable the design of strategies for activation map decomposition. Next, in Sec. 2.2, we discuss previous works that directly address target problem and highlight their limitations.
2.1. Low-rank Decomposition for Weight Compression
Low-rank approximation methods aimed at reducing the number of parameters required for training have demon- strated significant effectiveness with approaches like LoRA (Hu et al., 2021). By training a low-rank adapter, the number of trainable parameters is reduced by up to 10,000 without introducing additional inference latency.
Zhang et al. (2023) have shown that assigning the same rank to all fine-tuned layers is not an advisable strategy, as differ- ent layers contribute differently to the model's performance. To address this issue, they proposed AdaLoRA, which dy- namically allocates suitable ranks to each fine-tuned layer at each training step while adhering to a predefined parameter budget. Subsequent studies, such as DyLoRA (Valipour et al., 2022), SaLoRA (Hu et al., 2023), SoRA (Ding et al., 2023), and ALoRA (Liu et al., 2024), have similarly at- tempted to solve this problem by adaptively distributing ranks based on the importance of each layer to the per- formance of the model. A practical and effective solution emerged with the introduction of ASVD (Yuan et al., 2023). In this work, they propose analyzing the sensitivity of each layer to different compression rates (i.e., the memory ratio between the low-rank version and the original tensor). Be- fore training, they performed binary search to determine the most suitable rank for each layer that satisfies a prede- fined budget, inspecting the distribution of the generated activations and compressing the parameters to minimize the induced distributional shift on the produced output.
Inspired by previous studies on the stability of activation maps during training (Virmaux & Scaman, 2018), pro- pose ASI (Activation Subspace Iteration) to address the issue of reducing computational complexity while training the model (Fig. 1). By identifying the most suitable decom- position rank and projector once before training, we can effectively gain both in terms of memory occupation and computation. Leveraging gradient computation in the com- pressed space, we also able to reduce backpropagation computation, as showcased when deploying our solution on small devices like a RaspBerry Pi 5.
The key contributions of our work are below.
We propose a rank selection strategy to determine the most suitable ranks for each fine-tuned layer un- der a given budget constraint before training begins (Sec. 3.3). This is achieved by an estimation of the
All of these methods are specifically designed for weights
2

---

Beyond Low-rank Decomposition
10
10
Forward FLOPs
10
Backward FLOPs
10"
10
10²-
107-
10"-
10p-
10--
Pi P1a-22 m
Vanilla training HOSVD,
102-
Vanilaa training HOSVD,
16a
13.m
12
Nm [1:00
1.4
M1m m
10a
10
=
12,
20128
m 1.4
Di 115 m
D)
250
(1.4)
2. -2
m
Pum
Dim
128
m
[1.4]
258
Dim Pim
32
64
128
m
[]
256
6
5
124
15
(a)
(b)
(c)
(d)
Figure 2. For convolutional layer i with a single data batch of size B, (a) and (b) illustrate the predicted changes in the total number of FLOPs required to perform forward pass and backward pass, respectively, when comparing HOSVD, with vanilla training. (e) and (d) show the predicted changes in compression rate Re and speedup ratios Rs as functions of Ts,m, when comparing ASI with vanilla training.
and cannot be applied directly to activation maps, partially solving the problem of memory and computation reduction when deploying the targeted deep model on-device.
design of an activation perplexity measure, we estimate the best ranks only once before training to ensure strict constraint on memory consumption. Additionally, we lever- age single subspace iteration effectively estimate the low-rank components of activation maps and perform low- rank gradient calculations, significantly reducing the overall training cost.
2.2. Activation Map Compression
Recently, some works appeared in the context of activation map compression for convolutional neural networks, where activations pose big issue. Recognizing the severe memory bottleneck in backpropagation, Yang al. (2023) proposes gradient filtering as a possible solution. Given a predefined patch size, they applied pooling operators to it to generate approximated versions of activation maps and gradients. Their goal is to reduce resource consumption, specifically memory usage and FLOPs. However, relying solely on pooling techniques generally distorts the structured proper- ties of tensors in deep neural networks. Moreover, gradient approximation propagates eтror during the backward pass, leading to a significant performance drop as the number of fine-tuned layers increases.
3. Method
In this section, after recalling backpropagation and its im- pact memory consumption (Sec. 3.1) and low-rank de- composition through HOSVD (Sec. 3.2), we present how to perform rank selection in a compressed subspace for ac- tivation maps (Sec. 3.3) and how to apply to effective computation and memory savings (Sec. 3.4), followed by a computational complexity analysis (Sec. 3.5).
3.1. The Memory Bottleneck in Backpropagation
Let us assume we work with a deep convolutional ar- chitecture.' We note with the index of a layer in deep neural network, characterized by D, D, kernel W; RCxxGx,xxD, This layer processes input ten- sor A: E RBXCIxHXW. with C; channels and produces an output tensor A E [BB with C₁ chan- nels. Here, H, and W, represent the height and width of each input element, while HỤ and W/ denote the cor- responding dimensions of the output, and the variable B indicates the batch size. For convenience, here on we write D, (B, C. H,, W,) and DJ = (B, C₁ Hi, WT.
Later, Nguyen er al. proposed replacing pooling techniques with Higher-Orde Order Singular Value Decomposition (HOSVD) based on explained variance threshold e (HOSVDz), compressing only the activation maps (Nguyen et al., 2024). This method achieved impressive compression rates while preserving the structural properties of tensors and control- ling information loss caused by compression. However, HOSVD, incurs extremely high computational costs due to the HOSVD computation required at each training iteration, limiting its applicability due to high computation overheads. Besides, this approach lacks hard memory control (the plained variance is fixed, meaning that depending on the variance of the activations it can be more or less effective miser
Let us consider the backward pass at this layer, where the
'This is of interest as activation maps in convolutional layers are typically very large. The same conclusions we reach can be straightforwardly obtained for fully-connected layers.
Our proposed ASI goes beyond the limits posed by both Yang al. (2023) and Nguyen al. (2024): through the
3

---

Beyond Low-rank Decomposition
chain rule of backpropagation is computed follows:
Moreover, the majority of the activation map energy is focused on the first few principal components across all modes (Nguyen et al., 2024). Thus, small values of ri,m are sufficient to reconstruct A, from its low-rank components with acceptably low error.
ac ac
0A
OC
conv (Ai
(1)
aW;
8A oW;
0Aii1
for the parameter, while for the activations being
In the optimal scenario where Ti.m = 1, Fig. 2a and Fig. 2b depict the evolution in the total number of FLOPs between vanilla training and HOSVD, during both the forward and backward passes the size of the activation map increases. In the backward pass, the gradient can be computed di- rectly on the low-rank components, significantly enhancing computational efficiency compared to vanilla training. The computational speed during the backward pass improves as the rank ri.m decreases. However, performing HOSVD, at each training step introduces substantial computational over- head. This overhead grows exponentially with the activation map size, significantly increasing the forward pass FL.OPs. The computational savings from low-rank operations in the backward pass cannot compensate for this increased cost. A more detailed analysis can be found in Appendix A.2.
ac OC
0A
ac 0AA+1
conViull
0A, dA 0A:
(2)
with is the loss computed at the output of the model, conv()) is the Frobenius inner product, and rot(.) oc
180 rotation operation. It is evident that to compute aw.
and ac during the backward pass, A, and W; must be DA;
stored during the forward pass. This storage requirement is indeed the primary cause of memory bottleneck in back- propagation (Lin et al., 2022).
3.2. High-Order Singular Value Decomposition for Activation Maps Compression
In this section we recap how activation maps can be de- composed by HOSVD. For each mode m. the activation map Ai can be unfolded into Ai,m Rat. Xbs.", where Dro). For instance, in
In the next section, we will address the computational over- head problem by leveraging subspace iteration and a custom strategy for activation rank selection at specific layer's scale.
(di.m: bl.m) (Di.m:II the first mode (m = 1), unfolding A, results in Ar,1 E B.C.H.,W.
3.3. Subspace Iteration and Rank Selection
With optimal ranks r, No obtained for a given explained variance threshold E [0, 1], can perform truncated SVD with the corresponding rank rim E rj on A The final decomposed form reads:
To counter the computational complexity introduced by HOSVD, multiple choices could be made, among which subspace iteration (Stewart & Miller, 1975) poses itself as a possible candidate. Subspace iteration is a numerical technique commonly used for computing a subset of the dominant eigenvalues and corresponding eigenvectors of a large matrix (or tensor generalizations). has already been successfully applied deep learning in PowerSGD (Vogels et al., 2019) for gradient estimation in data data-parallel dis- tributed optimization, showcasing similar performance as singular value decomposition (SVD) but with significantly lower computational cost (Appendix A.
A x S X1 U4,1 X2 U12 X3 Us3 X1 Uss.
(3)
where S, E ITF X is the core tensor which can be viewed as a compressed version of A., Ü.. E Ra x are the factor matrices and their columns cor- respond to the principal components over the m mode. The m-mode product "Xm"" of -order tensor G e R P1 x2 xxx and a matrix B E R@xP= is a n -order tensor R E RR efixx""PPnn xQx PmxxPmmxxee Xa which can be expressed as:
The challenge when deploying such solution for activation map compression will be estimate the best rank selection, layer-by-layer. Some works are already attempting to attack this problem for parameter compression, passing through a perplexity estimator.
R Rm Po 15
P
Ip Pm P.Pm pm
(4)
Yuan et al. (2023) introduced Sensitivity- Truncation Rank Searching (STRS), a method to find adequate trun- cation rank to compress weights based on a metric called sensitivity, which is derived from the perplexity P. The latter serves metric to estimate the approximation error of the model when encountering new data and is defined as follows:
Therefore, during the forward pass of training step, in- stead of storing Q Opacce (II Dr.m.) elements of A,, the application of HOSVD, reduces this storage requirement to:
M II rim
Σ m
D Dim
(5)
P = ec
(6)
m=
4

---

Beyond Low-rank Decomposition
where L the loss of the model. Such perplexity can be eval- uated layer-by-layer, depending on how much such layer will be compressed- the goal there will be to compress the most while maintaining P the lowest. Yuan et al. evaluate P by feeding the model with short input sequences and various compression rates defined as the ratio between low-rank and full-rank tensor versions. Their analysis reveals that changes in layer's perplexity are inversely proportional to changes in compression rates, with each layer having its unique proportionality ratio. This ratio is denoted the layer's sensitivity measure: layers showing larger perplexity increases when compression rates decrease are considered more sensitive.
ASI
ASI without "warm start"
Topl Vulidation Accuracy (%)
86
75
70
65-
60-
55
10-2
10-1
10"
Peak Activation Memory (MB)
Perplexity for activation compression. In our particular case, compressing the activations only affects the parame- ters' derivatives and does not affect the output loss of the model. We thus re-define perplexity to activations PA, as
Figure 3. Performance of MCUNet pre-trained ImageNet and finetuned on CIFAR-10 with ASI.
ac the difference between aW, and BC to better represent the OW;
Rank Selection. We consider a memory budget constraint B for the activation memory across the set F of fine-tuned layers. Our goal here is to optimally select the ranks over each mode while adhering to the memory constraint.
error introduced when performing activation compression.
To estimate perplexity at different compression levels, we wish to perform HOSVD decomposition of four- dimensional tensors, resulting in an exponentially large number of rank combinations across each mode of each layer of the network. To overcome this practical limita- tion, we replace the set of compression rates with a set & of explained variance thresholds to create more equitable metric. We define & E (0, 1)=, corresponding to the num- ber of explained thresholds evaluated, each of which will result an efficient combination of ranks for all the layers considered. Our proposed perplexity estimation strategy follows two steps.
We define a recursive backtracking algorithm to identify the optimal combination of truncation ranks Rogt based on set of indices J E N n [1, E] across JF such that
Rupi, = RRi 1 T
(8)
FE ΣΣPs
F S
J = arg min
Pio
Σ
M,
B.
(9)
JEJ
where M; corresponds the resulting activation memory as expressed in (5) with r, Ropt-
Step 1. For each explained variance threshold Ej E E, we perform a forward pass with batch of pre-trained data. At layer i, we store both the uncompressed version of the acti- vation map (A:), and its low-rank variant (A:),, which is obtained by computing HOSVD and truncating the resulting components with respect Ej-
3.4. Activation Subspace Iteration (ASI)
Vogels et al. (2019) argue that between each training step, the gradient features only marginally change. Therefore, once properly selected, the projection subspace can be con- sidered stable over a sufficiently small number of steps. Intuitively, due to the lincarity of the subspace projection operator, the sequence steps in the same subspace (warm start) converges the eigenvector of the averaged stochastic gradients over the same. This results in having lower vari- ance with respect to recalculating the projecting subspace.
Jc ); and its corre- Step 2. During the backward pass, (ow,))
BC sponding estimated version aw; are computed. We de- fine the perplexity PA,, of layer given Ej as the Frobenius norm of the difference between the low-rank and original gradients:
When considering such behavior in the context of activation maps, a similar principle applies. Specifically, the activation map A; of layer is computed as follows:
[( OL aw.)
ac ow;
Pas
A a [conv Ai-1,,
(10)
This process is repeated across all layers, resulting in perplexity matrix P E Nx and corresponding rank tensor R. NxE4 containing the selected ranks across the 4 modes of activation tensors for each combination of layer and explained variance threshold.
where W:-1 represents the weights of layer i - 1, and a() is the non-linearity function. Assuming that only a small update made at each step, Wi-1 undergoes only minor changes, leading to minor changes in A,. Thus, over a small
5

---

where Pa = II d' (Nguyen et al., 2024). This is even more significant since the first few singular values in each mode typically carry most of the energy of the entire activation map. In other terms, the target rank r; obtained as an output of the rank selection algorithm presented in Sec. 3.3, is expected be low compared to the full di- mension, resulting in considerable compression ratios with limited information loss. Fig. 2d illustrates the speedup ratio Rg between vanilla training and ASI, defined as their Beyond Low-rank Decomposition
Vanilla training
ASI
HOSVD 8 0.8
Gradient filtering R2
Topp Vadidaaiinn Accuracy (%)
65-
65
60
60
1.56 smallee
FLOP's
56
55
30
50-
45
15
120.00 smaller in memorry
40-
40-
252.65 X smaller
FLOPs
10
10
10P
16
10t4
10
10
Peak Activation Memory (MB)
FLOPs
Figure 4. Performance of MCUNet pre-trained on ImageNet and finetuned on Pets with different strategies.
Algorithm ASI for layer with set of rank r; E Ropt
produce A., we replace the SVD step in the HOSVD, al- gorithm. other words, we perform a single subspace iteration with warm start four times, corresponding to the four modes of the activation map. Algorithmic details are provided in Algorithm 1.
Input:
Activation map A E RBXCxxM.XW at epoch E N n
Target ranks for 4 modes [1, min(ai.m. ml. where (as.m, bem) is shape of activation map A at mode m.
3.5. Computational Complexity
Function:
Initialize S. AA
In this section, we assess the computational complexity of the compression operations during the training process. Perplexity search and rank selection are performed offline and only once for each model; therefore, we exclude them from this evaluation.
for 1 to do
Aim unfold A along mode m such that A,m E Ras.m xb..
if = 0 then
Initialize V R xram from an i.d standard
Following the notation from previous sections, the com- putational overhead of our method, OASI given by (14). HOSVD; incurs the following overhead:
normal distribution.
else
V A7 (()) (0
end if (t)
OHoSVD, =
Σ
max Pa)2 x min (d, Pa), (11)
(Athogonalize(A,,.)
f.m
dED:
S. S, Xm k.m
end for
return S;, ((() im with m: 1-14
number of steps, A; can also be considered stable. This ussumption is further supported by the findings of (Virmaux & Scaman, 2018), where it was observed that most activa- tion functions, such as ReLU, Leaky ReLU, SoftPlus, Tanh, Sigmoid, AreTan, and Softsign, as well as max-pooling, have Lipschitz constant equal to 1. Based on this logic, propose the Activation Subspace Iteration (ASI) method, which applies a single subspace iteration with a "warm start" to effectively estimate the low-rank version of activation maps.
FLOPs ratio. Our method achieves faster training compared to vanilla training when considering smaller values of trun- cation ranks r, and larger activation maps, while reaching similar rates of memory savings as HOSVD, (Fig. 2c). De- tailed analysis can be found in Appendix A.2 and A.3.
4. Experiments
Given optimal ranks r, E Ropt obtained for a given budget following the method proposed See. 3.3, to effectively
In this section, we present experiments that demonstrate the effectiveness of our proposed method. First, we provide,
6

---

Beyond Low-rank Decomposition
Vanilla training
ASI
HOSVD,
Gradient filtering
10a
10c
10
Backwarrd Time (s)
Forward Time (s)
Total Time (8)
10'
10
X75-79
10-1
10P
10
x14.726
x90.00 in
10-1
10"
averaae
2
5
91
10
3
8
9
3
10
Layers
Layers
LLayers
Figure 5. Training time of MCUNet over iterations CIFAR-I0 with batch size 128 Raspberry 5.
in Sec. 4.1, an overview of the experimental setup. Then, in Sec. 4.2, we conduct ablation study analyze the impact of the warm-start process the performance of ASI. Sec. 4.3 reports the results obtained when applying ASI to state-of-the-an architectures and datasets. Finally, we compare the techniques a real-world deployment set- ting(Sec. 4.4). All simulation experiments are conducted using Py PyTorch 1.13.1 on an NVIDIA Quadro RTX A4500 with 20GB of VRAM, while the on-device experiments are performed on a Raspberry Pi with Cortex-A76 CPU and 8GB RAM.
and vanilla training, which remain constant.
4.2. Ablation Study
Concerning this experiment, we evaluate the impact of warm-start on ASI when fine-tuning MCUNet at different depths. The model is pre-trained ImageNet- Ik, with CIFAR-10 as the downstream dataset. The number of fine- tuned layers gradually increases from the final layer up- ward, where the first marker in each plot indicates the case in which only the last layer is fine-tuned. Fig. 3 presents the experimental results, showing that even when warm- start not applied equivalent to assuming that activation maps across layers are entirely independent-the model still converges with just single subspace iteration. On aver- age, incorporating "warm-start" improves ASI accuracy by 3.87%.
4.1. Experimental Setup
Overview. To evaluate the effectiveness of ASI, duct classification tasks six datasets: CIFAR-10, CIFAR- 100 (Krizhevsky. 2009), CUB (Wah et al., 2011), Flow- ers (Nilshack & Zisserman, 2008), Pets (Zhang et al., 2022), and ImageNet (Deng et al., 2009). Since practical feasibility key concern, most experiments are performed using MCUNet (Lin et al., 2022). model that has been success- fully deployed real edge devices with only 256kB of memory. Additionally, we extend our evaluation to other compact urchitectures, including ResNet-18, 18, ResNet-34 (He et al., 2016), and MobileNetV2 (Sandler et al., 2018). De- tailed setup information can be found in Appendix B.1.
4.3. Main results
MCUNet with Pets. We conduct similar experiment as in the previous section, using the Pets dataset as the down- stream task to compare the performance of ASI, HOSVD,. and vanilla training. Following the findings of Nguyen et al., we set 0.8 to achieve good compression ratio while minimizing information loss with HOSVD,. The peak memory usage of HOSVD, is then used the mem- ory budget constraint for ASI. The experimental results are illustrated in Fig. 4, where it is evident that ASI outperforms both others. It achieves a compression ratio comparable to HOSVD while consuming even fewer FLOPs than vanilla training for the same number fine-tuned layers.
Rank Selection. We employ a set of explained variance thresholds & = {0.4, 0.5, 0.6, 0.7, 0.8, 0.9) to measure the perplexity P of each layer.
Memory Budget. Since HOSVD, operates under an ex- plained variance constraint without strict memory control, does not enforce an explicit memory budget. To ensure a fair comparison, we set the peak memory consumption of HOSVD, as the memory budget for ASI when fine-tuning the number of layers.
Notably, a similar accuracy level, ASI reduces mem- ory usage by up to 120.09x compared to vanilla train- ing. Additionally, ASI requires 252.65 fewer FLOPss than HOSVD, Overall, ASI achieves comparable accuracy to that of HOSVD, while saving up to 1.86x of the computational cost. We also performed experiments on other downstream tasks, included in Appendix B.3.
Result Logging. The reported results include FLOPs and memory usage for both ASI and vanilla training. For HOSVD,, we report peak FLOPs and peak memory, as its resource consumption varies dynamically, unlike ASI
ImageNet- 1k Classification. Similar experiments with Im-

---

Beyond Low-rank Decomposition
Table Experimental results on ImageNet with "#I aal.erss" denotes the number of fine-tuned convolutional layers, counted from the model's end. Activation memory is presented in MegaBytes (MB), and computational complexity is expressed in terms of Gigaflops (GFLOPs). Gradient filtering is utilized with patch size R2.
MobileNetV2
ResNet18
Method
Method
#Layers
Acc
Mem (MB) 1 GFLOPs
#Luyers
Acc
Mem (MB) 1 GFLOPs
All
74.0
1651.84
830.82
All
72.8
532.88
291.79
Vanilla
Vanilla
2
62.6
I5.31
4.50
2
69.9
T2.25
29.60
training
training
4
65.8
28.71
57.48
4
71.5
30.63
46.45
Gradient filtering
2
62.6
5.00
4.50
Gradient filtering
68.7
4.00
29.60
4
65.2
9.38
57.50
4
69.3
7.00
47.70
HOSVD
2
61.1
0.15
3049.71
HOSVD
69.2
0.97
1581.42
(e 0.8)
4
63.9
0.73
5895.31
0.8)
4
70.5
2.89
4048.56
2
60.3
0.13
2.81
2
68.9
0.93
19.04
ASI
ASI
4
64.0
0.71
31.54
4
70.6
2.63
33.14
MCUNet
ResNet34
Method
Method
#Layers
Acc
Mem (MB) 1
GFLOPs
#Layers
Acc
Mem (MB) +
GFLOPs
All
67.4
632.98
248.84
All
75.6
839.00 09
528.55
Vanilla
Vanilla
2
62.1
13.78
19.31
2
69.6
12.25
25.60"
training
training
4
64.7
19.52
19.88
4
72.2
24.50
59.19
Gradient filtering
2
61.8
4.50
19.31
Gradient filtering
2
68.8
4.00
29.60
4
64.4
6.38
19.89
4
70.9
8.00
59.21
HOSVD
2
61.7
0.48
1988.05
HOSVD (5 0.8)
2
68.7
0.30
1579.64
(8 0.8)
4
63.9
0.88
2457.59
4
71.1
1.11
3160).66
2
61.7
0.38
11.01
2
68.3
0.25
16.44
ASI
ASI
4
63.5
0.83
11.93
4
71.1
1.09
35.31
ageNet were conducted across different models. The results shown in Table 1. We observe that, in all cases, for the same depth, ASI consistently consumes the least resources while achieving accuracy comparable to HOSVD,.
sufficient to compensate for the minimal overhead of ASI, making it 91.0x faster than HOSVD, 1.86x faster than gradient filtering, and 1.56x faster than vanilla training.
The memory consumption of HOSVD, is not uniform-2 tends to be lower at the beginning of training, then surges to peak before slightly decreasing as the model converges. Intuitively, the actual memory requirement of HOSVD, does necessarily correspond to its peak memory usage but rather lower value. This explains why, despite using HOSVD, 's memory consumption as the budget, ASI never actually reaches that threshold in practice.
5. Conclusion
In this work we introduced ASI (Activation Subspace It- eration) as a shortcut approach to address the activation memory bottleneck during backpropagation, enabling the feasibility of on-device learning. Taking advantage of the short-term stability of activation maps during training, the combination of a single subspace iteration and reusing pre- vious approximations enables us to achieve up to 120.09x activation memory reduction and up to 1.86x fewer training FLOPs while maintaining similar performances as vanilla training. Additionally, our method delivers strong improve- ments in latency, achieving up 91.0x times speedup when deployed on Raspberry Pi 5, compared state- of-the-art approaches. In the future, we wish to explore extending ASI to larger models, such as large language models, and further optimizing its computational aspects to broaden applicability.
4.4. On-device Latency
We conduct similar experiments in a real embedded envi- ronment, using Raspberry Pi 5. We report the estimations over the first 5 iterations of MCUNet trained on CIFAR-10, using batch size 128. The results of our experiment are shown in Fig. During the forward pass, due to the high computational complexity of performing HOSVD at each training step, HOSVD, takes, average, 106.13x longer than both ASI and vanilla training. In the backward pass, the low-rank computation enables ASI and HOSVD to be 3.95x faster than vanilla training. This speed advantage is
8

---

Beyond Low-rank Decomposition
Acknowledgements
Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B, The reversible residual network: Backpropagation with- out storing activations. Advances in neural information processing systems, 30, 2017.
Part of this work was funded by HilPARIS Center on Data Analyties and Artificial Intelligence. by the Euro- pean Union's Horizon Europe Research and Innovation Pro-
gramme under grant agreement No. 101 120237 (ELIAS), by the European Union's 's HORIZON Research and Inno- vation Programme under grant agreement No 101 120657, project ENFIELD (European Lighthouse to Manifest Trust- worthy and Green AI) and by French National Research Agency AANR-22-PEFT-00N and ANR-22-PEFT--NN as part of France 2030, the NF-NAI project and NF-FITNESS project.
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage, D. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 1eee/8811000000 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 2016.
Impact Statement
Hinton, G. The forward-forward algorithm: Some prelim- inary investigations. arXiv preprint arXiv:2212.13345, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:2106.0966 2021.
References
Alajlan, N. N. and Ibrahim, D. M. Tinyml: Enabling of inference deep learning models on ultra-low-power iot edge devices for ai applications. Micromachines, 13(6):
Hu, Y., Xie, Y., Wang, T., Chen, M., and Pan, Z. Structure- aware low-rank adaptation for parameter-efficient fine- tuning. Mathematics, 11 11(0))4317, 2023.
851, 2022.
Hu, Z., Tang, J., Wang. Z.., Zhang, K., Zhang, L., and Sun, Q. Deep learning for image-based cancer detection and diagnosis- survey. Pattern Recognition, 83:134-149, 2018.
Chen, L.C.., Papandreou, G., Schroff. F., and Adam, H. Rethinking atrous convolution for semantic image seg- mentation. arXiv preprint 1XXv:1700000587, 2017.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044 2019.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Lin, J., Zhu, L., Chen, W.-M.. Wang, W.-C., Gan, C., and Han, S. On-device training under 256kb memory. Ad- vances Neural Information Processing Systems, 35: 22941 222554, 2022.
Deng, J., Dong. W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009.
Liu, Z., Lyn, J., Zhu, W., Tian, X., and Graham, Y. Alora: Allocating low-rank adaptation for fine-tuning large lan- guage models. arXiv preprint arXiv:2403. 16187, 2024.
Dhar, S., Guo, J., Liu, J., Tripathi, S., Kurup, U., and Shah, M. A survey of on-device machine learning: An algo- rithms and learning theory perspective. ACM Transac- fions on Internet of Things, 2(3):1-49, 2021.
Long, Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 3431-3440, 2015.
Ding, N., Lv, X., Wang, Q., Chen, Y., Zhou, B., Liu, 7_, and Sun, M. Sparse low-rank adaptation of pre-trained lan- guage models. arXiv preprint urXiv:2311.116699 2023.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficcennn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Gemini Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Nguyen, L.-T., Quélennec, A., Tartaglione, E., Tardieu, S., and Nguyen, V.-T. Activation map compression through

---

Beyond Low-rank Decomposition
tensor decomposition for deep learning. In The Thirty- eighth Annual Conference Neural Information Pro- cessing Systems, 2024.
Virmaux, A. and Scaman, K. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Ad- vances in Neural Information Processing Systems, 31, 2018.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In Indiin Con- ference on Computer Vision, Graphics and Image Pro- cessing, 2008.
Vogels, T., Karimireddy, S. P., and Jaggi, M. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
Ozbayoglu, A. M., Gudelek, M. U., and Sezer, O. B. Deep learning for financial upplications: A survey. Applied soft computing, 93:106384, 2020.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd birds-200-2011 dataset, 2011.
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, J. Car- bon emissions and large neural network training. arXiv preprint ar ar0i 103.50 2021.
Xiao, T., Liu, Y., Zhou, B., Jiang. Y., and Sun, J. Unified per- ceptual parsing for scene understanding. Proceedings of the European conference on computer vision (ECCV). pp. 418-434, 2018.
Yang, Y., Li, G., and Marculescu, R. Efficient on-device training via gradient filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3811-3820, 2023.
Pau, D. P. and Aymone, F. M. Suitability of nerward-fowaate and pepita learning to micommons-ting benchmarks. In 2023 IEEE International Conference on Omni-layer In- telligent Systems (COINS). pp. 1-6. IEFE, 2023.
Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. Asvd: Activation-awar singular value decomposition for compressing large language models. arXiv preprint arXhv:2312.0582/, 2023.
Quélennec, A., Tartaglione, E., Mozharovskyi, P., and Nguyen, V.-T. Towards on-device learning on the edge: Ways to select neurons to update under budget con- straint. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pp. 685-694, 2024.
Zhang, H., Zhou, S., Li, G. Y., and Xiu, N. 0/1 deep neural networks via block coordinate descent. arXiv preprint arXiv:2206.0937. 2022.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et Improving language understanding by generative pre-training. .... 2018.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama:
An open-source small language model. arXiv preprint arXiv:2401.02335,, 2024.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Zhang, Q. Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for pfrfmeter-efficient fine- tuning. arXiv preprint xxiv:23033110512 2023.
Stewart, G. and Miller, J. Methods of simultancous itera- tion for caleulating eigenvectors of matrices. Topics in Numerical Analysis 11, 2, 1975.
Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. Pyramid scene parsing network. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2881- 2890, 2017.
Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intel- ligence, volume 34, pp. 13693-13696, 2020.
Thompson, N. C., Greenewald, K., Lee, K., and Manso, F. The computational limits of deep learning. arXiv preprint arXiv:2007.05538 10, 2020.
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free free low-rank adaptation. arXiv preprint 00.iv:2210007555 2022.
10

---

Beyond Low-rank Decomposition
A. Additional Theoretical Details
Following the notation introduced the main paper, we propose in this section to analitically study the overhead introduced in the forward pass when performing each compression algorithm, the speedup of efficient weight derivative computation and the required space store the compressed activations.
A.I. Subspace Iteration
In this section, we present the subspace iteration technique from (Vogels et al., 2019) for compressing activation maps instead of gradients. The core idea remains unchanged: a single step of subspace iteration (Stewart Miller, 1975) employed obtain fast low-rank approximation of the given matrix. In our case, this matrix corresponds to the four unfolded versions of the activation maps along their respective modes. However, as in the original work, single step of subspace iteration typically does not provide sufficiently accurate low-rank approximation. To mitigate this, Vogels et al. suggest reusing the low-rank approximation from the previous iteration as the initialization for the current iteration.
This reuse is particularly well-motivated our setting. Although activation maps evolve as network parameters are updated, their changes between consecutive iterations are relatively small. This behavior stems from Lipschitz continuity of activation functions (Virmaux & Scaman, 2018) and the "tiny" incremental nature of parameter updates during optimization. By leveraging the previous approximation, we effectively smooth the sequence of activation maps across iterations, reducing the variance of the low-rank approximation compared to the case without reuse. A formal proof of this property can be found in (Vogels et al., 2019).
Overhead. Consider a model's gradient matrix ( E Raxb at time t with targeted rank E [1, min(a, 6)] N.N. Algo- rithm 2 requires performing two matrix multiplications p(t) Q(QQ() and Q(+) M p(t), which cost OFLOps PFL.os(2abbr,, and orthogonalization using Gram-Schmidt that costs OFLOPS (r²). Therefore, the total overhead FLOPS is:
Osww 2abr + +,
(12)
Algorithm Subspace iteration with Warm start at epoch t
Input:
Epoch index
Matrix M e _axb
Targeted rank r E min((,
nN NN
Function:
t = 0 then
Initialize Q()) E cxx from an i.i.d standard normal distribution.
else
Q()) (1-1)) # This is "Warm start"
end if
p(t) M
p(1) = Orthogonalize(P())
Qi0 (1 p(t)
return (())
A.2. HOSVD Overhead
As proved by Nguyen et al. (2024), performing HOSVD at each training iteration yeilds to decompose an activation map
A; E T C.C.H.WWWW with D; [Bi, C,, H,, W.) yeilds:
OhoSVD. Σ max (d, Pa)2 x min (d, Pa).
(13)
dED,
where Pa- IIre

---

Beyond Low-rank Decomposition
A.3. Details of Overhead, Computational Speedup and Space Complexity between ASI and Vanilla Training
Overhead. Using ASI to decompose the tensor is equivalent performing subspace iteration on each of its mode, Considering mode m, decomposing the unfolded version Ai. E R of A; requires cost of OrLOPs cetoos(2dde 2dd'rs,, + r mm) where d and d' represent the dimensions of As... Consequently, performing subspace iteration across all four modes incurs a total cost of:
OASI Σ (2dd'r.. + +r,,m) d Da,,, d' D {d).
(14)
Speedup. Leveraging the low-rank gradient computation technique in Nguyen et al. (2024), the backward pass training step of ASI will cost:
CASI F.,,B. + +r F H,W, F,,, C.,,
(15)
While this cost of vanilla training is:
Cvanilla = D.C.C,BBBWWW
(16)
Moreover, the amount of FLOPs that vanilla training requires to perform the forward pass is:
O vanilla = D,C,C,,,H,W,
(17)
We then can deduce the speedup ratio Rg for training step between vanilla training and ASI as follow:
Vvanilla + Cvanilla
Rs vvanilla OASI + CASI
(18)
Space complexity. Essentially, ASI performs compression in a manner similar to HOSVD,, achieving a comparable compression ratio. As a result, the space complexity ratio (i.e., the memory ratio between vanilla training and ASI) is the same as HOSVD,, which is:
B,C,H,W; Rc 2 + Br,, Cra Hri, + Wr.
(19)
B. Additional Experimental Details
B.1. Detailed Experiment Setup
To ensure fair comparison, we adopt the same experimental setup as Nguyen et al. (2024), with the details outlined below:
ImageNet. The dataset is divided into two equal-sized, non-i.i.d. partitions using FedAvg (McMahan et al., 2017). Models are pretrained on the first partition, while the second partition is used for fine-tuning, with 80% allocated for training and the remaining 20% as the validation set. We fine-tune the provided checkpoints for 90 epochs, applying I.2 gradient clipping with threshold of 2.0. Optimization is performed using SGD with a weight decay of 1 10 and momentum of 0.9, Data augmentation techniques include random resizing, flipping, normalization, and mini-batching with size of 64. The loss function is cross-entropy. The learning rate increases linearly over four warm-up epochs, reaching 0.005, and then follows cosine annealing decay schedule.
Other Datasets. The models are first pretrained on ImageNet before being fine-tuned on a completely different downstream dataset with the same raining-validation ratio as above. Training is conducted using cross-entropy loss with the SGD optimizer. The initial learning rate is set to 0.05 and follows a cosine annealing decay schedule, Momentum is fixed O, and the weight decay remains at x 10-4 Additionally, L2 gradient clipping with a threshold of 2.0 applied.
Semantic Segmentation. Following the training policy described in Yang et al. (2023), we used the pretrained and calibrated checkpoints provided by the authors. These models were first pretrained on the Cityscapes dataset using MMSeggentation.. and then fine-tuned on the augmented Pascal VOC12 dataset. We used a learning rate that starts at 0.01 and follows a cosine annealing schedule. The weight decay was set to 5 x 10-4 and momentum to 0.9. We trained with a batch size of 8. For data augmentation, we applied random cropping, horizontal flipping, photometric distortions, and image normalization. The models were fine-tuned for a total of 20, 000 steps using cross-entropy loss.
B.2, Perplexity with different values of
12

---

Beyond Low-rank Decomposition
Fig. 6 illustrates the change in perplexity of the last four layers of MCUNet when evaluated with different explained variance thresholds values of E result in lower perplexity. However, when & decreases below 0.5, there is no noticeable change in perplexity. This phenomenon arises because most of the energy the activation maps concentrated in the first singular value, demonstrated by Nguyen et (2024); in this case, the first singular value alone captures more than 50% of the total energy. Therefore, experimenting with smaller than this value does not lead to any further improvement. To save computational resources, we limit our perplexity measurements to & > 0.4 in all experi- ments.
As expected, higher
Perplexity
looer 2dd kol lace
yr4
1" Lue hure
18.0
0.2
0.6
0.7
0.9
Figure 6. Perplexity a function of explained variance threshold for the last layers of MCUNet.
B.3. Additional Results
Additional Classification Results. Following the same training policy described Sec. 4.3, we compare the performance of various techniques across different models, all which are pretrained on ImageNet-IK. The results, presented Table demonstrate that ASI consistently achieves comparable compression rate HOSVD, while maintaining similar accuracy. Furthermore, ASI yields substantial reduction in FLOPs, even outperforming vanilla training in terms of computational efficiency.
Semantic Segmentation. Table 3 reports the performance of the evaluated techniques on the segmentation task. As previ- ously observed, ASI continues to demonstrate superior performance in terms of both compression ratio and computational cost, confirming its effectiveness beyond classification.
ASI on Large Language Models. this experiment, we fine-tune TinyL 1B (Zhang et al., 2024) on the BoolQ (Clark al., 2019) dataset to compare ASI with vanilla training. The dataset is divided into batches of size 8, with each sample having maximum length of 512 tokens. Due to computational cost, applying HOSVD, this model is infeasible. Therefore, instead of budget-based compression, we directly set the compression rank to 20. Table 4 presents the results when fine-tuning from 1 and 5 layers. Notably, fine-tuning 5 layers leads to up to a 2500x reduction in activation memory, and computational cost decreases by approximately 1.9x, while retaining decent accuracy. Remarkably, as the number of fine-tuned layers increases, both memory compression and FLOP savings improve significantly under ASI.
13

---

Beyond Low-rank Decomposition
Table 2. More classification results
Model
Method
#layems
Are
Mem (MBIL
CURR30
TELOKS]
Aec
Mem (MIB)
Flowers102
TELOPS
Are
Mem (MBB)
Pets
TELOSS]
Aee
CIPAR-18 Mem (MIDL
TILOPS
Are
Mon (MIBL)
CHEAR-100
TILONS
Vauils tnining
A4
14,25
30257
1301,07
0.00
1,5.
3013
2310.07
30.65
0.01
1.0
35.25
35134
3301.57
"30.55"
0,35
1255
10
13.25
330001
1.95
66.78
75.03
1503327
50563
003
53.73
57.42
0.81
53.15
57.42
0.11
89.53
57.42
0.01
89.45
57.42
0.11
67.57
57.42
0.01
Oraacer fitering R2
46.68
10:03
0231
50191
10:00
0.0
$5.28
10,03
0201
67.00
10.00
1100
64.72
10,03
0221
MohileNes V2
50.30
18.25
0.01
52.70
18.75
0.11
89.22
18.75
0.11
80.25
18:75
0.11
67.22
18.75
0,01
46.27
0.27
11.38
70.02
0.30
11.55
85.25
0.1
11.88
83.65
0.25
11.35
61.17
0,16
11.38
( 0.5)
80.11
0.67
32.91
81.58
0.26
22.90
50.00
0.97
22.93
65.45
0.65
22.00
66.00
0.71
22.90
ASI
46.44
$0,69
0:3h
0.63
0201
0.06
10115
81.58
0.25
0m6
01011
0.06
55.15
85.91
0115
0.76
0311
0.06
85700
65.03
0.6
0.25
1100
0.06
61.05
66.15
0.35
0.63
0301
0.06
Vinile Iniring
11.00
F1146
1205
0.50
44.51
1265 95
57.50
0.50
0.50
30.10
74.37
1265.90
50.36
0.54
1305
1014
-0.30
68.11
1205.90
27755
"07A"
39.05
0.04
39.05
Gradten fihering R2
9300
0334
1000
0031
5000
71.18
MCUNE
12.75
0.04
12:75
0134
12.75
HOSVD 0.53
7:15
0.26
9,56
9.56
49:71
ASI
0000
0.21
0002
0.03
Vanilla
34.40
illering Rt2
73.02
0.07
Vanilla
3,530"
23.50
'..
0.12
72.83
Gratern
3.00
0.66
8.00
0.12
16:00
90.57
BOSVD
58.77
0.80
0.89
69.96
1.70
12:55
1.48
12:36
70.68
ASI
6.60
1.58
0.0%
0.09
61:20
1:34
0.00
09:06
30.00
Iniinng
Nanila
Al
7551
7087
76.82
1001.25
153.75
5/8""
0.0
0.23
1201
100
89.29
3491.5
37.05
183.75
0TC
0.23
1:30
51.15
94.00
93.91
101125
183.75
"1.86"
0.1
0.25
1,40
04.00
05.22
309135
183.75
0.23
1.30
-9:35
57.08
30.65
1091.25
183.75
101
0.23
1,00
SWinT
BOSYD
1037
2.76
110001
53.38
133
11500
91.44
4.77
110.01
13.57
0.35
11600
77:15
5307
11K01
0.5)
76.66
4.20
232.01
39.51
6.23
232.03
94.38
7.99
232.01
94.99
11.08
252.02
50.19
9.97
232.00
ASI
75.08
1006
233
3.45
0.15
0778
33.11
68.05
5.89
331
0.17
93.91
93.84
7.98
135
0.87
000
05.15
0330
11.21
336
100
0.18
80.67
7331
9.87
4.8
0.18
0097
C. Limitation
In essence, the backtracking algorithm is a brute-force method. As the number of layers requiring fine-tuning increases, the size of P grows accordingly. Solving the optimization problem (9) becomes highly resource-intensive even though it can be performed offline on powerful server. More efficient search algorithms need to be developed to replace this approach.
14

---

Beyond Low-rank Decomposition
Table Experimental results for semantic segmentation. mloU is the mean Intersection over Union, and mAce is the micro averaged
accuracy.
PSPNet (Zhao et al., 2017)
PSPNet-M (Zhao et al., 2017)
Method
Method
#Layers
mloU
mAec 1
Mem (MB)
TFLOPs
#Layers
mloU
mAce
Mem (MB)
TFLOPS
All
54.97
920.78
All
48.92
62.11
2622.49
Vanilla
Vanilla
training
5
39.36
51.79
128.00
0.08
training
5
36.22
46.31
104.00
0.06
10
53.17
67.18
352.00
0.62
10
45.62
58.35
604.00
1.19
Gradient Filter
5
39.34
51.59
8.00
0.08
Gradient Filter
5
35.73
45.78
6.50
0.06
10
51,20
65.01
22.00
0.62
10
44.89
57.38
37.75
1.19
HOSVD
38.11
49.29
0.47
177.06
HOSVD
33.40
42.50
0.03
117.06
(5 0.8)
10
49.23
1.40
322.25
(c 0.8)
40.06
51.79
1.47
744.71
5
37.90
49.11
0.32
0.05
5
32.73
41.89
0.02
0.03
ASI
ASI
10
47.72
60.34
1.40
0.36
10
40.51
52.12
1.36
0.63
DLV3 (Chen et al., 2017)
DLV3-M (Chen al., 2017)
Method
Method
#aayers
mloU
mAce
Mem (MB)
TFLOPs
#laayers
mIoU
mAce
Mem (MB)
TFLOPs
All
58.44
72.03
1128.02
0.97
All
55.87
69.47
2758.01
3.02
Vanilla
Vanilla
training
5
40.75
52.95
336.00
0.17
training
5
38.58
49.61
240.00
0.12
10
55.04
69.07
560.00
0.71
10
47.91
61.67
620.00
0.71
Gradient Filter
5
32.18
42.93
27.47
0.17
Gradient Filter
5
35.70
46.71
20.71
0.12
10
47.44
60.08
83.47
0.71
10
45,40
58.97
65.62
0.71
HOSVD
5
38.52
50.14
2.66
247.63
HOSVD (c 0.8)
5
35.55
45.66
0.63
139.57
( 0.8)
10
50.11
63.14
1.93
392.78
42.68
54.17
1.04
611.29
5
38.43
50.16
2.37
0.12
5
35.76
45.93
0.62
0.07
ASI
ASI
10
45.72
58.45
1.89
0.41
10
42.30
54.13
1.03
0.38
FCN (Long
2015)
UPerNet (Xiao al., 2018)
Method
Method
#aayers
mloU
mAce 1
Mem (MB) J
TFLOPs
#Layers
mIoU f
mAce f
Mem (MB)
TFLOPs
All
45.36
59.53
952.00
0.90
All
64.71
77.32
2168. 78
3.55
Vanilla training
5
27.31
38.2
288.00
"6.4
Vanilla
5
48.05
61.66
1380.00
3,33
training
10
43.54
57.96
480.00
0.72
10
48.90
63.10
1436.00
3.35
Gradient Filter
5
27.24
18.00
0.41
Gradient Filter
46.79
60.50
33.00
3.33
10
36.91
120.00
36.50
HOSVD
5
26.34
36.14
1.43
206.12
HOSVD
5
45,28
57,80
1.35
10866.86
(s
10
30.91
41.19
3.77
295.93
(e
10
46.44
58.93
1.68
10881.59
5
26.51
36.26
0.99
0.23
5
42.73
54.99
1.25
1.77
ASI
ASI
10
36.44
48.05
3.65
0.43
10
45.18
58.00
1.57
1.78
Table Performance comparision between vanilla training and ASI when finc-tuning TinyLlama IB with BoolQ dataset.
Vanilla training
ASI (rank=20)
#Layers
Ace t
Mem (MB) 1
TFLOPs
Ace 1
Mem (MB) 1.
TFLOPs 1
65.94
1408
3.02
64.69
0.51
1.68
2
66.37
1920
6.04
64.81
0.74
3.33
66.91
2432
9.07
65.00
0.98
4.98
67.44
3840
12.09
66.31
1.49
6.66
67.78
4352
15.11
66.34
1.72
8.31
15


Reviews:
[{"review_id": "tN3idHTXLM", "reviewer": "Reviewer_dju2", "summary": "This paper proposes Activation Subspace Iteration (ASI), a novel technique to address memory bottlenecks in on-device learning. The method compresses activation maps in neural networks using low-rank decomposition strategies. The key innovations include: (1) a perplexity-based rank selection strategy that identifies optimal compression rates for each layer under a given memory budget constraint before training begins, (2) a single subspace iteration with \"warm start\" to replace traditional HOSVD-based compression methods, and (3) computation of gradients directly in the compressed space. The authors empirically demonstrate that ASI can reduce activation memory usage up to 120.09× compared to vanilla training while reducing training FLOPs up to 1.86×. On a resource-limited device (Raspberry Pi 5), ASI achieves significant speedups compared to alternative methods (91.0× faster than HOSVD, 1.86× faster than gradient filtering).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1: Use of \\citet.**\n\nThank you for this note. We will revise it in the camera-ready version.\n\n**Weakness 2: Other rank search algorithms besides brute-force?**\n\nYes, there are certainly alternative methods—for example, using dynamic programming, we might reduce the computational complexity from exponential to linear, at the cost of employing more memory.\n\nHowever, in the context of on-device learning, the models used are typically small with a limited number of layers, and we are heavily constrained by memory. As a result, brute-force search remains computationally feasible while ensuring an optimal solution.\n\nThat said, rank selection itself is not the core contribution of ASI. It would be interesting to explore smarter methods for rank-selection, relying on the learning dynamics of Deep Models (eg. leveraging something like the Information bottleneck); however, this still remains an open research quest. We leave further exploration of rank search algorithms for future work.\n\n**Weakness 3: Apply ASI to transformer models and LLMs.**\n\nWe appreciate your feedback. To address this, we expanded our experiments to Swin Transformer, image segmentation, and LLM task using TinyLlama with 1.1B parameters. *For further details, please refer to Question 2 of Reviewer N3Rd*.\n\n**Weakness 4: The problem of accuracy loss.**\n\nWe acknowledge that accuracy drops due to compression—this is the tradeoff. As such, this phenomenon also occurs with all other related state-of-the-art techniques, including $\\text{HOSVD}_\\varepsilon$, Gradient Filter, LoRA, and its variants. The key point is that ASI achieves a superior Pareto curve compared to other methods (Fig. 4).\n\n**Weakness 5: Stability of activation map and combination with GaLore.**\n\nThank you for suggesting GaLore. Our response is as follows:\n\n- *Stability of activation maps:*\n  - ASI is specifically designed for fine-tuning, where large learning rates are typically not used. As a result, the input to the activation function does not change drastically across training iterations, i.e., it remains stable within a small number of iterations.\n  - Moreover, *(Virmaux & Scaman, 2018)* indicated that most commonly used activation functions have a Lipschitz constant of 1, meaning that if the input remains stable, the output (activation map) remains stable as well.\n\n- *Effect of GaLore:*\n  - Based on our understanding, GaLore does not directly affect the stability of activation maps.\n  - GaLore is a gradient compression technique, while ASI specifically works on activation maps. In principle, both methods could be combined to maximize efficiency.\n  - However, GaLore is particularly useful for Adam, where optimizer state becomes a concern. In contrast, for on-device learning, which is our primary focus, SGD without momentum is preferable since it eliminates the need to store optimizer state. Consequently, GaLore offers little benefit in this setting.\n\nWe acknowledge that GaLore is an interesting approach, and leveraging it could lead to promising results-and will be added in the related works section. However, we have not observed how it influences the stability of activation maps in fine-tuning. Could you please clarify your thoughts on this?", "claims_and_evidence": "The major claims of the paper are generally well-supported by evidence:\n\n1. The activation memory reduction claim (up to 120.09×) is substantiated through extensive experimentation across multiple datasets (CIFAR-10/100, CUB, Flowers, Pets, ImageNet) and architectures (MCUNet, MobileNetV2, ResNet-18/34).\n\n2. The computational efficiency claim (up to 1.86× reduction in FLOPs) is backed by both theoretical analysis (Section 3.5) and empirical measurements (Section 4.3 and 4.4).\n\n3. The accuracy claim (comparable performance to vanilla training and HOSVD) is well-supported through tables and figures showing performance across multiple settings.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem of on-device learning:\n\n1. The use of real-world datasets (including ImageNet) is appropriate for evaluating model accuracy.\n\n2. The measurement of memory usage and FLOPs provides clear metrics for resource efficiency.\n\n3. The on-device measurements on Raspberry Pi 5 provide real-world validation of the approach's practicality.\n\n4. The comparison against baselines (HOSVD and gradient filtering) gives context to understand the relative advantages of ASI.", "theoretical_claims": "see Strengths And Weaknesses.", "experimental_designs_or_analyses": "see Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "see Strengths And Weaknesses.", "essential_references_not_discussed": "see Strengths And Weaknesses.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper addresses a critical practical challenge (memory bottleneck) in on-device learning with a novel solution.\n2. The paper provides a comprehensive evaluation across multiple datasets, model architectures, and settings.\n3. The real-world validation on Raspberry Pi 5 demonstrates that the approach is immediately applicable.\n\nWeaknesses:\n1. Authors should use \\citet when it is a part of sentence.\n2. The rank selection approach relies on a brute-force backtracking algorithm, which the authors acknowledge as a limitation in Appendix C. Is there other possible alternative method?\n3. The paper focuses exclusively on convolutional networks and CV task; it's unclear how well ASI would work for other tasks or architectures like transformers.\n4. While memory and computation reductions are significant, the accuracy sometimes drops substantially when fine-tuning deeper networks, suggesting scalability limitations.\n5. The warm-start strategy assumes that activation maps are stable across training iterations, but this assumption may not hold during early training or when learning rates are high. I am not sure if GaLore can help deal with it?", "other_comments_or_suggestions": "see Strengths And Weaknesses.", "questions_for_authors": "see Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "joMRksvymx", "reviewer": "Reviewer_Ekgz", "summary": "The authors focus on the problem of reducing activation memory usage and computational complexity during on-device learning. The authors try to deploy learning tasks on resource-constrained edge devices while maintaining acceptable performance. The evaluation based on the MCUNet model shows the performance on image classification tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review, below is the answer to your only question.\n\n**Question: How do we calculate training FLOPs?**\n\nCurrently, we measure training FLOPs based on theoretical calculations, which consist of the sum of the FLOPs required for both the forward and backward passes. The necessary formulas for a convolutional layer (with similar derivations for linear layers) are derived in equations (13)–(17) of our submission:  \n\n- **Vanilla training:**  \n  - Forward FLOPs: Defined in Eq. (17).  \n  - Backward FLOPs: Defined in Eq. (16).  \n\n- **ASI:**  \n  - Forward FLOPs: $O_{\\text{vanilla}} + O_{\\text{ASI}}$, where $O_{\\text{ASI}}$ is defined in Eq. (14).  \n  - Backward FLOPs: Defined in Eq. (15).\n\n- **$\\text{HOSVD}_\\varepsilon$:**  \n  - Forward FLOPs: $O_{\\text{vanilla}} + O_{\\text{HOSVD}\\varepsilon}$, where $O_{\\text{HOSVD}\\varepsilon}$ is defined in Eq. (13).  \n  - Backward FLOPs: Defined in Eq. (15), which is identical to ASI. \n\nThe key reason ASI achieves lower training FLOPs compared to vanilla training is that its low-rank gradient calculation significantly reduces computational costs. This reduction more than compensates for the minor overhead introduced by mapping activation maps to subspaces during the forward pass (see Fig. 5).", "claims_and_evidence": "Yes. The claims are easy to follow and the evidence is clearly supported.", "methods_and_evaluation_criteria": "Yes. The authors use typical on-device learning models and image classification tasks.", "theoretical_claims": "Yes. The rank selection and backward pass are clearly formulated.", "experimental_designs_or_analyses": "Yes. The experiments are correctly configured and the insights obtained from the experiments are clearly explained.", "relation_to_broader_scientific_literature": "This paper is strongly related to the on-device learning design and its deployment.", "essential_references_not_discussed": "No. I think the references are adequately covered.", "other_strengths_and_weaknesses": "The authors propose a rank selection strategy to determine the most suitable ranks for each fine-tuned layer under a given memory budget constraint before training begins. This is a practical idea to reduce activation memory usage and overall training FLOPs.", "other_comments_or_suggestions": "Overall, I think this paper is interesting and its technical depth is fine in most aspects.", "questions_for_authors": "In experiments, the proposed method reduces overall training FLOPs up to 1.86× compared to vanilla training,. Could you please give more details on how to measure the training FLOPs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A."}}, {"review_id": "FMRLMWZVGr", "reviewer": "Reviewer_N3Rd", "summary": "The paper proposes a novel shortcut approach—termed ASI (Activation Subspace Iteration)—that aims to improve the efficiency of on-device learning by addressing the activation memory bottleneck during backpropagation. The key idea is to perform a single subspace iteration with a “warm start” for low-rank decomposition of activation maps, accompanied by a rank selection strategy based on an activation perplexity measure. Experiments on compact networks like MCUNet and other small architectures demonstrate substantial reductions in memory usage and FLOPs compared to vanilla training and traditional HOSVD-based methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: How our rank selection strategy might scale to transformer-based models or LLMs?**\n\nOur rank selection strategy is fully applicable to transformer-based models and LLMs with billions of parameters, and it incurs only a **one-time cost**.\n\nThe main idea of our strategy involves performing a brute-force search over a 2D search space, where one dimension represents the number of layers to fine-tune and the other represents the number of explained variance values considered. Increasing or decreasing either of these factors directly impacts the search time for the optimal solution. The total number of parameters or model architecture does not directly affect our rank selection strategy. As long as backpropagation is required for training, it is still applicable.\n\nFurthermore, the choice of rank selection algorithm is not the key point of ASI (*see our response to Weakness 2 of Reviewer dju2*).\n\n**Question 2, Weakness 2 and Suggestion 3: Apply ASI to transformer models and LLMs.**\n\nWe have conducted additional experiments as follows:\n\n- **Transformer models**: We applied the same training strategy with ASI implemented on the linear layers within the MLP blocks:  \n  - **Swin Transformer for classification tasks**: We used Swin Transformer pretrained on ImageNet-1K from the PyTorch library and evaluated it on five different downstream datasets: CIFAR-10, CIFAR-100, Pets, Flowers, and CUB.  \n  - **TinyLlama on BoolQ dataset**: We extended our technique to TinyLlama, a large language model (LLM) with 1.1B parameters, using the BoolQ dataset, which consists of yes/no questions. Since TinyLlama is a very large model, applying $\\text{HOSVD}_\\varepsilon$ directly would be computationally infeasible given our available resources, making it impossible to construct a budget for comparison. Therefore, instead of using the proposed rank selection strategy, we set a fixed expected rank of **20** for ASI and compared it against vanilla training.  \n\n- **Image segmentation**: We applied the same training policy and downstream datasets as used in *(Nguyen et al., 2024)* and *(Yang et al., 2023)*.\n\nThese results are available at the following anonymized link: [https://imgur.com/a/qOsfYU5](https://imgur.com/a/qOsfYU5). Overall, ASI gives similar results to those in our submission. It outperforms other methods in both computational complexity and activation memory consumption. We will add these new results to the camera-ready version.\n\nPlease note that in addition to MCUNet, we also conducted numerous experiments with MobileNetV2, ResNet18, and ResNet34. The results of these experiments are presented in Tables 1 and 2 of our submission.\n\n**Question 3, Weakness 1, Suggestion 1 and Suggestion 2: Writing style and how ASI works.**\n\n- **Writing style:** Thank you for your feedback. We will carefully consider it and make appropriate revisions.\n\n- **How ASI works:** Regarding the steps in our method, as described in Fig. 1:\n  - **Step 1:** Before the training begins, we measure the perplexity $\\mathcal{P}$ (defined in Eq. (7)) of the pretrained model for each predefined explained variance by feeding a minibatch of pretrained data. We then save $\\mathcal{P}$ to a file.\n  - **Step 2:** Using the saved file, we run a brute-force search algorithm to find the optimal ranks for each fine-tuned layer. The optimal rank is the one that satisfies Eq. (8) and Eq. (9), i.e., the rank that minimizes the total perplexity while ensuring the memory does not exceed the given budget $\\mathcal{B}$.\n  - **Step 3:** We use ASI to \"compress\" the activation map of each fine-tuned layer into a subspace with the rank corresponding to the one found in Step 2. These ranks remain fixed throughout the training process.\n\n\n**Question 4: Challenges in applying ASI to other architectures.**\n\nWe do not foresee any issues applying ASI to models architecturally different than those we have tested. As long as the model requires backpropagation and includes convolutional/linear layers, ASI can still be applied. Would you suggest some architectures where you think ASI might face challenges? We would be happy to hear your thoughts.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "1. The paper’s method centers on a single subspace iteration with a warm start for decomposing activation maps, along with a rank selection mechanism guided by activation perplexity.\n2. Evaluation criteria include reductions in activation memory, computational FLOPs, and training latency.\n3. A concern is that while the approach shows promise on compact models, the paper does not sufficiently clarify its applicability or scalability when deployed on much larger models such as transformers or large language models (LLMs).", "theoretical_claims": "1. The authors derive and analyze the computational complexity of ASI compared to HOSVD-based methods, providing equations for memory savings and speedup ratios.\n2. Although the theoretical framework is solid, the paper’s writing is dense and could benefit from clearer explanations to enhance reader comprehension.", "experimental_designs_or_analyses": "1. Experiments are conducted on several standard benchmarks and on a Raspberry Pi 5 to validate the on-device feasibility.\n2. However, the experiments focus on MCUNet and similar small networks, leaving open the critical question of whether the rank selection and decomposition strategy can scale to transformer-based models or LLMs with billions of parameters.", "relation_to_broader_scientific_literature": "1. The work builds on prior research in low-rank decomposition for weight and activation compression, including methods like HOSVD and LoRA.\n2. It contributes by proposing a shortcut approach (ASI) that potentially reduces computational overhead.\n3. However, similar techniques have been explored in previous works, and the novelty may be limited if the approach does not generalize beyond small-scale networks.", "essential_references_not_discussed": "The paper might benefit from discussing more recent works on scaling low-rank decomposition techniques to transformer architectures and large language models, as these are critical for assessing the broader impact of the proposed method.", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of using a single subspace iteration with a warm start for activation compression is interesting and shows promising efficiency gains on small networks.\n2. Experimental results demonstrate significant improvements in memory and computational efficiency on resource-constrained devices.\n\nWeaknessesL\n1. The abstract and overall writing need improvement; the description of the method is not as clear or detailed as it should be, making the paper hard to read.\n2. The applicability and scalability of the approach are not clearly addressed, particularly when considering transformer-based models or large language models. If the rank selection strategy does not scale to LLMs, the impact of the work would be considerably diminished.", "other_comments_or_suggestions": "1. Consider revising the abstract to include more specific methodological details about ASI and the rank selection strategy.\n2. Improve the overall writing style and structure to enhance clarity and readability.\n3. It is crucial to discuss the limits of the method’s applicability; a dedicated discussion on whether and how the approach could extend to transformer architectures would greatly strengthen the paper.", "questions_for_authors": "1. Could you elaborate on how your rank selection strategy might scale to transformer-based models or LLMs with billions of parameters?\n2. Have you conducted any preliminary experiments on larger-scale models beyond MCUNet? If so, what were the outcomes; if not, what are your expectations?\n3. The paper would benefit from a clearer description in the abstract regarding the methodological details of ASI—could you provide a more detailed outline of the steps involved?\n4. Do you foresee any challenges in applying your approach to models with significantly different architectures than those tested in your experiments, and how might you address them?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Activation Subspace Iteration (ASI), a novel technique for addressing memory bottlenecks in on-device learning through efficient compression of activation maps. The work introduces several key innovations, including a perplexity-based rank selection strategy and a single subspace iteration with "warm start" that replaces traditional HOSVD-based compression methods.

The paper has received generally positive reviews, with scores ranging from weak accept to accept. The reviewers consistently praise the paper's practical significance and thorough empirical validation. The method demonstrates impressive results, achieving up to 120.09× reduction in activation memory usage and 1.86× reduction in training FLOPs compared to vanilla training.

During the rebuttal phase, the authors have effectively addressed several key concerns raised by the reviewers. They provided additional experiments demonstrating ASI's applicability to transformer models and LLMs, including results on Swin Transformer and TinyLlama with 1.1B parameters. The authors also clarified their FLOPs calculation methodology and provided detailed insights into the rank selection strategy's scalability.

The authors acknowledge certain limitations of their approach, such as the use of brute-force search for rank selection and potential accuracy drops when fine-tuning deeper networks. However, they have provided reasonable justifications for these design choices, noting that the rank selection is computationally feasible for typical on-device learning scenarios where models are relatively small.

Some writing issues were noted, particularly regarding citation style and abstract clarity. The authors have committed to addressing these in the camera-ready version. The theoretical foundations are sound, though some reviewers suggested the presentation could be more accessible.
Decision: Accept (poster)