Efficient Resource-Constrained Training of Transformers via Subspace Optimization
Authors: Le-Trung Nguyen
Enzo Tartaglione
Van-Tam Nguyen
Venue: ICLR 2026 Oral (2026)
Keywords: Deep Learning
Computer Vision
Compression
Low rank
Primary area: optimization
TL;DR: We propose a novel method that enables training vision transformer models within a low-rank subspace to optimize computational resources, making on-device learning practically feasible.

Abstract: As AI increasingly shapes daily life, energy consumption and data privacy have become pressing concerns. On-device learning trains models directly on edge devices, cutting energy consumption and safeguarding data privacy. However, the expanding scale of modern neural networks creates a major obstacle for on-device training. Although prior work has concentrated on compact convolutional architectures, we instead apply subspace-based training to transformer models. Motivated by the idea that a model's essential information lies in a fixed subspace, we introduce Weight-Activation Subspace Iteration (WASI), a method that mitigates the memory bottleneck of backpropagation and boosts inference efficiency in transformer models by restricting training to this subspace. Our results demonstrate that WASI maintains accuracy comparable to vanilla training while reducing memory usage by up to $62\times$ and computational cost (FLOPs) by up to $2\times$. On a Raspberry Pi 5, WASI achieves roughly $1.4\times$ faster training and inference than vanilla training. The code is available at https://github.com/Le-TrungNguyen/ICLR2026-WASI.git.

Published as a conference paper at ICLR 2026
EFFICIENT RESOURCE-CONSTRRINNE TRAINING OF TRANSFORMERS VIA SUBSPACE OPTIMIZATION
Le-Trung Nguyen Enzo Tartaglione Van-Tam Nguyen LTCI, Télécom Paris, Institut Polytechnique de Paris, France (name. surname ) lecom aaraii ris.sse fr
ABSTRACT
As Al increasingly shapes daily life, energy consumption and data privacy have become pressing concerns. On-device learning trains models directly edge devices, cutting energy consumption and safeguarding data privacy. However, the expanding scale of modern neural networks creates major obstacle for device training. Although prior work has concentrated on compact convolutional architectures, instead apply subspace-based training to transformer models. Motivated by the idea that model's essential information lies in fixed sub- space, we introduce Weight-Activation Subspace Iteration (WASI), a method that mitigates the memory bottleneck of backpropagation and boosts inference effi- ciency transformer models by restricting training to this subspace. Our results demonstrate that WASI maintains accuracy comparable to vanilla training while reducing memory usage by up to 62x and computational cost (FLOPs) by up to 2x. On Raspberry Pi 5, WASI achieves roughly 1.4x faster training and inference than vanilla training. The code available at /htips:/github.comm// YmangNuuyem/CCLR2220WWWWW
INTRODUCTION
On-device learning has recently emerged as promising re- search direction, enabling deep learning models to be fine- tuned directly on resource-constrained edge devices. This ap- proach addresses critical issues such as privacy and energy consumption, improves scalability, and places control of AI capabilities directly "in user's hands" (Dhar et al., 2021). Prior work on on-device learning has largely focused on vision tasks using convolutional neural network models, primarily because of their compact architectures (Lin et al.] [2022] Nguyen et al.
Sabopage
Subepsez
I
R
&
A
Ausa
39%
39%
2024 Yang al. 2023b Quélennec al. 2024 Bragagnolo
et al. 2022 Nguyen et al. 2025).
In many real-world applications, however, transformer-based models have become the de facto choice due to their unique architectural mechanisms (Vaswani al., 2017). Specifically, these models employ efficient forward propagation through the composition of linear layers. process large-scale data in par- allel, and alleviate the vanishing gradient problem thanks to self-attention - key advantages that make them well-suited for handling long-range dependencies, whether in extended text sequences or high-resolution images. Notable examples of such models include GPT (Brown al. 2020). Gemini (Team et al. 2023). LLaMA (Tourron jtt al. 2023), and DeepSeek (Liu et al. 2024a). Nevertheless, these mechanisms make training and deployment of transformer models resource- resource-intensive.. This is even worse when considering the device learning context, where models need to be trained on separate edge devices and are often resource-constrained.
RL
Original space
Original space
Ferwanl
Booksoed
selivation
Loweaads
Figure 1: Overview of WAST in single training iteration.

---

Published as a conference paper at ICLR 2026
A significant fraction of training costs arises from backpropagation, especially the memory and com- putations needed for storing tensors in model layers (Lin et al. 2022). Various research has emerged to address the inefficiencies of backpropagation and enable learning directly on devices. For in- stance, Lin et al. 2002)) demonstrated the feasibility of fine-tuning a predefined subnetwork under 256KB memory constraint device while still maintaining competitive performance. Quélennec iet al. (2024) took this further by dynamically adapting the subnetwork during training rather than relying on static one, leading to better accuracy within tight memory budgets. Beyond the scope of on-device learning, many methods aim to reduce training overhead through garameter-efficien approaches, such as LoRA (Hu et al. 2022) and its variants (Xu et al., [023] Zhang et al. [023] Hayou et al., 2024 Liu et al. 2024b). While these techniques successfully limit the number of pa- rameters updated at training time, they often overlook the cost of storing intermediate calculations (activation maps). Nguyen et al. (2024) address this by compressing activation maps under a con- trolled information-loss constraint, but lack robust memory budget control and incur considerable compression overhead.
None of these methods enhances the neural architecture itself, and inference proceeds as usual, resulting in high deployment costs edge devices. This issue has been further addressed by ASVD Yuann et al., 2023) and FWSVD (Hsu et al. (022), which employ truncated Singular Value Decomposition (SVD) to decompose the model architecture, but lack theoretical basis for choos- ing which singular values to truncate. Subsequently, SVD-LLMM (Wang et al.) 2024) was developed overcome this limitation and outperforms the aforementioned approaches. However, these meth- ods are specifically designed for large language models (LLMs) and are not readily applicable to all vision transformer-based models (see Appendix, A.44). Another similar effort, ESPACE (Sakr & Khailany 2024), requires access to downstream dataset, which is not feasible in on-device learning scenarios.
Inspired by prior studies on the stability of parameter subspaces during fine-tuning (Radiya-Dinn Dixit Wang 2020 & Zhang 202T), we present WASI (Fig. [ the first method for efficient model- setivaaioo-decomposiitonnn.
training. WASI enables transformer models to be fine-tuned and exccuted entirely in a low-rank representation, substantially reducing hardware costs and making vision transformer tasks feasible on edge devices. We assess its effectiveness on vision trans- former models, including the Swin Transformer (Swin (Liu et al. 2021), the Vision Transformer (ViT) (Dosovitskiy et al. 2020)), and even TinyLlama (Zhang et al. 2024).
Our main contributions are summarized as follows.
Based on the previous studies, we formulate that the essential information of model pa- rameters resides in a stable subspace throughout fine-tuning (Sec. 3.3 which is then veri- fied in Sec. 4.2
Leveraging this hypothesis, we propose Weight-AAttivtioo Subspace Iteration (WASI) in Sec. 3.3 to effectively compress the model architecture under controlled information-loss constraint.
We showcase the effectiveness of our approach through extensive experiments on multiple tasks (Sec. 4.3 and Sec, 4.4)
2 RELATED WORKS
In this section, we review low-rank decomposition techniques applied to two key components of deep learning models: model weights and activation maps. Other research directions such as compact model design, quantization, sparsification, and knowledge distillation also exist, but they fall outside the scope of this work-low-rank decomposition (Cheng et al.) 2017: Deng et al. 2020
Therefore they are not discussed here (see Appendix A.5 for details).
Low-rank Decompasition for Model Weights. Low-rank approximation methods for model weights have been extensively studied and can generally be categorized into two main approaches: Low-rank Adapters and Low-rank Models.
LoRA (Hu et al. 2022 is the most prominent example of the first category, which introduces an additional low-rank adapter while freezing the original model architecture. This strategy can reduce the number of trainable parameters by up to four orders of magnitude, but comes with two notable drawbacks. During training, memory usage grows because both the frozen weights and the new
2

---

Published as a conference paper at ICLR 2026
adapter must co-exist in memory. At inference time, the adapter is merged back into the model, resulting in inference performance that is identical to the original model, and thus losing the com- putational advantages of low-rank decomposition.
Low-rank Models are an alternative line of research that factorizes the weight matrices themselves and trains only the low-rank components, enabling inference to directly on the compressed repre- sentation. Methods such as ASVD Yuaan et al. (023) and FWSVD (Hsu et al. (022) achieve this by applying truncated SVD to each layer. These approaches, however, lack theoretical link between the truncation loss and model performance loss, which is latter addressed by SVD-LLM (Wang et al.) 2024). It important note that, except for SVD-LLM, all aforementioned methods are specifically tailored for LLMs, and even SVD-LLM cannot be directly applied to all vision transformer-basse models with activation maps of four or more dimensions (see Appendix (A.4).
Low-rank Decomposition for Activation Maps. In addition to model weights, activation maps are a major contributor to memory consumption during training. Gradient Filter (Yang al. (023b) is pioneering work that addresses this issue in on-device learning by generating approximated ver- sions of activation maps through pooling operations with a predefined patch size, aiming to reduce memory usage and FLOPs during fine-tuning. However, this method is limited to convolutional models, and also has the drawback of the accumulated errors as fine- tine-tuning progresses deeper into the model (Nguyen
al., 2024). To overcome this drawback, Nguyen et al. (2024) introduced Activation Map Compression (AMC), which applies High-Order Singular Value Decomposition (HOSVD) to compress activation maps while controlling the information loss via threshold pa- rameter e. While AMC achieves impressive memory savings to 120x, it incurs significant com- putational overhead due to the need for full HOSVD at every iteration. Additionally, the varying ranks required to meet the error threshold lead to fluctuating memory usage, which complicates de- ployment on devices with fixed memory budgets.
Activation Subspace Iteration (ASI) (Nguyen et al. 2025) addresses both of these issues. Instead of controlling the reconstruction error, ASI fixes the activation ranks using a perplexity -base heuristic. This approach stabilizes memory usage throughout fine-tuning and allows for replacing the expen- sive HOSVD with subspace iteration. As a result, ASI preserves the high compression ratio of AMC while reducing computational cost by up 252. x5 On Raspherry Pi 5, fine-tuning with ASI is 1.56x faster than vanilla training when being tested on a highly compact convolutional model. Beyond this scope, L.P-WWHT (Yang et al. 2023b) has also been explored. However, it focuses solely on reducing computational cost during training by applying the Walsh-Hadamard Transfor- mation tensors in gradient computations, and does not address memory bottlenecks.
Our proposed WASI overcomes the limitations posed by prior works. Hypothesizing the stabil- ity of the essential subspace of model weights, we introduce novel method that simultaneously compresses the model architecture and activation maps while carefully controlling information loss throughout the fine-tuning process. This capability makes it feasible to fine-tune transformer-bast models in on-device learning scenarios.
3 METHOD
In this section, we first identify the computational bottlenecks of training and inference (Sec. Next, we review how activation maps can be efficiently compressed (Sec. 3.2 We then introduce a compression-aware-r training strategy for both model weights and activation maps that controls information loss (Sec. 3.3). Finally, we analyze the computational complexity of our method and discuss its practical advantages (Sec. 3.4).
3.1 BOTTLENECKS IN TRAINING AND INFERENCE
Consider a deep transformer- based model, where denotes the index of linear layer. This layer is represented by weight matrix W; RO;x1; which takes input tensor A, E BxN,i and produces an output tensor A++1 E x Here, B is the batch size, N; is the sequence length (or number of tokens), 4, is the input feature dimension, and O is the output feature dimension. We denote the dimensionality of the input as D, {B, N;, 1.).
During the forward pass (similarly in inference). the output of this layer is computed as:
A++1 A.W.T.
(1)
3

---

Published as a conference paper at ICLR 2026
Algorithm Weight Subspace Iteration WSI at iteration
1: Input:
Weight Wi,()) at iteration
Explained variance threshold E E [0, 1].
2: Function:
3: if = then
4: I1,00: Ra,0)) SVD (Weice).
(see Eq.
Eq. 6
and Eq.
5: else
6:
Ri W s.10) Li,(1-1)
7: Li,(() Orthogonalize (We()) RoT1
(Using Gram- (oum-Schmidt)
8:
endif
9:
return Li(()- Rt,(1)
where denotes the matrix transpose. Eq. presents batch matrix multiplication applied over the last two dimensions of A;; that is, for each sample in the batch and each token, matrix multiplica- tion is performed between 1 x 1; vector and the transposed weight matrix of size 1₁ x O,
Similarly, in the backward pass the chain rule of backpropagation is computed follows:
OC
Oc
0Ai+1
Oc Ai.
(2)
Ow:
0A1+1
OW;
0Ai+1
0C
0C
0AH+1
OL
0A:
0A
0A:
8Ai+1
where is the loss computed at the output of the model. Apparently, to compute Sw, and 0C
0c
0.4,
during the backward pass, A, and W; must be stored during the forward pass. The large size of these tensors is the primary cause of memory bottlenecks during backpropagation (Lin et al. 2022). Additionally, also contributes to high inference costs, as multiplying between large W; and A; requires significant computational resources.
3.2 ACTIVATION SUBSPACE ITERATION
Here, we recap how activation maps can be decomposed by subspace iteration. Given an activation memory budget B, ASI performs brute-force optimization before fine-tuning find an optimal rank vector r; E N3 for each layer such that the resulting memory does not exceed B. Then, for each mode m E (1, the activation map A, is unfolded into matrix Asm R xham where (ar,m, )).m)
(i.m. [ Da.).
Vogels et al. (2019) showed that warm- startettrred subspace iteration matches SVD performance on stable tensors much lower cost. Exploiting the stability of activation maps during fine-tuning, ASI applies this technique to each Ai.. The resulting approximation takes the form of a Tucker decomposition (Tucker 1966):
A 2 S X1 0.0) X2 0 X3 0,(0)
where S; R is the tensor, representing a compressed version of A,, and each factor matrix 0.(") E gai. xra. contains the principal components along the mh mode.
Consequently, instead of storing all Oxpace (IT Di.m)
elements of A,, ASI reduces the storage
requirement to Ospacc (II= Fim + Dim Dim
Details of the algorithm can be found in Appendix A.2
3.3 WEIGHT ACTIVATION SUBSPACE ITERATION
Stability of Model Parameters Subspace. While prior work has shown that parameeereeeriiie models in fact reside in low-dimensional intrinsic subspace (Aghajanyan al. 2020 Li et al.
4

---

Published as a conference paper at ICLR 2026
D
O; = 32 m [1,3]
Dim
O, 128
m E [1.3]
D.m O 64
[1.3]
Dum
O, 256
[1,3]
10
10y
Creiniig
Cinfinenes
S
50
Snfureane
100
10!
10a.
10
0
0
9
13
9
13
1
13
1
13
K; Fú
12
F13
For the linear layer with single data batch of size B, given varying dimensions of W; and A; and different values of rims Ciaining and Cinterence illustrate the evolution in compression rates for training and inference, respectively; while Smaining and Sinforence forecast the speedup ratios for these processes.
Figure 2:
2018), we further observe that fine-tuning introduces only minor updates each training step due to the use of a small learning rate. As a result, our key insight is that the intrinsic subspace re- mains relatively stable after each training iteration and can therefore be reused the following one (confirmed in Sec. 4.2 Fig. B}. This is supported by the findings of Radiya- Dixit & Wang (2020) and [i & Zhang (2021). who showed that the fine-tuned models are close in parameter space the pre-trained counterpart.
Weight Subspace Iteration. Besides activation maps, model parameters (weights) W; are another major source of memory bottlenecks during training. To address this, we propose a low-rank weight decomposition strategy that projects each weight tensor into a smaller subspace at every training iteration, thereby preserving the meaningful subspace. The method works follows:
Step 1. For the weight tensor W; at layer i, its SVD form is given by:
(5) where Σ; is a diagonal matrix containing r; singular values St.jE(1.r:) and r; is the rank of W. As shown in Eq. truncating U, E,, and Vł" inevitably introduces error into ac which then propagates backward during training. other words, low-rank decomposition the weights affects model convergence due to the accumulation of truncation error.
W: = -UE.V
E Σ, EROx V VRRLL
aA,
To control this effect, we constrain the truncation error by enforcing a target explained variance threshold similar to the strategy used in Nguyen al. (202)). Specifically, measure the variance explained by the jth singular value as oi 5/g/ 2x 7 Assuming the singular values are sorted descending order (ss.j 2 Si,k, Vj S k), the optimal rank is defined as the smallest K, integer K; E rd such that 2j=1 of,1 2 We then identify the essential subspace with rank K; of W,, represented by L, and R such that:
: W. L,R,,
(6)
where
L,
R² VT ((K)) U2,(K)) E ROOK, E.( RKiK, V(,K,) BLXc..
(7)
Step 2. Performing full SVDs at every iteration, however, is computationally prohibitive for on- device training (Nguyen et al. 2025). Leveraging the stability of parameter subspaces established above, Σ; can be expected to remain relatively stable. Thus, for fixed E, the optimal rank K; should also remain consistent (verified in Sec. 4.2). Consequently, instead of recomputing the SVD at every iteration, we compute it once at the beginning to determine the essential subspace. Subspace iteration is applied during training to minimize computational overhead. We refer to this method as Weight Subspace Iteration (WSI), with the full procedure outlined in Algorithm
Weight-Activation Subspace Iteration. While WSI reduces weight-related overhead, activa- tion maps also dominate memory usage in backpropagation (Sec. 3.1)) Previous work has shown that most of the energy in activation maps is concentrated the first few principal components across all modes (Nguyen et al. 2024). Such distribution makes them highly compressible while
5

---

Published as a conference paper at ICLR 2026
still achieving high-fidelity reconstruction (confirmed in Sec. 4.2 Fig. and Sec. 4.3 Motivated by this property, we propose a unified framework in which both weights and activations are com- pressed under stable low-rank subspaces. Specifically, strategy that determines r; by minimizing memory usage under a target pre-tuning perplexity, rather than relying on fixed budget B, thereby reducing the search cost from exponential to linear (Appendix A and (ii) an extension to support 3D activation tensors (Ap- pendix A.I
redesign ASI with two improvements: (i)
a
Together, WSI and ASI form the proposed Weight-Activation Subspace Iteration (WASI), novel framework for low-rank training that jointly leverages the stability of both weights and activations. Under this scheme, the forward and backward passes are computed as follows:
A = A Li'
(8)
ac fLR
Oc A 0Ait1
(9)
OW;
Oc
Oc
L,Ri,
(10)
04,-
0Ai+1
where fLR(.) denotes linear operator applied in the low-rank space (see Appendix A.I With learning rate the weight update is then computed as:
dC
LRR; L, + O.,
(11)
3.4 MEMORY EFFICIENCY AND COMPUTATIONAL COMPLEXITY ANALYSIS
For simplicity, we assume that the same optimal rank is applied to both Ai and Wi. By varying this value, we can predict total memory usage and speedup for WASI compared to vanilla training (Fig. 2 As model size grows and the optimal rank decreases, WASI delivers greater memory compression (Ciminingg Cinference) and speedup (Smining. Sinference). property especially valuable in on-device learning where models are typically over-parameterized and reside low-dimensional subspaces (Aghajanyan et al. 2020; Li et al., (018). Conversely, the optimal rank increases, WASI's computational cost approaches that vanilla training, and the speedup ratios converge to 1, reflecting the upper bound set by vanilla training.
Detailed derives Cusining, Cinferences Suaining, and Sinterence can be found in Appendix A.3
EXPERIMENTS
In this section, present experiments designed to demonstrate the effectiveness of WASI. We begin by outlining the experimental setup in Sec. 4.1 Then, in Sec, 4.2 we conduct experiments to validate the assumptions introduced in Sec. (3. and Sec. 33. Sec. [4.3 compares WASI with various state-of-the-art methods across multiple datasets. Finally, all methods are evaluated in real-world deployment scenario (Sec. 4.4 All simulation experiments are conducted using PyTorch 1.13.1 on an NVIDIA Quadro RTX A4500 with 20 GB of VRAM, while on-device experiments are run on Raspberry Pi 5 equipped with a Cortex-A76 CPU and 8 GB of RAM.
4.1 EXPERIMENTAL SETUP
Our goal is to enable on-device training of transformer models, where networks pretrained on large- scale datasets are fine-tuned locally with task-specific data (Murshed al., 2021). We evaluate
WASI on image classification using ViT and Swin'T, both pretrained on ImageNet- - K
(Deng et al.)
2009) across five downstream datasets: CIFAR-10/106 (Krizhevsky) 2009). CUB (Wah al. 2011).
Flowers (Nilsback & Zisserman 2008), and Pets (Zhagg et al. (022).
Comparisons are made against three directly comparable baselines at the time of conducting experi ments: ASI, SVD-LLM, and vanilla training (as discussed in Secs. 2 Appendix A.5). We mea- sure memory and computation costs during training and inference, focusing on linear layers within multi-perceptre blocks for fair comparison with previous methods (extended results with attention layers in Appendix B.3). All experiments are run with the same set of hyperparameter, detailed in Appendix B.1

---

Published as a conference paper at ICLR 2026
700
Topl Validation Accuracy (96)
WSI
SVD
Singnlar Value Index
800
Singular Valuee
70
500
40-
16:30 higher
Ace
30
FLOPs
10
20
30
40
Epoch
FLOPs
(a)
(b)
Figure 3: When fine-tuning VIT on the Pets dataset, (a) illustrates the evolution of singular values of Ws across epochs; (b) compares WSI and full SVD in terms of accuracy and training FLOPs under varying explained variance thresholds, E {0.4, 0.5, 0.6, 0.7, 0.8, 0.9}.
Mode
Mode
Mode
Dook rod Veduin
Exparnal Vettta
0.6
0.6
0.4
0.8
0.4
Eaph
0.2
0.2
0.0
0.0
20
20
20g
Latrr luceer
10
100
Singadan Value Indees
Lator Iudes
10
40
Singalas Value
fudes
Laowr Budos
G0U
Ssoeetoo Vilcon
800
(a)
(b)
(c)
Figure 4: Explained variance each singular value of A, across all of its modes when fine-tuning VIT on the Pets dataset.
4.2 PRELIMINARY RESULTS
In these experiments, we focus on fine-tuning VIT model using Pets dataset.
Stability of Layer Ranks. We apply truncated SVD the weight tensors of the linear layers within VIT's MLP blocks at each training iteration. We constrain the decomposition by setting & = 0.8 and monitor the layer ranks K, throughout the course of training. As shown in Fig. 3a] observe that the ranks exhibit remarkable stability across epochs. This observation validates our insight in Sec. 3.3 confirming the stability layer ranks during training.
WSI vs SVD. Next, we compare two strategies: (1) reapplying truncated SVD at every training iteration, and (2) WSI. We evaluate their performance across range of values specifically, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 with each value represented by different marker in Fig. 35 The results demonstrate that incorporating subspace iteration through WSI leads significant reduction in computational complexity compared to performing a full SVD at every iteration. Specifically, WSI requires x.36x fewer FLOOPs than SVD to achieve the same level of accuracy. Moreover, when both methods are constrained to use the same amount of FLOPs, WSI outperforms SVD by approximately 35% in terms of accuracy. This result verifies that reusing the subspace in subsequent training iterations does not degrade model convergence.
Explained Variance Distribution of Activation Maps. Fig. illustrate the explained variances J.j.m of each singular value in mode of the activation map A,, As anticipated in Sec. 3.3 most activation-man energy lies in the first few singular values, which capture the key information during fine-tuning.
7

---

Published as a conference paper at ICLR 2026
WASI
ASI
SVD-LLM
Vanilla
Accurecy (%)
80-
80-
80-
102
107
10P
1011
2012
1011 Inference FLOPs
Training Mem (MB)
Inference Mem (MB)
Training FLOPs
Figure 5: Resource consumption during fine-tume tne-tunnng and inference ViT on the CIFAR-10 dataset. Each marker in the plots corresponds to different compression rate, with the red diamond indicating vanilla training.
Pets
Flowers
CUB
CIFAR-10
CIFAR-100
Avcuracy (96)
220F Sxietaxct" 5x307
0x:00
20
1x09t Soctphi Snitf
Training Mem (MB)
Inference Mem (MH)
Traiing FLOPs
Inference FLOPs
Figure 6: Resource consumption when applying WASI for fine-tuning and inference of Swin'T across different datasets. Each marker along the curves represents a different compression rate, while the final marker each curve corresponds to vanilla training.
4.3
MAIN RESULTS
VIT CIFAR-10. Fig. presents the results of fine-tuning a ViT pretrained on ImageNet-1K using CIFAR-10. 10. Each curve for WASI and ASI contains six markers, corresponding to explained variance thresholds
(0.4.0.5, 0.6, 0.7, 0.8, 0.9) from left right. The red diamond indicates vanilla training, and for fairness, the same compression ratios are applied to SVD-LL..
WASI achieves up to 100x higher memory efficiency than SVD-L.LM at similar accuracy, owing to its avoidance of LoRA adapters. Its accuracy also improves steadily as e increases. In contrast, at the lowest compression rates (last two markers), SVD-LLM consumes even more memory than vanilla training because of the overhead of storing sub-layer activations.
In terms of computation, LoRA adapters allow SVD-LLM to achieve the lowest FLOPs, followed by WASI, which jointly compresses weights and activations into low-rank subspace. Since ASI only compresses activations while keeping weights intact, its computational cost is higher, and at 0.9, it exceeds vanilla training (confirmed in Tab. 2). On the other hand, ASI maintains stable accuracy across compression rates, supporting the stability assumption discussed in Sec. 333 At inference, both WASI and SVD-LLM achieve similar memory/FLOPs savings, while ASI resem- bles vanilla since the architecture is unchanged.
SWinT Multiple Datasets. Fig. 6 compares WASI and vanilla across datasets, additional baselines are in Appendix B.3 Each marker along a curve from left to right, indicates different (0.4....) 1.0), with 1.0 as vanilla. Across all datasets, WASI consistently provides better sccuracy-efficienee trade-off. At 0.9. it matches vanilla accuracy while cutting memory by up to 62x and FLOPSs by 1.5x, and even surpasses vanilla on CUB.
WASI on Tama. The initial goal of WASI was to enable training wansformer-basee models on edge devices, so we focused on ViT and SwinT. To test its generality, we extended our experiments to Llaya, a decoder-only transformer model. The downstream dataset used is BoolQ (Clark et al. 2019). Due to limited resources, we only fine-tune up to the last layers the model and set the WASI E to 0.1. All other training hyperparameters followes the same configuration in previous experiments. For comparison, we log the resource consumption only at the layers that are fine-tuned. The results shown in Fig.
8

---

Published as a conference paper at ICLR 2026
WASI
Vanilla
Accurecy (%)
66-
66-
64
64-
10t
10P
10
107
1013
1011
10
10
10
Activation Mem (MB)
Weight Mem. (MB)
Training FLOPs
Inference FLOPs
Figure 7: Performance WASI vs. vanilla training when fine-tuning TinyLlama on BoolQ. Each marker indicates the number of layers fine-tuned from the last layer upward: the marker closest to the y-axis of each figure corresponds to fine-tuning only the last layer, the next marker corresponds to the last two layers, and on.
WASI again outperforms vanilla: activation and weight memory drop by up 953.86 and 30.12x, while training and inference FLOPPs fall by 13.11x x and 30.27 all without accuracy loss.
Additional results, including ViT more datasets and extended baselines for Swin'T are in Ap-
pendix B.3
4.4 ON-DEVICE LATENCY
Training time
We evaluate the practical efficiency WASI resource-constrainct hardware by fine-tuning VIT on CIFAR-10 using a Raspberry Fig. rreport the average time required to com- plete a single iteration of both training and inference across different explained variance thresholds = E {0.4, 0.5, 0.6, 0.7, 0.8, 0.9}, along with vanilla training. As expected, the runtime for both training and inference under WASI increases as s be- comes larger. This trend aligns with the intu- ition that higher g values retain more informa- tion and thus result in higher-rank approxima- tions, which require more compute and mem- ory. However, despite this increase, WASI consistently outperforms vanilla training in terms of speed. For instance, even at 0.9, which corresponds to the least aggressive compression setting in this experiment, WASI remains approximately 1.4x faster than vanilla training. Thus, WASI de- livers clear benefits even when preserving much the original information.
20
Infermnce time
15-
5.
Time
5-
0.4
0.5
0.7
0.8
vanilla
Figure 8: Training and inference time per itera- tion for ViT on CIFAR-10 (batch size 128) us- ing a Raspberry 5, measured under different ex- plained variance thresholds e. The final marker on each curve represents vanilla training.
Importantly, WASI helps to reduce runtime without causing significant accuracy degradation, dis- cussed in earlier sections. This ability makes it a strong candidate for the deployment transformer- based model in real-world on-device learning scenarios, where computational resources are severely constrained. Further numerical results can be found in Appendix. B.3
5 CONCLUSION
In this work, we introduced WASI, an efficient training method for resource-constrainee fine- tuning of transformer models. Assuming that essential parameter information lies in stable low- dimensional subspace, WASI applies SVD and subspace iteration to obtain low-rank approximations of both weights and activations during each training iteration. This yields significant gains in mem- ory and computation while tightly controlling information loss.
Building on prior theory and validated through extensive experiments, WASI outperforms state- of-the-art methods, reducing training memory usage by up to 62x and achieving 1.4x speedup over vanilla training on a Raspberry Pi 5. These results show the potential of WASI for enabling on-device learning with transformers, domain traditionally dominated by CNNs. While exper- iments focus on transformers, the underlying principles apply broadly to any neural network trained with backpropagation.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
Part of this work was funded by Hi!PARIS Center on Data Analytics and Artificial Intelligence, by the European Union's 's Horizon Europe Research and Innovation Programme under grant agree- ment No. 101120233 (ELIAS European Lighthouse of Al for Sustainability) and No. 101 120657 (ENFIELD European Lighthouse Manifest Trustworthy and Green AD), by the French National Research Agency (ANR) in the framework of the IA Cluster project "Hi! PARIS Cluster 2030" (ANR-23-ACCI-0055) the NF-NAI project (ANR-22-PEFT-000) and NE-FITNESS project (ANR- 22-PEFT-0007) part of France 2030.
REPRODUCIBILITY STATEMENT
Detailed description of our algorithm is provided in Sec. 3.3 Appendix and Appendix A.2 Full details of the training policy, including hyperparameters. datasets, and other configurations, are pre- sented in Appendix B.]] Code reproduce the main experiments is included in the Supplementary Material zip file. We commit to open-sourcing the complete code upon acceptance of this paper.
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13325: 2020.
Andrea Bragagnolo, Enzo Tartaglione, and Marco Grangetto. To update or not to update? neurons at equilibrium in deep models. Advances in neural information processing systems, 35:22149- 22160, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv: 2017.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:/905. 10044, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Transactions on Internet of Things, 2(3):1 49, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11000/./19999 1929, 2020.
Marawan Gamal Abdel Hameed, Marzieh Tahaei, Ali Mosleh, and Vahid Partovi Nia. Convo- lutional neural network compression through generalized kronecker product decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 771-779, 2022.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models. arXiv preprint 200 2024.
10

---

Published as a conference paper at ICLR 2026
Yen- Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, report, University of Toronto, 2009.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv: 08838, 2018
Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. Advances in Neural Information Processing Systems, 01-27249-27262 2021.
Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems, 35:22941- 22954, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 24ti:224/2.19337, 2024a.
Shih- Yang Liu, Chien-Yt Yi Wang, Hongxu Yin, Pavio Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: International Conference Machine Learning, 2024b.
low-rank adaptation. In Forty-first
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ivan Markovsky. Structured low-rank approximation and its applications. Automat- ica, 4494),99-990, 2008. ISSN 0005-1098, doi: http:://bo..or///100000// automatia.
2007.09.011. URL htts://www.scieneeirreeee http://wweececcccccccco//e/eeear
500051098070033200
MG Sarwar Murshed, Christopher Murphy, Daqing Hou, Nazar Khan, Ganesh Ananthanarayanan, and Faraz Hussain. Machine learning at the network edge: A survey. ACM Computing Surveys (CSUR), 1438) 2021.
Le- Le-Trung Nguyen, Aæl Quélennec, Enzo Tartaglione, Samuel Tardieu, and Van-Tam Nguyen, Ac- tivation map compression through tensor decomposition for deep learning. Advances in Neural Information Processing Systems, 37:130384 130407, 2024.
Le-Trung Nguyen, AEl Quélennec, Van-Tam Nguyen, and Enzo Tartaglione. Beyond low-rank de- composition: A shortcut approach for efficient on-device learning. In Forty-second International Conference on Machine Learning. 2025.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008
Aël Quélennee, Enzo Tartaglione, Pavlo Mozharovskyi, and Van-Tam Nguyen. Towards on-device learning the edge: Ways to select neurons to update under budget constraint. In Proceedings of the JEEE/CVF Winter Conference on Applications of Computer Vision pp. 685-694, 2024.
Evani Radiya-Diss Dixit and Xin Wang. How fine can fine-tuning be? learning efficient language models. In International Conference on Artificial Intelligence and Statistics, pp. 2435-2443, PMLR, 2020.
Charbel Sakr and Brucek Khailany. Espace: Dimensionality reduction of activations for model compression. arXiv preprint arXiv:2410.05437, 2024.
GW Stewart and JH Miller. Methods of simultaneous iteration for calculating eigenvectors of ma- trices. Topics in Numerical Analysis 11, 2, 1975.
11

---

Published as a conference paper at ICLR 2026
Gemini Team, Rohan Anil, Sebastian Borgeaud, Sean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arX1v:2312.11805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1rXiv:2302.1377/ 2023.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3): 279-311, 1966.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances Neural Information Processing Systems, 32, 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset, 2011.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024.
Ou Xinwei, Chen Zhangxin, Zhu Ce, and Liu Yipeng. Low rank optimization for efficient deep learning: Making a balance between compact architecture and fast training. Journal of Systems Engineering and Electronics, 2023.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xi- aopeng Zhang, and Qi Tian. Qa-lora: Quantization-awan low-rank adaptation of large language models. arXiv preprint urXiv:2309.14717, 2023.
Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of deep neural network acoustic models with 2013.
singular value decomposition. In Interspeech, pp.
Yuedong Yang, Hung- Yueh Chiang, Guihong Li, Diana Marculescu, and Radu Marculescu, Efficient low-rank backpropagation for vision transformer adaptation. Advances in Neural Information Processing Systems, 36: 16:15421-47766 2023a.
Yuedong Yang, Guihong Li, and Radu Marculescu. Efficient on-device training via gradient filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3811-3820, 2023b.
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0555 2023.
Hui Zhang, Shenglong Zhou, Geoffrey Ye Li, and Naihua Xiu. 0/1 deep neural networks via block coordinate descent. arXiv preprint arXiv:2206.0933 09379, 2022.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint urXiv:2308.03303. 2023.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.022. 02385, 2024.
Hengling Zhao, Yipeng Liu, Xiaolin Huang, and Ce Zhu. Semi-tensor product- based tensordecom- position for neural network compression. arXiv preprint
2021.
12

---

Published as a conference paper at ICLR 2026
A ADDITIONAL THEORETICAL DETAILS
A.1 DETAILS OF BACKPROPAGATION IN LOW-RANK SUBSPACE
this section, we explain the definition (IR(.) Eq. 9 For simplicity, we denote the activation tensor A; as I, the output gradient UW: AW, and the gradient with respect to the output 8.A44 as Ay.
BC
3D Activation Maps. For each activation map I E IR BxNxX applying ASI with optimal rank
r E R resulting in the following approximation:
F1
P2
F2
Io
Σ
rarrar 0 ( (
(12)
1,72
which transforms the weight gradient calculation (Eq. into:
N
0
AW
ΣΣΣΣ
Io
(13)
b=1
0
1
ΣΣΣΣΣΣΣ
maraa b.ri n,r2 1.r AY
(0 ((2)) ((9)
(14)
1= of me
By reordering and grouping terms, we obtain:
2
n,0.
Avonn (1)
b,7
(15)
b
(2)
((2)
(16)
PLifs.
n,72"
1==1
2(3)
(
((3)
(17)
71.ia
r==1
N
AW.,
Σ
2(1)
2(3)
(18)
0,0,7
m=1n==
4D Activation Maps, In some transformer-based models, such as SwinT, the activation maps are 4D: [BxHxWxx When applying ASI with optimal rank r R.. the activation map is approximated as:
r1
P2
F4
Iohhwo
Σ
ΣΣ>
S Snarirs 0 ((9) ((3) 0
(19)
Reorganizing the terms yields:
H ΣΣΣΣΣ
B
W
O
AW.Y
Ib.b.w
(20)
B w o
#
Snara b,n h.ry wors 5,r A.b.h,oo
((22) ((3) (((4)
(21)
13

---

Published as a conference paper at ICLR 2026
Again, by reordering and grouping operator, Eq.
21 becomes:
2
bun
(22)
M,,
F2 Pa.r4 h,r2'
z(2)
(23)
r2=1
ng
2
(1)
(13)
(24)
b.h,w,o
w.rs!
2(4)
2 (2)
((4))
(25)
F1.h.ra.r
H
r8
AWo
Nr.h 2
(3)
(4)
(26)
h=1
1==1
A.2 ADDITIONAL ALGORITHMIC DETAILS
The i-Mode Product Operation. The i-mode product "x," of nhh tensor G R PPxxxxX- and a matrix B E RQxP, is a n -order tensor R. E can be expressed as:
P......P-11... 9 x1 Gx:Q=
Σ =1=1
Ip Pr....pP Dg.pc
(27)
Subspace Iteration. Here, we present how can activation maps be compressed by the subspace iter- ation technique of PowerSGD (Vogels et al. 2019). The underlying idea remains the same: single step of subspace iteration (Stewart & Miller 1975h is used to obtain a fast low-rank approximation of matrix, which in this case corresponds to unfolding of the activation maps along their respec- tive modes. As noted in the original work, however, single iteration often yields an approximation that is not sufficiently accurate. To address this, Vogels al. propose initializing each step with the low-rank approximation obtained in the previous iteration.
This reuse is particularly well justified in activation maps. While activation maps evolve as the net- work parameters are updated, the changes across consecutive iterations remain small. This stability arises from the Lipschitz continuity of activation functions (Virmaux & Scaman 2018) together with the incremental nature of parameter updates during optimization. By reusing the previous approxi- mation, the sequence of activation maps across iterations can be effectively smoothed and reduced the variance of the low-rank approximation compared to the case without reuse. A formal proof of this property is provided in (Vogels et al. 2019).
Perplexity for Activation Compression. The perplexity of activation PA, is defined as the differ- and ac ow;
ence between OW;
ac
to represent the error introduced when performing activation compres-
sion.
Estimating perplexity over compression levels requires HOSVD on four-dimensional activation ten- sors, which naively induces exponential number of rank combinations across modes and layers. To avoid this combinatorial blow-up, the search over ranks is replaced by a set E of explained- variance thresholds. Let & (0, 1 denote the E thresholds evaluated; each threshold yields consistent set of truncation ranks for all considered layers. The procedure has two steps:
Step 1. For each threshold e, &, run forward pass on held-out batch. At layer i, cache the orig-
inal activation (Ai), and its low-rank approximation (Ā.),, obtained via HOSVD with components truncated according to Ej-
Step 2. During backpropagation, compute the exact gradient ( ( ), and its approximated counter-
DC part (w, )5- The layer perplexity at threshold Ey is the Frobenius norm of their difference:
Pij = (0x6)
dc O.
(28)
14

---

Published as a conference paper at ICLR 2026
Algorithm 2 ASI for layer with set of rank E Ropt
1: Input:
Activation map A! BBxNNXO, at epoch
Target ranks for modes ri No n [I, min(ai,, dimm], where (aimm bi.m) is the shape of
at mode m.
2: Function:
3: Initialize S: - !
4: for m to do
5: A.m unfold A! along mode m, Atm E R2.,mxb
6: if then
7:
Initialize V E R Kri.m from an ii.d. standard normal distribution.
8:
else
9:
V = A i,
10:
end if
U
Orthogonalize(A (As,,
S. = St Xm U(())
end for
14: return S₁, im
with m = 1,....3
This process is repeated across all layers, resulting in perplexity matrix P E NxX and cor- responding rank tensor Rxxxxx containing the selected ranks across the modes of activation tensors for each combination of layer and explained variance threshold.
Rank Selection. Given memory budget B for activations over the fine-tuned layer set F, the goal is to choose mode-wise truncation ranks that satisfy the budget while minimizing total perplexity. A recursive backtracking routine identifies an index set J NN [1, E]] and the corresponding optimal ranks r such that
Fim
Ri.jo JE
(29)
J arg
min M.SB
Σ
P..
(30)
I J.
1=1 JEJ
where
M2
rim
Σ
Dim
(31)
denotes the activation memory induced by the chosen ranks r, for layer i. Detail ASI shown Algorithm?? 2
Rank Selection in WASI. For WASI, I is found such that:
F
3
J
arg min J
Σ
II Rij. +
Σ
Dim Risg.mm
(32)
m==1
A.3 DETAILS OF COMPUTATIONAL SPEEDUP AND SPACE COMPLEXITY
Computational Speedup. We derive the computational speedup the ratio between the total FLOPs required for vanilla training and those required by WASI.
First, the number of FLOPs required perform forward pass in vanilla training (Eq. 1 is:
Fvanilla ~ 2BN,IOO;
(33)
Meanwhile, backward pass includes Eq. and Eq. costing:
B 4BN,IO;
(34)
15

---

Published as a conference paper at ICLR 2026
In contrast, WASI performs the forward pass in low-rank subspace (Eq. 8 with complexity of:
FWASI R 24,, +
(35)
However, this does not account for the overhead from weight subspace decomposition (Algorithm) and ASI decomposition. These add the following costs:
I
Owsl =4,0,K, + 20,K?,
(36)
3 OASI = Σ (4dd'r,, + 2dr?,,)
where
Dim ; d' = D; {d}
(37)
m=1
The backward pass in WASI follows Eq. 10 cost of:
Eq.
15
Eq.
16
Eq.
and Eq. 18 with a total FLOPPs
BWAST 2BN.K BN.O%. P + T.IIINN + I,,,N,, (38)
Eq
TD
Eq
15
Eq
B
The speedup ratios between vanilla training and WASI defined as:
Fvaiilla + Bvanilla
Strainin
(39)
FWASt + Owsi + OASI + BWAST
F vanillla
Sinference
(40)
FWAST
Memory Usage. In vanilla training, the total memory consists of the memory for weights and the memory for storing activations:
((() I,O;
(41)
vanilla BN.I,
(42)
In WASI, these become:
((W) = KT(T+
(43)
WASI
3
WAST = II
(A)
Tim + Σ Dim
(44)
m=1
m=1
Thus, the memory reduction ratios between vanilla and WASI are:
(WW)
+ (As)
Cuaining
vanilla
vanilla
(45)
(W))
+ +MAA (A.
MWASI
M(W)
Cnffrrence
vanilla
(46)
(WW)
Similar ratios can derived for the case of 4D activation maps.
A.4 LIMITATIONS OF SVD-LLM
We analyze the 3D activation map Ar E RBXN, as considered throughout the paper. The core idea of SVD-LLMM is incorporate "Truncation-awaal Data Whitening" mechanism that enables direct relationship between singular values and the resulting compression loss.
To do this, SVD-LLM whitens the activation using transformation of the form S X,, where X, RRV: N; is obtained by summing A, over the batch dimension. The goal is make the trans- formed activation orthonormal, i.e., (S- X.)" = Here, S, computed via Cholesky decomposition on X.
16

---

Published as a conference paper at ICLR 2026
SVD is then applied to the transformed weight matrix W,S, resulting in U, E, and Vi- Based on the optimal rank K; found by a desired compression ratio, the smallest singular values in Σj are truncated (denoted by (K.)) to obtain two low-ranking matrices:
w = U..(K)) (KK,)) W 21/2 VI
((K.) S.
(47)
and the final compressed matrix is given by:
Wi w((") -UKK (K. ViLK,
(48)
However, a current limitation of SVD-LLM is that "Truncation-awee Data Whitening" is only defined for 3D activation maps. does not generalize to 4D activations, which are common in certain architectures such as Swin'T As a result, SVD-LLM cannot be directly applied to models that rely on 4D activation structures.
A.5
OTHER RESEARCH DIRECTIONS
Model compression and acceleration have become central topics in deep learning research, with efforts ranging from hardware improvements to algorithmic techniques. On the algorithmic side, the main directions include compact model design. quantization, sparsification, knowledge distillation, and low-rank decomposition (Cheng et al.) 2017 Deng et al., 2020). The shared goal of these approaches is to shrink the size of neural networks while keeping their accuracy intact. Among them, low-rank decomposition has gained particular attention because it combines solid theoretical foundations with practical advantages for deployment (Xinwei et al., 2023). Originally developed in systems theory and signal processing (Markovsky) 2008). low-rank methods were first used in deep learning to compress fully connected layers through singular value decomposition (SVD) (Xue al., 2013). Since then, more advanced forms such as generalized Kronecker product decomposition (GKPD) (Hameed et al. 2022). semi-tensor products (STP) (Zhao et al.) 2021), and applications to vision transformers (Yang et al. 2023a) have pushed the field forward. Progress is largely measured in terms of compression ratio, inference speed, and power efficiency, with the challenge being to achieve these gains while minimizing performance loss.
Note that the research directions mentioned above are orthogonal each other and can be combined to leverage their respective strengths.
B ADDITIONAL EXPERIMENTAL DETAILS
B. DETAILS OF EXPERIMENTAL SETUP
To ensure fair comparison, we follow the same experimental setup described in (Nguyen al. 2024). The key details are as follows:
General hyperparameters. All models are first pretrained on ImageNet-I then fine-tuned on different downstream dataset using an 80% 20% crain-validation split in 50 epochs. We use cross- entropy loss and optimize the models with SGD. The initial learning rate is set to 0.05 and decayed using a cosine annealing schedule. Momentum is set to 0, and weight decay is fixed at x 10 We apply L2 gradient clipping with a threshold 2.0. For data augmentation, we use random resizing. horizontal flipping, normalization, and mini-batch size of 128.
ASI. We follow the same strategy as proposed in (Nguyen et al. 2025). Specifically, in our main experiments, we apply AMC with range of values: 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9. For each value of e, we record the peak activation memory consumption of AMC and use it as the activation memory budget for ASI. The perplexity of ASI is also measured based these values.
SVD-LLM. Unlike WASI, SVD-LLM compresses models based on a fixed compression ratio. To compare fairly, we compute the compression ratios achieved by WASI at each and use the same ratios for SVD-LLM. We also adopt the same LoRA adapter settings as used in their original paper: 16 and rank = 8.
17

---

Published as a conference paper at ICLR 2026
Peta
F Flowers
CUB
CIFAR-10
CIFAR-
100
150
1:0
Asarney (56)
70-
50
50
50
25
25
25
10P
10P
10P
Traiing Mem (MB)
Inference Mem (MB)
Thining PLOPs
Infermnce FLOPs
Figure 10: WASI performance when fine-tuning ViT across multiple datasets. In each plot, markers from left to right represent increasing values of E; the rightmost marker corresponds to vanilla train- ing.
WASI
ASI
Vanilla
56-
96-
Accuney (%6
07
97-
96-
96-
96-
10P
1:37 S*IPaRI '5-37
10
2xter a-teacie "xte
Training Mem (MB)
Inference Mem (MB)
Training
FLOPs
Infervsiee PLOPS
Figure 11: Comparison of different methods when fine-tuning Swin'T on CIFAR-10. For each plot, markers from left to right correspond increasing values of s.
B.2 VARIANCE ACROSS DIFFERENT RANDOM SEEDS
Fig. 9 presents the results of fine-tuning a ViT model pretrained on ImageNet-IK K using WASI on Pets dataset. We test different values of from (0.4, 0.5, 0.6, 0.7, 0.8, 0.9). with each marker on the plot (from left to right) corre-
Topl Validation Ace (%)
80
Mean Accuracy + Stdl
70
60-
sponding to one of these settings. For each
50
we report the average accuracy and peak training memory usage over three random seeds (233, 234, and 235), with error bars represent- ing standard deviation.
40
30
20
As expected, increasing & leads to higher ac- curacy and greater memory usage, reflecting the trade-off between compression and per- formance. Importantly, the variance across seeds is minimal, which is reasonable given that WASI mainly uses deterministic compo- nents such as SVD, Gram-Schmidt orthogonal- ization, and matrix multiplications. Therefore, we fix the random seed to 233 in all other experi- ments.
40
60
80
100
120
Training Mem (MB)
Figure 9: WASI performance on the with differ- ent values, showing mean accuracy and memory usage across three random seeds. Error bars rep- resent standard deviation.
B.3 ADDITIONAL RESULTS
Additional Transformer-basee Results. We conduct experiments similar to those in Sec. Fig. [0 Fig. II show consistent trends with our earlier findings. Note that due to the architec- tural design of Swin'T which generates 4-dimensional activation maps, the "Truncation-Aware Data Whitening" mechanism used in SVD-LLM is not applicable (see Appendix. A.4). Therefore, this method is excluded from the experiments in Fig. II
The accuracy of WASI improves steadily as E increases, demonstrating the effectiveness of control- ling compression error through the explained variance threshold. Overall, WASI achieves up one
18

---

Published as a conference paper at ICLR 2026
Table Performance of WASI with different values all linear layers (including attention blocks and MLP blocks) of VIT using the CIFAR-10 dataset. Note that 1.0 corresponds to vanilla training.
Train Mem (MB)
Infer Mem (MB)
Train FLOPs
Infer FLOPs
Acc. (%)
0.4
39.39
32.43
3.92x 10
1.08x 10
68.99
0.5
54.02
47.06
4.99x10*
x.57
78.53
0.6
72.28
65.32
6.33x1011
2.19x
85.50
0.7
95.70
88.74
8.06x1011
x.97
90.52
0.8
127.85
120.89
1.04x1012
4.05x10'1
94.07
0.9
179.61
172.65
1,43x
5.79x1011
96.24
1.0
2349.00
324.00
3.26x1012
1.09x10
97.32
order of magnitude reduction training memory when fine-tuning VIT, with similarly favorable results observed in terms of computational cost and inference memory.
WSI on Convolutional Neural Network. In this experiment, scope of WSI to convolutional neural networks. Specifically.
extend the application
Vanilla training
0.9
0.8
0.75
MCUNet (Lin al.] (0222)
pretrained on ImageNet-IK, with Pets dataset the downstream task. WSI is applied to fine- tune the last 1 to convolutional layers of the model. When
00
Accurasyy (%)
46-
0.9, applying WSI unex- pectedly increases the weight memory. This due to the fact that the optimal rank found is too high, resulting in principal components whose total number elements exceeds that of the original weight tensor. In contrast, for smaller values of E (0.75 and 0.8), applying WSI to more layers leads to reduced memory usage for the weights, as expected, at the cost of some accuracy degradation. However, this trade-off is not worthwhile, as the memory sav- ings are marginal and the convolutional layers are already highly compact by design.
44-
42
40-
Weight Mem (MB)
Figure 12: Performance of WSI when applied to fine-tune MCUNet (pretrained on ImageNet- K) on Pets dataset. The number next to each marker indicates how many convolutional layers WSI was applied to.
Extending WASI to multi-head attention projections. For fair comparison with state-of-the-arr baselines, we relied on their efficient implementations and applied WASI only to the linear layers in MLP blocks for the main experiments. To demonstrate the broader applicability WASI, we then extended the experiments to attention blocks as well. The setup was identical to that described in Fig. and the results are summarized Tab. []]
Numerical On-device Latency. Tab. Tab. preports the latency results of WASI, ASI, and vanilla training when fine-tuning ViT on a Raspberry Pi 5. We used the same setup as in Sec. 4.4 As the compres- sion rank increases (i.e., larger e), ASI becomes progressively slower, eventually even slower than vanilla training. In contrast, WASI consistently maintains its speed. These findings are further
confirmed by Fig. 5 and Fig.
10
Latency on Multiple Edge Devices. We conducted additional experiments using the same setup
described in Sec. 4.4 The results are shown in Tab. B
On-device Energy Consumption Results. We reran the experiment Sec. 4.4 by fine-tuning a VIT on a single minibatch of 128 CIFAR-10 samples initialized with ImageNet- pretrained weights. Energy consumption was measured on Jetson Orin using its on- on-board INA3221 power sensor. Tab. reports the energy consumption for one inference pass and training iteration.
19

---

Published as a conference paper at ICLR 2026
Table 2:
Comparison of inference and training time (s) when applying WASI, ASI, and vanilla
training to fine-tune ViT on Raspberry Pi 5 at different & values.
WASI
ASI
Vanilla
8
Infer.
Train.
Infer.
Train.
Infer.
Train.
0.4
3.15
8.49
7.69
18.35
0.5
3.35
9.54
7.76
18.38
0.6
3.56
10.21
7.94
19.22
0.7
3.88
11.61
7.85
21.30
0.8
4.49
13.75
7.71
22.57
0.9
5.58
16.57
7.91
25.52
1.0
7.87
23.87
Table On-device latency of fine-tuning ViT on one minibatch of CIFAR- samples initialized with SrageNet-prettainnee weights. We report the time for one inference pass and one training iteration on three edge devices.
Jetson Orin
Jetson Nano
Raspberry
Infer (s)
Train (s)
Infer (s)
Train (s)
Infer (s)
Train (s)
0.4
1.60
5.73
5.94
71.30
5.01
16.32
0.5
1.97
6.79
9.30
91.73
5.55
18.03
0.6
2.26
7.61
12.84
85.43
6.57
20.65
0.7
2.80
8.88
19.22
93.94
7.95
24.59
0.8
3.56
10.68
20.36
117.91
9.78
29.26
0.9
4.57
13.58
22.67
118.60
13.14
38.15
1.0
6.84
21.79
29.47
241.90
20.82
65.42
C
LIMITATIONS
Our goal is enable the training of transformer models edge devices. So far, experiments have primarily focused on vision tasks, which allows for direct comparison with existing work in this domain. While our experiments with TinyLlama demonstrate the potential of WASI for LLMs (Fig. 7. current hardware limitations prevent us from evaluating larger-scale models. In future work, plan extend our approach to broader range of tasks, with particular emphasis LLMs.
D LLM USAGE
We used LLM for grammar editing. All research ideas and the article structure were conceived and developed by the authors.
20

---

Published as a conference paper at ICLR 2026
Table 4: Energy consumption of WASI on Jetson Orin with different c. Note that = 1.0 corre- sponds vanilla training.
Inference Energy (J)
Training Energy (J)
0.4
27.86
92.42
0.5
29.52
96.36
0.6
30.92
97.78
0.7
33.67
104.21
0.8
38.00
1 110.13
0.9
43.43
120.98
1.0
47.51
141.87
21


Reviews:
[{"review_id": "wtFQ5GJiq1", "reviewer": "Reviewer_ZTZ2", "summary": "WASI is a novel method that applies subspace-based training to Transformer models, primarily aiming for highly efficient on-device fine-tuning by restricting parameter updates to a low-rank subspace that captures the model's essential information. The method overcomes the memory bottleneck of backpropagation and decreases inference latency in transformer models important for edge devices. The results show that WASI maintains accuracy comparable to vanilla training while reducing memory usage by up to 62× and computational cost (FLOPs) by up to 2×. On a Raspberry Pi 5, WASI achieves roughly 1.5× faster training and inference compared to vanilla training.", "questions": "What was the variance between the results with diffence seeds? What is average of results if you try 5 different seeds.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "WASI solves an important problem, which is the memory bottleneck of backpropagation. Previous parameter-efficient methods often ignored the activation memory footprint, which scales linearly with batch size and context length. This comes from the fact that WASI iteratively maintains a subspace that efficiently handles both the weights and the intermediate activations.\n\nThe method has been shown to reduce memory usage by up to 62 times compared to vanilla training. This is crucial for enabling on-device fine-tuning of large models with severely limited RAM.\n\nBy operating within a low-rank subspace, the overall number of floating-point operations (FLOPs) required for both forward and backward passes is significantly reduced. Research reports up to a 2x reduction in computational cost, leading to demonstrably faster training and inference speedups on constrained hardware (e.g., 1.5x faster on devices like the Raspberry Pi).\n\nUnlike methods that aggressively prune models, WASI aims to preserve the model's learning capacity by training within a space believed to contain the essential information. The key strength is maintaining task accuracy comparable to vanilla training, which means the huge efficiency gains do not come at a major performance cost.\n\nI find the writing is clear and the paper is well structured. \n\nThe experiment on Raspberry PI is good. \n\nThe choice of baseline models, such as VIT, SwinT, on a couple of datasets is satisfactory.", "weaknesses": "The method is novel yet incremental. The fundamental idea of restricting model updates to a low-dimensional subspace is established. We see this in earlier methods like Stochastic Weight Averaging (SWA) and subsequent research focused on finding paths or subspaces of high-accuracy models.\n\nThe major gains in efficiency are obtained when compared to vanilla and ASI methods. The gains compared to SVD-LLM are minimal, i.e, only for memory consumption during training. \n\nNo power consumption results on the device.  \n\nThe results are shown to work for seed 233. It is mentioned that the variance between the results for different seeds is not high, but there are no results showing that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Frfn0myxfd", "reviewer": "Reviewer_qswV", "summary": "The paper proposes WASI, a weight–activation subspace iteration framework that trains and runs transformers entirely in a learned low-rank subspace. WASI couples (i) Weight Subspace Iteration (WSI)—initial SVD with explained-variance threshold ε and iterative subspace updates—with (ii) a redesigned Activation Subspace Iteration (ASI) (dynamic-programming rank selection, 3D/4D activations).", "questions": "- Can you provide a micro-benchmark of one WASI iteration (basis update, orthogonalization, matmuls) on Pi-5 vs. GPU to attribute the speedups precisely?\n\n- Can you adopt more concrete edge devices for expriments?\n\n- How sensitive are results to ε and the number of subspace-iteration steps? Any auto-tuning strategy that targets a latency or memory cap directly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Unlike LoRA-style PEFT that re-inflates at inference, WASI keeps both weights and activations in low rank throughout forward/backward, with explicit FLOPs/memory formulas and subspace-space equations, yielding predictable savings end-to-end. \n+ The paper motivates stable layer ranks during fine-tuning  and shows singular-value/rank stability, enabling subspace iteration instead of per-step SVD; empirically WSI dominates repeated SVD in FLOPs/accuracy.\n+ The on-device results seems interesting.", "weaknesses": "- The edge device used in this paper is only Rasperry pi 5. More edge devices like Jeston Nano should be included and evaluated.\n\n- The final accuracy should be highlighted and reported with Tables. Now it is hard to find it in the draft.\n\n- Main comparisons focus on MLP/linear blocks for fairness with baselines; attention-layer coverage is deferred to appendix. Non-IID, partial-participation, and straggler/client-heterogeneity studies (key for on-device contexts) are not central, leaving external validity to real deployments somewhat open.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R9YgopYx3f", "reviewer": "Reviewer_cFHJ", "summary": "This paper proposes Weight–Activation Subspace Iteration (WASI), a method that performs model training entirely within a low-rank subspace of both weights and activation. Experimental results demonstrate that WASI significantly reduces memory usage and roughly halves the computational cost (FLOPs), making on-device training feasible even on constrained hardware such as the Raspberry Pi.", "questions": "- Could the authors evaluate the proposed method on a larger-scale dataset, such as training from scratch on ImageNet-1K? Even if direct on-device testing is infeasible in this scenario, a simulated experiment on full GPUs would provide stronger evidence of the method’s scalability and the effectiveness of the low-rank approximation at large scale.\n- Is there a principled way to choose the ε  other than grid sweeping? For example, can it be adaptively determined or estimated from training dynamics?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n\n- The proposed method can be applied to various architectures, including ViT, Swin Transformer, and TinyLlama.\n\n- Experiments convincingly show that WASI drastically reduces memory consumption and FLOPs while maintaining comparable accuracy to full fine-tuning.", "weaknesses": "- The core ideas, low-rank approximation of parameters and activations via subspace iteration, have been explored in prior work. Thus, the novelty is somewhat limited, and the main contribution is largely engineering integration rather than conceptual innovation.\n\n- The ablation study indicates that the method can be sensitive to certain hyperparameters and datasets, which may affect stability.\n\n- Experiments are conducted on relatively small datasets, which could lead to higher variance and limit generalization.\n\n- Minor: The figures could be improved, for example, by labeling the ε thresholds more clearly to make the trends easier to interpret.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)