INSTANT: Compressing Gradients and Activations for Resource-Efficient Training
Authors: Tuan-Kiet Doan
Trung-Hieu Tran
Enzo Tartaglione
Nikola Simidjievski
Van-Tam Nguyen
Venue: ICLR 2026 Poster (2026)
Keywords: Gradient Compression
Activation Compression
Resource-Constraint Training
Primary area: optimization
TL;DR: This paper introduces INSTANT, a method for efficient training using low-rank gradient and activation compression.

Abstract: Deep learning has advanced at an unprecedented pace. This progress has led to a significant increase in its complexity. However, despite extensive research on accelerating inference, training deep models directly within a resource-constrained budget remains a considerable challenge due to its high computational and memory requirements. In this paper, we introduce INSTANT (compressIng gradieNtS and acTivAtions for resource-efficieNt Training), a method designed to address both the computational and the memory bottlenecks when training. INSTANT reduces resource demands during backpropagation by projecting gradients and activations into a low-rank subspace and performing computation within that compressed representation. Experimental results demonstrate that INSTANT achieves a $15\times$ reduction in computational cost and $32\times$ reduction in activation memory with negligible impact on model performance. The code will be made publicly available upon the paper's acceptance.

Published as a conference paper at ICLR 2026
INSTANT: COMPRESSING GRADIENTS AND ACTIVATIONS FOR EFFICIET TRAINING
Tuan-Kiet Doan; Trung-Hiee Tran; Enzo Tartaglione, Nikola Simidjievski, Van-Tam Nguyen LTCI, Télécom Paris, Institut Polytechnique de Paris, France
tuan. muan.dcame tuan.dca@@teleene watpetteeeecooorrise.eea. paris. fr, trung-hieu. tewa@teeeecommppeett.er.eeee fr,
enzo. anze.tartagl i www.@teeeecoo fr, nikola. ciiimdddeessi paclleco--arri.... van-tam.nguyen car-tam..ggeantttcccoooooop.....
ABSTRACT
Deep learning has advanced at an unprecedented pace. This progress has led to a significant increase in its complexity. However, despite extensive research on accelerating inference, training deep models directly within a resource-constrained budget remains a considerable challenge due to its high computational and memory requirements. In this paper, we introduce INSTANT (compressIng gradieNtS and ac TcTivAtions for resource-efficieNt Training), a method designed to address both the computational and the memory bottlenecks when training. INSTANT reduces resource demands during backpropagation by projecting gradients and activations into a low-rank subspace and performing computation within that compressed representation. Experimental results demonstrate that INSTANT achieves a 15x reduction in computational cost and x2x reduction in activation memory with negligible impact on model performance. The code is available at INSTANT
INTRODUCTION
Deep learning has become the backbone of many practical applications in diverse fields such as computer vision (CV) (Dosovitskiy et al. 2020 Liu et al. [222 natural language processing
(NLP) (Devlin et al. 2019; Liu et al. 2019; Radford et al., 2018), signal processing (Gong et al. 2021), and multimodal learning (Radford et al. [ et all (022). Although these applications offer undeniable benefits, the large-scale design of deep learning models prevents their deployment on devices with limited resources. To address these shortcomings, research has mainly focused on two main directions. Many works focus on developing architecture modifications to adapt to hardware constraints (Sun al. 2020 et al. 2022). Many others enhance quantization techniques to reduce memory footprint and improve inference speed for large models (Liu et al., 202Tb), leveraging hardware that supports fast operations on low-bit datatypes. To summarize, the majority of the work focuses on model inference, while the training part is typically performed on an independent geghrpeerformncc infrastructure.
Resource-efficient training faces two main challenges: high memory usage and computational cost. The memory overhead has been partially addressed in prior work (Nguyen et al., 2024 Yuan et al., 2023 Wang et al., 2025), typically employing singular value decomposition (SVD), to construct low-rank space for activations and/or weights. However, these SVD-based methods incur substantial computational overhead, ultimately increasing training time (Nguyen al. 2024). On the other hand, reducing computational cost during training remains an open area of research. (Sakr & Khailany 2024) reduces the cost of tensor decomposition by constructing a periodically updated low-rank space for activation throughout the training process. (Yang et al. 2023b) leverages low-frequency characteristics of images to project tensors into predetermined low-rank spaces, which helps to save both activation memory and computational overhead for CV tasks. However, this method has a limited compression rate and is only effectively applicable to low-frequency data components such as images.
Inspired by tensor decomposition strategies, we propose INSTANT (compressIng gradieNtS and acTivAtions for essource-effiieeN Training). INSTANT reduces the resource demands of backpropa- gation, both in terms of computation and memory, and applies to a wide range of data distributions.
Equul contribution.

---

Published as a conference paper at ICLR 2026
A
Act1
Ac+2
Full-rank Forward
Full-rank Forward
Forward
A
Å41
Memory
INSTANT
A
A
Compression
aL
öL
0L
öl
0.4
0A
aA
0Ap+1
0Ai+2
Low-rank
Low-rank
Backward
Backward
Backward
Figure INSTANT performs compression on activations and gradients: In the forward pass (Green), activation A; normally propagates in space while its compact version A, is saved for the backpropagation to reduce memory consumption. In the backward pass (Red), the gradient for the
AL is compressed low-rank version OL Then, low-rank operations are implemented output 8A:41
8A at to save backward computation.
with compressed activation A, and compressed gradient 8A.41
Our method periodically identifies critical tensor features to generate dynamic low-rank projections, optimizing backpropagation efficiency. Our method is orthogonal to non-compressive acceleration techniques (Yu et al. 2022 Kwon et al. 2023). low-rank adaptation (Hu et al. 2022). low-rank param- eter gradient (Zhao al. 2024), and existing tensor compression techniques such quantization (Xi
et al. 2024). Our key contributions are as follows:
We introduce low-cost calibration technique generate calibrated orthonormal bases for tensor projection, enabling significant reductions in memory and computations (Sec. 3.2
We project activation tensors and gradients onto these orthonormal bases. To knowledge, this is the first work to exploit the low-rank structure of activation gradients for all types of data distribution. We provide an error analysis of our gradient compression, illustrating that high compression ratio is achievable with limited performance degradation (Sec. 3.3
We evaluate INSTANT across multiple datasets and model architectures, consistently demonstrating good performance, achieving up to 32x memory savings and 15x computational cost reduction with only 1% trade-off in accuracy compared vanilla fine-tuning (Sec.
2 RELATED WORK
Activation compression. Activation compression is recently emerging research direction that addresses the memory challenges during training. This approach offers several key advantages based on the following observations: (i) model weights remain uncompressed during training. thereby preserving their expressive capacity; (ii) activations are often large and exhibit significant redundancy, making them suitable for compression (Sakr & Khailany| 2024 Miles al., (2024). (Nguyen et al.) 2024) applies SVD to compress activations to reduce huge memory usage for activations. However, this approach raises substantial computational overhead due the high cost of performing SVD in each training iteration. (Sakr & Khailany 2024) (ESPACE) tackles SVD computational expense by using calibrated subspaces, which are periodically updated, to compress activations. They enable activation compression in the forward pass, reducing computational overhead in both the forward and backward phases. However, ESPACE prone error accumulation, as relies on the universal fixed subspace across varying activations.
Optimizer state compression. Weight gradients are inherently low-rank (Yang et al. 2023a).
Previous studies (Bernstein et al. 2018 Vogels et al. 2019) have leveraged this characteristic to address communication bottlenecks in distributed learning by reducing inter- data transmission. GaLore (Zhao et al. 2024) and its variances (Muhamed et al. 2024 Shamshoum al. 2025) leverage the low-rank property of weight gradients for compressing them to reduce memory usage in the optimizer state significantly. (SoppcctShamshoun et al. 2025) further reduces the memory overhead
2

---

Published as a conference paper at ICLR 2026
by compressing both optimizer state and activation memory. Nonetheless, in all aforementioned techniques, the activation gradient computations still rely entirely on high-cost backpropagation.
Activation gradient compression. Gradient filtering (Yang al. 2023c) proposed to pool both activation and gradient activation into a low-rank space for efficient computing. Since this technique helps increase efficiency, the performance drop significantly large. (Yang et al. 2023b) proposes low-rank backpropagation via Walsh Hadamard transformation (LBP-WHT) to compress the gradient of output for multiplications in the low-rank space, which reduces the computational complexity. However, LBP-WHT' is restricted to low-frequency inputs such as images, and achieves only modest compression ratios, resulting in limited gains in both memory efficiency and computational reduction. INSTANT, on the other hand, overcomes the low- wwwffreuuency assumption via SVD, compressing the gradient into smaller space compared to LBP-WHT, and as result, can be applied variety of input types.
3 INSTANT
We start with discussing the computational and memory issues of vanilla backpropagation (Sec. 3.1 Next, we present our efficient construction of low-rank projectors for both activation and activation gradient in Sec. 3.2 Finally, we demonstrate our approach to perform low-rank multiplications the backward by using these projectors, instead of full-rank computations (Sec. B.3). Our objective here to reduce both memory and computational consumption for the backpropagation.
3.1 PROBLEM STATEMENT
In the following, for simplicity, foeus linear layers. Extension to convolutional layers is presented in Appendix H Within each layer, the batch dimension and bias are omitted without loss of generality, Let RZXC= denote the input activation, y E R LxCC, denote the output activation, and w E IR C xC= denote the weight matrix, where L denote the sequence length and C2 and Cy are layer channel dimensions, which predefined by model architecture.
In the forward phase, the input x is propagated through the layer to compute the output: = y= w In supervised learning, the output of the final layer is compared against the ground truth label to calculate a loss value C. The backward pass, also known as backpropagation, computes the gradients Oc
of this loss to input x and weight w. We denote gy = By E R LxC, as gradient of the loss with respect to the output y. Gradient of the loss with respect to the input x, and the weight w are denoted BC respectively, calculated as:
#C and gw Ow as gx Ux
gx = By W, gw = gy x.
Here, gw E RC is used update the weight w, and gx RLCC. is propagated the preceding layer.
Training overhead. The backward pass (equation 1 includes two matrix multiplications, each with the same cost, resulting in total of 4 L C2 C., floating point operations (FLOPs). In Transformer- based models, the values L, C., and Cy are typically large, leading to a computational burden, especially during the backward stage. Moreover, since the backward pass requires storing the input activation x for computing gw, memory consumption can significantly increase.
To address the dual problems of high memory and computation, we propose tensor decomposition strategy for both input activation x and the output gradient gy. As shown in Fig. I we project the activation x and gradient gy into lower-dimensional space for storage and computation. Unlike previous works (Yang et al. 2023b) that use universal projections for both of these tensors, we adopt adaptive projections that better capture the crucial information of each tensor. Next, discuss our construction scheme of these low-rank projections for each layer.
3.2 EFFICIENT CONSTRUCTION FOR TENSOR PROJECTION
Low-rank and low-cost projections are required effectively compress the input activation and the output gradient gy. Previous approaches (Nguyen et al. (204) use SVD with an operational cost of (n²²) to compute these subspaces at every training step, making them impractical for large-scale training.
3

---

Published as a conference paper at ICLR 2026
gy (L C2)
CG (L L)
U LLxL)
Q ((k + p) x L)
Process
SVD
Truncation
(a) Constructing compression tensors Q for output gradient gy. Similarly, P is constructed for activation x.
X
P
Update w
P
x
P
pfl
g'
Full-rank space
Low-rank space
Low-rank
multiplication
Q
By
g'
W
Current Laver
W
Pre. Layer
(b) Low-rank Backpropagation algorithm with INSTANT. Instead of full-rank multiplication as Vanilla, INSTANT decomposes x, gy to approximate gw and gx by low-rank multiplications.
Figure 2: INSTANT backpropagation involves calibrated low-rank tensors P, Q, which are updated in calibration process. These low-rank projections reduce memory consumption of saving activation and reduce computations thanks to low-rank multiplications.
To alleviate this overhead, we adopt a calibrating strategy in which we create periodically updated compression subspaces for compressing activation and output gradient gy in substantial N, training steps. For good approximations, each tensor demands its specific compression (Appendix. B.I.. Our approach is inspired by ESPACE's theoretical foundation (Sakr & Khailany (024) for fixed subspace compress the activation. Given activation tensor E X. by performing SVD acti- vation auto-correlatin correlaarrtt::: Cx = E x x UEUT where U are singular vectors and are their associated singular values, the projection matrix built from U minimizes decomposing activation x (Appendix, B.2). Different from ESPACE, we decompose both activation x and output gradient gy, and do not stretch their batch dimension for these decompositions.
square (MSE) of
Data preprocessing. To approximate auto-correlation tensors, many data batches are required to get sufficient statistics of activation and output gradient gy- In this calibration stage, if we naively save all of this data for post-pree post-processing. it will lead to memory accumulation, which is at odds with the goal of reducing memory. Therefore, we implement low-cost data preprocessing to limit its storage demand below that of the training phase, ensuring nov-computational calibration and peak memory usage does not increase (Appendix.) [B.3). For simplicity, this section, we present the approach to find the compression subspace Q for gradient gy (Fig. 2ah. A similar strategy is applied for finding compression subspace of activation x.
Singular value decomposition. After getting output gradient auto-correlation CG = E(gy gy via data preprocessing, we decompose this to get the left singular vectors U and their associated singular values Σ in decreasing order. Projection matrix built from U is proved to minimize MSE of decomposing gy.
Truncation with energy threshold. To reduce dimensionality while preserving information, define energy threshold e (e < 1) as the portion of tensor energy remaining after decomposition. We define & as the sum of squares of all eigenvalues σ, in E:
Σ of = CC
(2)
where ||G is the squared Frobenius norm of Cg. Following that, we truncate k vectors of U to form Uk. where truncation index l is the minimal integer satisfying: Σ of E. When this truncation strategy preserves most of tensor energy, hence assigning Q U reduces decompo-
sition MSE between gy and its reconstruction gy Q Q gy. Compression tensor Q is extremely
4

---

Published as a conference paper at ICLR 2026
smaller than Ey due to its low-rank characteristic (as shown in Sec. 4.2 therefore, Q reduce memory and computational expense when joining backward operations (Fig. 2b
Truncation with Oversampling. The energy threshold ensures that certain amount of information is preserved when the calibration happens. However, as training progresses, this projection may no longer suffice maintain that amount of information, since the core bases which to project can vary. To address this, we propose oversampling, which increases the number of base projections by a fixed amount, thus reducing information loss when the core bases change (Fig. 2ap. This truncation technique is proven to be effective in Sec. 4.4 and further investigated in Appendix.
Dropping lower singular values leads to an accumulation of energy loss when backpropagating reconstructed tensors. We, therefore, propose an energy offset mechanism to compensate for this loss each truncation. Finally, Q is given as:
k+p Σ of
Q = Uk
where Q E RRyxL with Ry k p. Applying the similar strategy, we find compression subspace P E RR for low-rank activation x.
3.3 LOW-RANK BACKPROPAGATION WITH INSTANT
The activation and gradient can be mapped onto low-rank spaces with two projection tensors con- structed in Sec. 3.2 This part will demonstrate the use of these low-rank matrices P and Q in our training process. Our overall training pipeline is depicted in Fig. while the detailed technique for handling the low-rank backward is illustrated in Fig. 2b
Since changing the forward pass of the network can cause a significant performance drop, we keep the forward pass y w unchanged. Meanwhile, the activation map x is projected into smaller space: & = P x where P E IR R2 & E IR R. with Rg < L. This compressed activation is retained in memory in place of x for the backward pass, which saves large amount of memory.
The backpropagation process of INSTANT is described in Fig. 2b By the property of the low-rank projection tensor P built in Sec. 3.2 x P P x, weight gradient is approximated:
T gw - By
x By T (P..P..
P x. - (g) P (P
This reordering results gw & Bw By: 8 with 2 low-dimensional spaces θy, E R,x, and & E R2xxx with R2 < minn(A I., Cv, Cy), illustrated in Fig. 2b
Similarly, by the property of low-rank projection tensor Q built in Sec.
3.2 gy ~ Q Q gy, input
gradient is approximated with low-rank multiplications:
yyz = Q By,
= W,
=x=Q Q 8x.
(5)
where Byz E B,, 8x E XC, (2 low-rank spaces). Finally, the approximated input gradient 8x E LxC, is propagated to the preceding layer.
Equation and equation 5 will cause error compared traditional backpropagation. This error is mathematically proved to be negligible with our low-rank projection scheme (Appendix.
Training overhead. In total, the computational cost of INSTANT for backpropagating one layer, which includes low-rank compression, low-rank computation, and reverse projection, is 2- (R.- + Ryy) (Cz C. + I. C. I.. C.CC) FLOPs as shown in the Appendix. D.1 Given that R₂ + R2 < min CI, Cy), this number is much smaller than 4. L C. C, of vanilla train- ing. For example, one BERT block has L 512, C₂ = C, = 768, choosing Rt = Ry = 8 can save about 27x FLOPs. Remarkably, P. Q. and & P are only used during training, meaning that at inference time, there is no trade-off in memory or computation compared to traditional inference. Moreover, INSTANT focuses solely reducing computational and memory costs during backpropa- gation, without modifying the optimizer state. INSTANT is orthogonal and can combined with any technique that compresses the optimizer state.
5

---

Published as a conference paper at ICLR 2026
4 RESULTS
4.1 EXPERIMENTAL SETUP
Computer vision tasks. We conduct experiments for image classification with similar setup to
LBP-WHT (Yang et al. 2023b). We assess our method on ImageNet (Russakovsky et al. 2015)-
pretrained Vision Transformer models (EfficientFomerrLLI eL. 2022), EfficientFormerV V2-S0 (Li et al. 2023), and Swin V2-Smal (Liu et al. 2022) by fine-tuning them on five different datasets for 50 epochs. Details of the datasets are discussed in the Appendix. We use batch size of 64 and use AdamW optimizer with the same learning rate schedules as in (Yang et al. 2023b). To
balance training efficiency with performance. the calibration process carried out every N; 200 iterations, and an energy threshold = 95% is kept constant throughout all experiments, with only the oversampling hyperparameter p being varied. In Tab. and Tab. 2 we denote INSTANT-p as our method with energy threshold of 95% and oversampling p vectors. For fine-tuning the last layer, p is tested with values of 0,5, and 7, while for fine-tuning the entire model, is tested with values of 5, 10, and 15.
Natural language processing tasks. We employ two Transformer- based models (BERT (Devlin eet 2019) and DistilBERT (Sanh 2019)), which are pretrained on Wikipedia and BookCorpus. We utilise the GLUE benchmark (Wang et al. 2018) to evaluate model performance, including datasets: CoLA, QNLI, MRPC, RTE, SST-2, and MÑLL. For each dataset, each model is fine-tuned for 10 epochs with the Adam optimizer and a batch size of 32. To balance between training efficiency and performance, the calibration process is carried out after each N, 50 iterations, and we maintain an energy threshold 95% across all experiments, adjusting only the oversampling hyperparameter p. For fine-tuning the last layer, we test p with values of 0, 7, and 15, while for fine-tuning the entire model, is tested with values of 7, 15, and 25.
Baseline. Besides vanilla fine-tuning (Vanilla in Tab. 1 2). we also reproduce and evaluate 2 other methods: Gradient Filter (Yang et al.] (023c) and LBP-WHT (Yang et al.] (0236). For image classification tasks, we strictly follow the authors' setup and configuration, whereas for language tasks, we reshape the sequence length of each input into an appropriate two-dimensional tensor like the authors, with o denoting the
to apply their method. We use the notation order that determines the rank of the compressing subspace. We compare each method based on average accuracy (mAcc), average Mega FLOPs (MFLOPs), and average activation memory consumption (Mem). Notably, FLOPs and memory are only reported for the Linear layers, which are the heaviest computational components in these architectures. We measure computational cost in FLOPs rather than time, as it's unaffected by implementation details. Therefore, this metric allows us to evaluate the efficiency gains from a better algorithm rather than implementation aspects. All experiments are performed on NVIDIA TESLA V100, and the source code uses PyTorch 1.13.1. We use the MMCV library (Contributors) 2018 for CV tasks and use the Hugging Face library (Wolf al. 2020) for NLP tasks.
4.2 THE ACTIVATION GRADIENT IS LOW-RANK
We conduct experiments with BERT on the MRPC dataset. In the fine-tuning process, we randomly select samples and track their gradients corresponding to the output at each layer. SVD is then applied to each gradient to extract its eigenvalues. As shown in Fig. 3 the number of ranks required to keep c = 95% of energy only 6. Since most of the energy concentrated in few top eigenvalues, it suggests that a small number of ranks can retain a significant portion of the gradient's iett information. We have validated this phenomenon in a wider range of samples and layers. Further details for other layers in other architectures are provided in the Appendix K.5] This observation supports our idea of projecting the activation gradient into a smaller subspace, where we perform computations to reduce computational cost while preserving large amount of the gradient's 's information.
4.3 MAIN RESULTS
Computer vision tasks. Tab. I presents the results of INSTANT in comparison with vanilla fine- tuning and other gradient and activation compression methods Vision Transformer models. It is noticeable that INSTANT achieves a significant reduction in both MFLOPs and memory usage compared to Vanilla. In EfficientFommer, INSTANT-0 witnesses drops from 2% to 5% compared to

---

Published as conference paper at ICLR 2026
BERT encoderlayer. www.:err.....ooppppo. ooupuutteeeept
ResNet-50 Performance on CIFAR10
-4000
1.00
93.18
93.08
Normalized Energy
0.95
3236
92.16
-3500
-3000
0.90
Energy
0.95
Memory (MB)
3.4
0.85
k=6
2500
MELOPS
-2000
0.80
0.75
11,59
1306
-1500
-1000
40
50
741
Number Eigenvalues
500
10.03
0.04
0.05
Figure 3: The percentage of energy retained depends the number of eigenvalues in one layer in BERT.
Vanilla
Ours-0
Ours-5
Ours-7
Figure 4: Fine-tuning ResNet-50 on CIFARIO
Table Experimental results across 5 different CV datasets, presented for both fine-tuning the last block and the entire model. We report the MFLOPs and memory (Mem) required for training single sample. Detailed measurement methodologies can found in the Appendix, D.1
D.2
D33
Fine-tuning the Last Block
Datasets Flowers 1
Model
Method
MFLOPs
Mem (MB)
CF100
CF10
Food
Pets f
mAcc
Vanilla
1484
1.95
79.28
95.23
95.50
84.04
93.13
89.44
OradienFFlliiinn
0.64
68.25
90.72
"7.66"
38.02"
32.51
Efficient
LBP WHT-2
95
0.12
75.61
93.35
95.07
79.65
92.34
87.20
Former-L1
LBP-WHT-4
335
0.40
78.27
94.6
95.53
82.37
93.16
88.79
LBP-WHT-8
1227
79.34
89.43
INSTANT-0
270
0.16
77.64
94.66
92.23
81.97
92.64
87.83
INSTANT-5
78.65
95.07
95.93
82.84
93.21
89.14
INSTANT-7
544
0.45
79.01
95.92
83.05
89.25
Vanilla
349
1.47
72.37
92.63
92.73
81,44
90,52
85.94
Grudient Fillering
0.03
64.77
37.03
37.9
"73.67
33..55
79.66
Efficient
LBP-WHT-2
0.09
65.75
88.68
89.51
74.72
87.49
81.23
FormerV2-S0
3PT-
99
0.30
69.03
90.88
90.73
79.45
89.29
83.88
IBP-WHT-8
71.90
INSTANT-0
119
0.03
64.51
89,62
87.94
76.77
88.12
81.39
INSTANT-5
0.15
68.61
90.85
90.86
79.07
88.83
83.64
INSTANT-7
69.42
90.52
88.88
83.88
Vanilla
2718
2.25
80.84
96.07
97.61
88.31
95.53
91.67
Grudhent Filiering
37
0.04
80.19
95.68
97.4
'87.3
99.9
91.10
WHT-2
SwinV2
Small
LBP-WHT-4
499
0.41
80.39
95.71
97.54
88.32
94.66
91.32
LBP W
1830
1.48
80,80
95.80
97.56
88.19
95.07
91.48
INSTANT-O
0.05
80.55
95.69
97.43
88.30
95.12
91.42
INSTANT-S
435
0.26
80.85
95.91
97.38
88.42
95.20
91.55
Full fine-tuning
Vanilla
4528
18.46
84.84
96.99
94.84
85
93.16
91.09
Grudient Filiering
0.34
41.05
75.22
37.63
Efficient
LBP-WHT-4
1211
3.36
77.97
94.13
93.38
41.88
92.23
79.92
Former-L]
LBP-WHT-6
2560
7.05
83.00
96.45
94.58
83.77
92.86
90.13
LBP-WHT-8
12.08
83.88
96.78
85.08
90.75
INSTANT-S
2107
1.98
82.41
96.29
94.50
83.87
92.78
89.97
INSTANT-10
83.05
96.48
94.73
84.66
93.13
90.41
INSTANT-15
2884
3.45
83.56
96.85
94.47
84.85
93.1
90.57
Vanilla, especially in high variance datasets such as CF100, Flowers, and Foods. This possibly be- cause the strict energy threshold approach in INSTANT-0 is not efficient with varying distributions of these datasets. INSTANT-5 and INSTANT successfully address this issue, supporting our hypothe- sis that considering only tensor energy is insufficient. Compared and V2, INSTANT-5 gains comparable performance to LBP- WHT-8 with only 23% computation and 18% memory consumption. In the full-finetuning, INSTANT-10 10 outperforms JBP-WHT-6 and
LBP-WHT, in EfficientFonner-J

---

Published as a conference paper at ICLR 2026
Table 2: Experimental results on GLUE benchmark, presented for both fine-tuning the last layer and the entire model. We report the MFLOPs and memory (Mem) required for training a single sample. Detailed measurement methodologies can be found in the Appendices D.I
D.2
D.3
Fine-tuning the Last Block
Model
Method
MFLOPs
Mem (MB)
MRPC
CoLA
QNLIT
Datasets
RTET
SST-2T
MNLI
mAce
Vanilla
14495
13.50
83.92
43.55
86. 12
59.21
91.63
78.31
73.79
Gradient Filter
226
0.21
82.16
40,97
75.87
38.48
88.30
63.12
68.15
LBP-WHT-2
732
0.63
82.21
41.09
79.32
59.21
89.11
68.17
69.85
BERT
LBP-WHT-4
2464
2.11
82.33
42.46
83.82
58.84
90.71
73.62
71.96
INSTANT-0
175
0.03
82.21
41.35
79.33
61.73
90.71
64.39
69.95
INSTANT-7
0.21
82.33
41.58
84.13
62.09
90.94
72.92
72.33
INSTANT-15
1018
83.31
43.02
84.68
9163
74.68
73.06
Vanilla
14495
13,50
82.55
35.52
82.85
59.21
88.99
73.60
70.45
Gradient Filier
226
0.21
82.46
32,80
73,99
58.48
85.89
54.31
64.66
Distil- BERT
LBP-WHT-2
732
0.63
82.58
33.54
76.86
58.12
87.61
60.43
66.52
LBP-WHT-4
2464
2.11
82.46
31.73
80.62
56.68
89.11
67.56
68.03
INSTANT
66.07
INSTANT-7
546
0.21
82.41
33.50
80.49
59.57
89.56
68.52
69.01
INSTANT-15
999
0.42
82.41
32.96
80.93
58.12
88.99
69.68
68.85
Full fine-tuning
Vanilla
173946
162
90.23
$8.69
91.43
67.51
93.23
84.46
80.76
Gradieni Filler
27.66
22.5.
34.49
4371Z
W.62""
6...%%
WT.96
63.60
71.7%
LBP-WHT-2
8784
7.59
86.64
48.60
84.14
63.90
89.11
70.91
73.88
BERT
LBP W
29579
25.31
87.80
46.12
86.16
65.70
90.94
77.32
75.67
INSTANT-7
9143
2.83
87.93
57.71
64.62
92.22
81.36
78.92
IS
15353
5.43
87.97
58.08
90.63
62.82
92.43
83.06
79.17
INSTANT-25
89.47
57.29
90.79
93.35
79.65
compared to Vanilla, it achieves 1.8x computational reduction and 6.3x memory reduction average, at the expense of 0.8% accuracy drop. It is noticeable that LBP-WHT-8 gains good perfor- mance at the expense of a small compression rate, especially in computation, Meanwhile, although the Gradient Filter can save large amount of computational resources, the performance drops remarkably. This shows that this method cannot keep enough information for low-rank projection.
Natural language processing tasks. Tab. iillustrates INSTANT's effectiveness on 6 datasets of the GLUE benchmark. In fine-tuming the last block. compared Vanilla, INSTANT-15 can save 14.37x FL.PPs and x in memory, with small reduction in accuracy of 1. 1.17%. Notably, the difference in mAce mainly comes from the MNLI dataset. This may be due to the large distribution of the dataset, which causes the method to struggle in finding an appropriate low-rank space. We believe that performance on this dataset can be further improved by increasing the oversampling amount. Compared to LBP-WHT, in BERT, INSTANT-7 achieves 2.48% accuracy boost over LBP-WHT-2, while it requires only about 75% of the FLOPs and 33% the memory. the full fine-tuning, INSTANT shows slight performance decline relative to the Vanilla baseline. In contrast, LBP-WHT shows larger performance degradation across various datasets. This indicates that the low-frequency transformation assumption may not hold well for language tasks. INSTANT, with its SVD-based projection, can mitigate this issue to some extent, resulting in better performance compared to LBP- WHT. Note that calibration computational overhead is excluded from Tab Tab This overhead is proven to be much smaller than training overhead in Appendix B.3
Extension to convolution. INSTANT is applicable to convolutional layers (as shown in Ap- pendix. H.. We conduct experiments with MobileNetV2 and ResNet-50 architectures CIFARIO and CIFARI00. As indicated Fig g INSTANT can achieve small accuracy boost with 3x computation savings and 79x memory savings, compared to Vanilla. Extra results are indicated Fig These results demonstrate the efficiency of INSTANT across all architectures, including Transformer- and Convolutional-basee models.
Edge device latency. We conduct experiments on Raspberry (CPU ARM Cortex-A76), We report the average training time over 1 epoch CIFAR1O. The experimental setup is the same the one indicated as Sec 4.1 By saving computations, INSTANT reduces 2x of backward time, which decreases the average total training time, compared to Vanilla, as shown in Fig. 5 The time reduction up to 12x in another architecture, as indicated in the Appendix. However, INSTANT may increase GPU latency, as shown in the Appendix.
8

---

Published as a conference paper at ICLR 2026
Table Comparison of INSTANT with GaLore and CompAct, presented for both fine-tuning the last layer and the entire model. We report the MFLOPs and memory (Mem) required for training single sample on QNLI and SST2.
Fine-tuning the Last Block
Datasets
Model
Method
MFLOPs
Mem (MB)
QNLI
SST-2
Vanilla
14495
13,50
86.12
91.63
GaLore-8
[4495
13.50
82.45
90.47
GaLore-32
14495
13.50
84.37
91.28
CompAct-8
7978
0.11
80.65
88.30
BERT
CompAct-32
8355
0.44
84.37
91.28
INSTANT-0
175
0.03
79.33
90.71
INSTANT-7
565
0.21
84.13
90.94
INSTANT-15
1018
0.42
84.68
91.63
Full fine-tuning
Vanilla
173946
162
91.43
93.23
GaLore-B
173946
162
90,44
9I.74
GaLore-32
173946
162
91.31
92.09
CompAct-8
95736
1.32
89.09
91.28
BERT
CompAct-32
100260
5.28
90.30
92.09
INSTANT-O
9143
2.83
89.66
92.22
INSTANT-15
15353
5.43
90.63
92.43
INSTANT-25
20753
90.79
93.35
Comparison of INSTANT with optimizer state compression techniques. We provide additional results to compare INSTANT with two optimizer compression techniques, which GaLore (Zhao et al. 2024) and CompAct Shamsshom et al. 2025), as presented in Tab. 3 GaL consistently achieves better performance than CompAct under the same low-rank constraint; however, its acti- vation memory consumption and computational cost remain similar to Vanilla training. In contrast, CompAct substantially reduces activation memory by compressing activations during the forward pass. Compared to CompAct, under the same activation memory budget, INSTANT can save large portion of the backward computational cost while achieving better performance.
4.4 ABLATION STUDY
Compressing both components has practical benefits. We conduct ablation studies compressing only gy (INSTANT (compress gy)) and compressing only x (INSTANT (compress x)); presented in Tab. 4). Compressing only gy) achieves good performance, even surpassing Vanilla training. However, due to saving full-rank activation, INSTANT (compress gy) requires the same memory as Vanilla. On the other hand, compressing only x achieves good performance with a high compression rate, which saves large amount of memory. However, this method incurs low compression rate in computation, because it involves one full-rank multiplication in the backward pass. INSTANT (compressing both x, gy) achieves the highest computation compression rate and high memory compression rate with a negligible performance drop compared to Vanilla. In short, INSTANT (only compress x) is good choice if tasks only require small memory budget. INSTANT (compressing both x, gy) is good at tasks requiring a small memory budget and small training time, making it efficient for resource-efficient training.
Energy is not enough. The energy threshold is local measure for truncating the compression tensor, which is based completely on the calibration data. This hyperparameter does not account for changes in the low-rank space during training, potentially leading performance drop. As shown Fig. 6) with similar budget of FLOPs/memory, e-only truncation achieves a lower accuracy compared to truncation with oversampling.
Efficient training-aware subspaces. INSTANT allows for selecting adaptive subspaces customized for each layer, via truncation strategies (Sec. 3.2). With different datasets, keep the same amount of information, there a large difference the subspace ranks, which makes FLOPs vary during the training process (Appendix K.6). This rank adaptiveness controls the loss of tensor information,

---

Published as a conference paper at ICLR 2026
Table Ablation study of partial compressions, across different CV datasets when fine-tuning the last block of EificientFormer-.1... We report the MFLOPs and memory (Mem) required for training single sample.
Fine-tuning the Last Block of EfficientFormer-LI
Datasets Flowers
Method
MELOPS
Mem (MB)
mAcc
CF100 1
CFIO
Food 1
Pets 1
Vanilla
1484
1.95
79.28
95.23
95.5
84.04
93.13
89.44
973
1.95
78.45
95.02
95.74
82,91
93.08
89.04
INSTANT
1069
1.95
79.00
95.26
96.16
83.49
93.02
89.39
(compress Sy)
79.18
95.39
95.95
83.73
93.08
89.47
796
0.12
78.80
95.07
94,99
82.63
92.83
88.86
INSTANT
78.99
95.35
95.37
83.01
92.75
89.09
(compress
921
0.41
78.93
95.40
95,43
83.19
92.80
89.15
77.64
92.23
81,97
92.64
87.83
INSTANT
475
0.38
78.65
95.07
95.93
82.84
93.21
89.14
544
0.45
79.01
95.23
95.92
83.05
93.02
89.25
Vanill
INSTANT
18P-WHT-4
GF
Performance FLOPs
Performance Memory
Backward Time
83
Venill
INSTANT
3.40
84
LBP-WHT-44
Accurscy
GF
0.5
Total Time
Vanill
3.45
INSTANT
LOP-W-IT-4
5779
onlyy
80
only
GF
1.3
Time (4)
LFLOPs
Momory (MB)
Figure 5: Training time of EfficientFormer-LI on CIFAR1O on Raspherry Pi
Figure 6: INSTANT performance with oversam- pling
leading to a small drop accuracy while reducing a huge number FLOPs by eliminating the projection onto redundant subspaces. In addition, this also helps to achieve more efficient memory storage. As we observe, with the same energy threshold e, the rank R2 of truncated activation & is typically smaller than the rank Rg of truncated gradients By and much smaller than the natural rank L of activation x. Therefore, INSTANT requires much lower activation storage than Vanilla.
5 CONCLUSION
In this study, we introduced INSTANT, highly efficient backpropagation method for Transformer models that targets two key bottlenecks in resource-efficient training: memory consumption and computational complexity. INSTANT constructs efficient low-rank projectors to compress both the activation during the forward pass and the gradient of the output during the backward pass (Sec. B). We demonstrate the low-rank nature of the activation gradient (Sec. 4.2). and show that jointly compressing activation and gradient substantially reduces the computational overhead of the training (Sec. (4.) This work investigates the relatively underexplored area of gradient compression and low-cost tensor decomposition as means of scalable, resource-efficien model training.
ACKNOWLEDGEMENTY
Part of this work funded by Hi!PARIS Center Data Analytics and Artificial Intelligence, by the European Union's 's HORIZON Research and Innovation Programme under grant agreement No 101 120657, project ENFIEL (European Lighthouse Manifest Trustworthy and Green Al) and by
10

---

Published as a conference paper at ICLR 2026
French National Research Agency (ANR-22 -2- 2----22-P2P222 12-PEFF-0003 and ANR-22-PEFT-0007) part of France 2030, the NF-NAI project and NF-FITNESS project.
REFERENCES
Jeremy Bernstein, Yu-Xiang Wang. Kamyar Azizzadenesheli, and Animashree Anandkumar, signsgd: Compressed optimisation for non-convex problems. In International Conference Machine Learning, pp. 560-569. PMLR, 2018.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance convolutional neural networks for document processing, In Tenth international workshop on frontiers in handwriting recognition. Suvisoft, 2006.
Christopher Clark, Kenton Lee, Ming-Wos Meig-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1005044905.10044 2019.
MMCV Contributors. MMCV: Mpen computer vision foundation. https://giit
com/open mm. ab habb/mmcy 2018.
Jacob Devlin, Ming-Wad Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). pp. 4171-4188 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition scale, 2021. URL wnt.t::////// // httpaaxiivv...o org/ abs / 2000000000111 11929
Yuan Gong. Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2 01778, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 61 611-62,, 2023.
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 15:12234-122444 2022.
Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for mobilenet size and speed. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 16889-16900, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.

---

Published as a conference paper at ICLR 2026
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision. pp. 10012-10022, 202
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12009-12019, 2022.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092-2810
2021b.
Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. Velora: Memory efficient training using rank-1 sub-token projections. Advances in Neural Information Processing Systems, 37: 42292-42310, 2024.
Aashiq Muhamed, Oscar Li, David Woodruff, Mona Diab, and Virginia Smith. Grass: Compute effi- cient low-memory Ilm training with structured sparse gradients. arXiv preprint arXiv:2406. / 17660, 2024.
Le- Trung Nguyen, Acl Quélennec, Enzo Tartaglione, Samuel Tardicu, and Van- Nguyen. Activation map compression through tensor decomposition for deep learning. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https //openreview.net//ooee wtt.orrwwmm..dss
M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 066) volume 2, pp. 1447-1454. IEEE, 2006.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEFE, 2012.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-255 2015.
Charbel Sakr and Brucek Khailany. Espace: Dimensionality reduction of activations for model compression. arXiv preprint arXiv:2410.05437, 2024.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 2019.
Yara Shamshoum, Nitzan Hodos, Yuval Sieradzki, and Assaf Schuster. Compact: Compressed activations for memory effinoreefffiientttt Ilm training. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1511-1524, 2025.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 15638- 15650, 2022.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint 2rXiv:2004.02988 2020.
12

---

Published as a conference paper at ICLR 2026
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rXiv:180.007460 2018.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation- awarte singular value decomposition for large language model compression, 2025. URL httptt://axxiv..oo
abs/2403.075 07378
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-ther oftt--o--he-art art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, and Song Han. Coat: Compressing optimizer states and activation for memory-efficient fp8 training. arXiv preprint arXiv:2410. 19313, 2024.
Greg Yang. James B Simon, and Jeremy Bernstein. A spectral condition for feature learning, arXiv preprint arXiv:2310.17811 2023a.
Yuedong Yang, Hung- Yueh
Hung-Yueh Chiang, Guihong Li, Diana Marculescu, and Radu Marculescu. Efficient low-rank backpropagation for vision transformer adaptation. Advances in Neural Information Processing Systems, 36: 14725-147766 2023b.
Yuedong Yang, Guihong Li, and Radu Marculescu. Efficient on-device training via gradient filtering. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 3811-3820, 2023c.
Gyeong-In In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orea: A distributed serving system for (Transsormer-BBseee)) generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0588 2023.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient I1m training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1466 2017.
13

---

Published as a conference paper at ICLR 2026
A
MATH NOTATIONS.
We provide the table of all our math notations:
Table 5: Table of symbols
Symbol
Space
Meaning
R
Sequence length
C
R
Layer input features
Cy
E
Layer output features
R
I
Rank of actívation in low-rank space
Ry
IR
Rank of gradient activation in low-rank space The number of training steps for each calibration The i-th eigenvalue The energy threshold
Nt
R
R
R
3
R
Sum of squares of all eigenvalues The eigenvalue's index keep total energy Oversampling value
E
R
R
Loss value used to backpropagate Input of a linear layer without batch dimension
LxC,
X
y
R RAxC,
Output of a linear layer without batch dimension
R xCxx²=
Weight of linear layer
gx
R LxCx
Gradient w.r.t input of linear layer without batch dimension
By
R LxC,
Gradient w.r. output of linear layer without batch dimension
P
R XL
Low-rank matrix to project activation into low-rank space
Q
R R
Low-rank matrix to project gradient activation into low-rank space
B PROJECTION CONSTRUCTION ISSUES
This appendix provides details for Sec. 3.2, including explanation (Sec. B.1 theoretical foundation (Sec. B.2). and our efficient calibration approach (Sec. B.3).
B. WHY TWO COMPRESSION TENSORS REQUIRED FOR ACTIVATION AND GRADIENT COMPRESSION?
Low-rank approximation method reduces computational complexity by doing multiplications in the smaller low-rank space. Given A B with A E mm=x" B E Rexp, this multiplica- tion is Omm n p) complexity. Low-rank approximation introduces low-rank matrix e prxm (r min(m, n, p)) such that A =e P P A. The original multiplication becomes:
((P -). The right expression in equation 6 is of complexity O(r (n p n p)) < O(m p).
A.B P.A.P-BBBBP
(6)
The problem of this decomposition size (m m), because rank(P P) rank(())
PT. P Imxm with Imxm be the identity matrix of &
raank[mxx)) Singular Value Decomposition (SVD) is used to find good approximation that, given a full-rank matrix A, SVD decomposes A E. V with U E mm being left singular vectors. If we assign multiplication in equation the approximation in equation is O(m (n-p P- m n m p)) Z O(m n. p).
for PT P. A because P P Imx m- but the complexity of
Therefore, we need to truncate U to U, (Sec. 3.2) to have & min(m, n) to reduce computations. Although U.T U, # Imxm. with appropriate truncation strategy, U.T U, = A. This approxi- mation with complexity of O(r (n p m + m p)) << O(m p) so assigning P = U, satisfies and significantly reduces computational cost.
Why do we need both P. Q compression matrices?. Why do we not only involve P for both activation compression(x P, x) and gradient compression(g, P, gy Assuming that we decom- pose activation x and truncate its singular vectors get P U,. The truncated U, successfully
14

---

Published as a conference paper at ICLR 2026
recovers but does not work with By - U, gy gy) and U U, * mm well. Therefore, it is required to decompose By to get an additional compression matrix Q for a correct approximation of this output gradient.
B.2 PROOF OF ESPACE THEEORE
Given vector x E X, subspace P and X E X is the recovery of x, the mean squared error (MSE) of the decomposition is defined as:
MSE x) = Blix x1?
where x P.P.PPP....
P EE (Pi. x)p
(pil/ are transpose of orthonormal row vectors of i.e.
(pi, Pı) ((m=)> Vi,j (I...., L}.
(8)
Given x and &, we examine the squared error:
||x x||²
||x|
| - 2x x
(9)
=|
| 2 2
(p. x) Pi x
(10)
L
=
||x|2 + || - 2 Σ (p.
To x)2
(11)
We have L2 norm x:
1.
T
1.
x 2
x
x
Σ (p. x) pi
Σ
(p. x) pi
(12)
i=1
and since Eq.
we have:
|
Σ (P.T.. x))
(13)
We reconsider the expression for the squared error:
||x - x||² = -|x1||
+
(p.
x)²
-2
(p.
T x)2
(14)
i=1
i=1
||x
(p.
x)2
(15)
i=1
We take expectation on both sides and obtain a formula for the MSE:
L
E x x
2 =
E |||
E
(p.
x)2
(16)
[==
=B
[
E
Σ P (x-xx)
xT) ())pp. pi
(pT x pi)
(17)
|||
Σ
p..
E
x
x T| Pi-
(18)
In this expression, E [|x]|]]] does not depend on (p.) and therefore, minimizing the MSE is equivalent to maximizing the following expression:
t
L.
Σ
p.
E
x
x _1 p.
=
Σ
p Cx
p..
(19)
15

---

Published as a conference paper at ICLR 2026
Because the set {p,) forms orthonormal basis, the expression be interpreted as sum of Rayleigh quotients: E R(Cx. pi) = E Cx Pj- The Rayleigh quotient is maximized when {px are the eigenvectors corresponding the largest eigenvalues Cx. Therefore, the optimal choice is select (p: the top eigenvectors of the auto-correlation Cx.
B.3 LOW-COST CALIBRATION PROOF
Considering our targets of reducing both peak memory and FLOPs, our calibration needs to be efficient, i.e does not increase peak memory and adds small extra computation cost.
INSTANT calibration does not increase peak memory. To reduce peak memory consumption in the calibration step, propose 2 strategies:
On-policy processing, Instead of saving multiple x and gy for post-processingg after each calibrating iteration, the data batch is processed to store key elements and deleted afterwards. This approach does not increase the processing time.
Smaller batch, In each training step, B samples are put into the system. In calibration, we set batch to samples, where n, is the number of mini-batcen inside one batch.
Algorithm Efficient Calibration Without Memory Accumulation
Require: Model M, number of calibration iterations N₂, calibration batch samples B, m, parameters Op-
initial
1: Data processing Calibration dataloader D. containing No Bg calibration samples.
2: Initializing Set dictionaries A and G
3: for each calibration batch be De do
4: Freeze model parameters and optimizer states.
5: for each layer do
6: X. By M(b.)
7:
Cx - x
Cg t By gy
8:
8x E C
B.
8G Σ.. Cali
9:
A + A[] + 8x
G[]] <- G[l + 8G
10: end for
11:
end for
for each layer do
Ux, Ex SVD(A(Z)
UG. Eg SVD(G(I) Q + OVERSAMPLE(U
14: ++ OVVERSAMPLLE
15: end for
Instead of storing x, gy with 4 Nz B- C2 and N. B. Cy bytes, respectively, we just store dictionaries A, G. These dictionaries have the same size of 4- Nr. T,² bytes (Step 9 in Algorithm[1 [ with Nr the number of layers that we implement INSTANT. In application, adaptively calibrate to get each dictionary of size 4 Nr, min(L, C) and N₂ < to preserve memory.
In summary, the calibration step does not increase peak memory due to on-policy processing of x, gy- smaller batch, and freezing optimizer states strategies (Alg. [ Moreover, propose dynamic data processing in which the number of mini-batchin n,, can be adaptively increased to reduce peak memory of the calibration step.
16

---

Published as a conference paper at ICLR 2026
INSTANT calibration adding small extra computation. thanks to long-term subspaces utiliza- tion, which is proved via the following analysis of the ratio between calibration FLOP (flope) and training FLOP (flop:).
Experiment setup: 5 calibration iterations for each N, = 200 training iteration (as in our experiments). For the sake of simplicity, our linear layer: y x w" with æ R BxLxC. RC,xx, v BxLC,
FLOP training: slop: = 200 (6 B I. Gz Cy) (forward and backward pass)
FLOP calibration (Alg : So = 55- L. C. B.- Cy) (step 6) +5- - (step 7) +14/3 1 + 4/3 1.3] (SVD cost, step 13).
30 C, Ratio: fe/f=
1 (L. < min Cy))
1200-B-L.C
40
Therefore, FLOP of calibration is small compared FLOP of training. Taking Tab 1 example, INSTANT-5 with reported 475 MFLOPs for finetuning EfficientFormerL. requires extra 37 MFLOPs for calibration. Therefore, only reporting training flop still reflects fair comparison with other methods.
C PROPAGATED ERROR ANALYSIS
Similar Sec. 3.2, we denote E gy gy T| CG =
= Σte 0; u, u. where u, is the 5th
column of left singular vectors Q when decomposing Cg. Let M Q notably that Q Q ΣE u, u. The signal-to-noise ratio is calculated as:
2 By
o T
E
w gy W
1
SNR
(20)
2 2
We have: M = Q = EB, 6-1 u. + Ei=R, u, u,
and: M gy
w12 2
M gy w where:
R
M
gg
W
1
(u [ gy w)))+ +
Σ (u, (u. By w,))
(21)
i=1
Rl
[1
gy Wj u,) +
(u. gy W, w.))).
(22)
c
R
17

---

Published as a conference paper at ICLR 2026
As a result:
B2
M By wyll =
c 6-1
(u. By w, w. us)
T (u. gy w, u.)
1=1
(n
Bg w uz) (u. By W u.)
(23)
Ryt
R
2
(n. By w2) !!
(n. gg w2) u,
(n. gy w uz (u. By Wj u:)
(24)
R+
R
(w) Ey u:) (u, gv W3) u.
( gy : u.) (u. gy wy) u, u,
(25)
Ry
Ry
2
(w) gy u:) (u. gy wj)
f=1
(w] By us) (n. gy w,)
(26)
R
Therefore, the expectation is calculated as:
E |I gy w112
E
gy
u.
gy
wj
W
E
By
u,
uf
gy
Wj
(27)
R2
1
0
W
W +
σ W
W
(28)
t= R
(I
E
2
Ry
+
L. Σ
| 1
(29)
2
imi
im Rl
1
Similarly, we have:
R2 pi
2
E
Q
Q
||
(30)
gy
w
2
1=1
Therefore, the signal-to-noise ratio is computed as:
Rg
E
q Q By
w
1
0x
SNR-
i=1
(31)
E
M gy w|/
L
(1-e)²
0
d
i=
R.y
+1
18

---

Published as a conference paper at ICLR 2026
SNR depends on energy threshold
8
6
SNR
5
4
3
2
0.800 0.825 0.850 0.875 0.900 0.925 0.950 0.975 1.000 Energy threshold (E)
Figure 7: Signal-to-noise ratio analysis of one layer. When the energy threshold goes to I, the SNR ratio will increase enormously. With large enough value of energy threshold and oversampling, INSTANT can guarantee a high SNR ratio of training.
When R, + L, therefore, SNR oo, and error will become 0 shown in Fig. 7 Here, is noticeable that the plot has some segments with breaks. This is because with some near the same rank Ry is selected.
D FLOPs AND MEMORY ANALYSIS
In this section, the FLOPs and memory analysis for our INSTANT (Sec. D.I and LBP-WHT
(Sec. D.3) are described.
D.1 FLOPS AND MEMORY ANALYSIS FOR INSTANT
By compressing activation at the forward pass, we are able to reduce the matrix storage from L2 RCC=2 by only storing two additional tensors P R. x and Q E R,xL As- suming that the activation is stored in fp32, which is 4 bytes for each value, the total memory for storing an activation sample at one layer reduces from 4 C22, to Rg Cz. It is noticeable that P and Q remain constant across batches, meaning that even when training in mini-batch setup, as typically do, P and Q are still woo-dimensional tensors. This ensures that the memory cost for storing them is negligible.
In terms of computation, our training involves three main computational cost components: low-rank projection, low-rank matrix multiplication, and reverse projection. Since there three low-rank projections, the total FLOPS of these projections FLOP, P, is:
FLOP, = 2 R2 I. C. + Ry L C.y 2 Rs. T. Cy-
(32)
Eq. and Eq. 5 illustratee two low-rank matrix multiplications of INSTANT. Since both of them are computed in low-rank spaces, the computational complexity FLOP, is:
FLOP, = 2. Rg Cy C2 2 R CC...
(33)
The cost for reverse projection of gx is FLOP, = 2 R.y L C.. Ultimately, the total computational complexity of our backpropagation process F FLOP, is:
FLOP: FLOP, FLOP., FLOP, 2(R R2)(C, Cy + I- C. L. Cy) Note that this is the computational cost one sample at one layer; for training one batch B samples, the total computational cost is B FIOPP.
(34)
19

---

Published as a conference paper at ICLR 2026
D.2 FLOPS AND MEMORY ANALYSIS FOR GRADIENT FILTER
The Gradient Filter splits the gradient and activations into small patches, then applies pooling to each patch reduce the number of unique elements. By storing only these unique elements in the memory, the Gradient Filter can save 64x x memory compared to vanilla training with a patch size 8 x 8. Let:
11/8),
Pue [w/8].
(35)
Gradient Filter compresses the activation x E REXC. with L. h x w into E R''''= with L' Ph x Pu- The activation storage is calculated 4 C2 Ph PPo- Similarly, the gradient gy R is compressed into gy E RL''C, The total compression cost for activation and gradient is:
FLOP. (C2 Ph 8
(36)
After compressing, by doing both 2 matrix multiplications in the low-rank space, the total computa- tional cost in hackpropagation of Gradient Filter is:
FLOP, = 4 x C2 x C. C,
Cy Ph Pw
(37)
Finally, the total computational cost of backpropagation with Gradient Filter is: =LOP=FLOM + FLOP. x C₂ x C2 x Ph x Pw + (Ca + Cy) ph Pp x 8 (38)
D.3 FLOPS AND MEMORY ANALYSIS FOR LBP-WHT
LBP-WHT is originally designed for computer vision tasks. Considering a linear layer, let x E RLxC, as the input. L.P-WWT first splits the sequence length into H and W, such that I, = H.W, then reshapes to x E RCcxHXW, Afterward, at each channel, the matrix shape [H, W' is split into Ph Pw) patches (8 8) with:
Ph [h/8], Pw [w/8]
(39)
At each patch, the the WHT, where R is the hyperparameter. The FLOPSs of this projection W WFLOP, pz is:
transformation is applied with only R out of the total 64 bases of
WFLOP pz = 2 R. C. Ph. Pw 8²
(40)
where 82 is because of the patch size 8 x 8. This compressed version is stored in memory instead of the original x, therefore, the activation storage is: 4 R C2 Ph Pw-
In the backward pass, the gradient is projected by the same transformation as the activation. Therefore, the computational cost W FTOPP pg is:
WFLOP 2 R- Cy Ph Pw 82.
(41)
LBP-WHT does two matrix multiplications in the low-rank space calculate the weight gradient and compressed input gradient. The total cost of these multiplications is:
WFOOP,,. = 2 R. Cy Ph Pw Cz + R Cy C2 Ph Pw + R Oz Cy-
(42)
Finally, the compressed input gradient is mapped to the original space to propagate to the previous layer with cost:
WFLOP, = R C. Ph Pw 8
(43)
In conclusion, the total FLOPs of LBP-WHT is computed as:
WFLOP = WFLOPp pa + WFLOP, pg + WFLOP + WFLOP,
(44)
4 R. C2 C. Ph Pmo R- C. Cy + 256 R C. ph. Pw + 128 R - Cy Ph P
(45)
We recommend that the readers be familiar with LBP-WIH WHT (Yang et al. 2023b) paper and the source code for better understanding of the calculation in each step.
20

---

Published as a conference paper at ICLR 2026
E
DATASET DESCRIPTION
Computer vision tasks.
CIFARIO (CF10) (Krizhevsky et al., 2009) is a widely used image classification dataset consisting of 60,000 32 32 color images across 10 classes, such as airplane, cat, and truck. It includes 50,000 training images and 10,000 test images, evenly distributed across classes.
CIFAR100 (CF100) (Krizhevsky al. 2009) is image classification dataset similar to CIFAR-10 but with 100 classes. It contains 60,000 color images of size 32 x 32 pix- els-50,000 for training and 10,000 for testing. CIFAR-100 is more challenging than CIFAR-I0 due to the larger number of categories and finer-grained distinctions between them, making a popular benchmark for evaluating model performance more complex classification tasks.
Flowers (Nilsback & Zisserman, 2006 contains 8,189 images of 102 flower categories found in the UK. Each class has between 40 and 258 images. The images have high variability in scale, pose, and lighting, making the dataset useful benchmark for testing the robustness of computer vision models in fine-grained classification.
Food-101 (Bossard et al. 2014) is popular benchmark for food image classification, containing 101,000 images across 101 food categories such as pizza, sushi, and apple pie. Each class has 1,000 images-750 for training and 250 for resting-wite real-world variability in presentation, lighting, and background.
The Oxford-IIIT Pet PParkh et al. 2012) is popular image dataset for pet classification and segmentation tasks. It contains 7,349 images of 37 different breeds of cats and dogs, with roughly 200 images per breed. The dataset features high variability in poses, lighting, and background, making it ideal for training and evaluating models in fine-grained classification.
Natural language processing tasks.
QNLI (Question Natural Language Inference) has around I 110k samples derived from the SQuAD dataset and reframed as a binary classification task, where the model determines given sentence contains the answer question (entailment not).
SST-2 (Stanford Sentiment Treebank) a sentiment analysis dataset with about 70k movie review sentences. Each sample is labeled as either positive or negative, making it simple and clean binary classification task for sentiment understanding.
MNLI (Multi-Genre Natural Language Inference) contains 433k sentence pairs from various text genres. The task is to determine whether hypothesis is entailed, neutral, or contradicted by a given premise, making it a 3-class classification problem and a robust test of a model's reasoning ability.
CoLA (Corpus of Linguistic Acceptability) comprises over 10k English sentences labeled as grammatically acceptable or unacceptable. It evaluates a model's ability to judge gram- matical correctness, testing syntactic understanding.
MRPC (Microsoft Research Paraphrase Corpus) includes around 5.8k sentence pairs from news sources, with binary labels indicating whether the two sentences paraphrases. It's common benchmark for paraphrase detection.
RTE (Recognizing Textual Entailment) is a binary classification task with around 2.8k samples, where the goal is to decide whether a premise sentence entails a given hypothesis. It's small dataset, but it's useful for testing generalization.
F THE REASON BEHIND OVERSAMPLING
In this section, we will prove our argument that oversampling is helpful since the energy threshold only guarantee the amount of information when the calibration happens. More concretely, will show that if we calibrate more, the oversampling is unnecessary. Conversely, we also show that oversampling can help even without calibration.
21

---

Published as a conference paper at ICLR 2026
Accuracy vs Calibration frequency
Accuracy vs Oversampling Value
With Calibration
66
Over-sampling
No Calibration
R
Accuracy (96
Accuracy (%)
22
60-
77
70-
16-
76-
175
74
Number of Steps after Calibration
Over-sampling Value
Figure 8: The effectiveness oversampling, This experiment is conducted with BERT on the QNLI dataset.
Table 6: INSTANT combine with LoRA to achieve low-rank space of weight, activation, and gradient.
Model
Method
MFLOPs |
Mem
CF10 f
CF100 1
LoRA
896
1.95
94.48
77.27
LoRA + INSTANT-0
335
0.06
93.91
75.79
Efficient
LoRA + INSTANT-5
413
0.26
94.35
76.73
Former-L1
LoRA + INSTANT-7
449
0.34
94.38
76,80
As shown in Fig. without oversampling, increasing the calibration frequency (or reducing the number of steps after each calibration) results in a remarkable increase in accuracy. This shows that when the calibration happens more, the energy threshold is enough for capturing the information. However, since the cost for calibration is larger compared to training with INSTANT, oversampling can help in reducing the frequency of this phase. addition, we experiment with only one calibration before training starts. Fig (right) strengthens our argument by showing that without oversampling, the performance degrades significantly. Meanwhile, with few values of oversampling, the accuracy increases approximately 10%.
G INSTANT ORTHOGONAL TO LORA
LoRA (Hu et al. 2022), and its variances, are popular for low-rank adaptation. However, this research focuses on the low-rank characteristic weights, while INSTANT applied to the low-rank space of the activation and gradients. We, therefore, provide an additional ablation study to show that these 2 methods are orthogonal and can be combined.
While LoRA successfully reduces the number of trainable parameters, it still requires storing the original activations in memory to compute gradients, thus failing to reduce memory usage. Further- more, LoRA necessitates the multiplication of two large matrices to compute the activation gradient, which is then backpropagated to the previous layer. As a result, the computational cost and memory of applying INSTANT much smaller compared to LoRA only, shown in Tab. 6
H EXTENSION TO CONVOLUTION
H.1 CONVERT CONVOLUTIONAL OPERATION TO LINEAR OPERATION
Many convolutional architectures (ResNet, VGGNet....) utilize image-to-column transforma- tion (Chellapilla et al. 2006) when working on convolutional operations. By stretching matrices,
22

---

Published as a conference paper at ICLR 2026
this transformation turns convolutional operation into linear operation to utilize matrix multi- plication optimization of deep learning libraries. We apply this idea to transform the convolutional operation into linear operation, before applying tensor decomposition and low-rank backprop- agation (Sec. 3.2 3.3 to save memory and computation. Due to the low-cost transformation of image-to-column, achieve up 3x computation reduction.
H.2
RESULTS
shows the efficiency of INSTANT on 2 popular convolutional architectures: ResNer-50 and MobileNetV2, on datasets CIFARIO and CIFARI00. In all experiments, our method can save significant memory and FLOPs while maintaining high accuracy, compared to Vanilla. Specifically, INSTANT-5 and INSTANT-7 consistently outperform Vanilla in all three reported metrics. ResNet- 50, INSTANT-5 gains better performance with 3x in FLOPs and 79x in memory. These results indicate the efficiency INSTANT on all architectures, including both Transformer-based and Convolutional-base models.
Fig.
I TRAINING LATENCY ANALYSIS
1.1 EDGE DEVICES TRAINING LATENCY
Table 7: Iteration and backward time when finetuning 9 last layers of ViT-B/32 on CIFARIO, We
also report backward speedup compared to Vanilla.
Raspberry Pi5
Intel E5-2267
Method
Total time (8)
Backward time (s)
Speedup
Total time (s)
Backward time (S)
Speedup
Vanilla
5.52
2.69
Gradient Filtering 1.85
0.84
2.17x
0.76
2.37x
LBP-WHT-4
20.67
14.23
0.39
14.83
9.76
0.18x
INSTANT-0
1.43
0.44
12.55
1.19
0.22
8.18x
We provide extra results of INSTANT on EfficientFormer--I11 on CPU Intel E5-2667 in Fig. T0, We conduct experiments with the same setup as in Sec. [4. using the PyTorch framework. Considering the larger architecture ViT-B/32, in which INSTANT-0 can save 17x computation compared to Vanilla.
We provide additional results of INSTANT on ViT-B/32 using Raspberry Pi 5 and an Intel B5-2267 CPU, as presented in 7 Noticeably, we can save 8x backward time on Intel E5-2267 and 12x backward time Raspberry Pi 5. The (12x) time reduction is not comparable to (17x) FLOP reduction [T). This reduction gap is reduced when implementing in other frameworks, such as CUDNN, as shown in Gradient Filter (Yang al. 2023c). We also observe that the time reduction strongly affected by the use the device. However, these engineering-leve optimizations fall outside the paper's scope. Generally, computation reduction converted into time reduction, which proves the efficiency and applicability of INSTANT.
1.2 GPU TRAINING LATENCY
We conducted experiments on the framework of MMCV, model EfficientFormerLL. with datasets CIFARIO, and with all methods (LBP-WHT, Gradient Filtering, INSTANT (ours), Vanilla) as provided in Tab. [ We observed similar results on CIFAR100. As shown in Tab. g on V100 GPU, INSTANT's backward time is 1.4x compared Vanilla backwards time. The total training time (epoch time) of INSTANT is slightly higher than Vanilla. In short, INSTANT's FLOP reductions are not converted into time reduction like training on CPU.
J ABLATION STUDY
23

---

Published as a conference paper at ICLR 2026
Vanilla
INSTANT-0
INSTANT-5
INSTANT-7
ResNet-50
ResNet-50
ResNet-50
CIFAR10
CIFAR10
CIFAR10
Accuracy (96)
Memory (MB)
66
4000
64
93.18
23.08
3.5
8500
92.16
3.0
62
2500
882
20
15
1500
1159
00
05
0.04
0.0
Vesill
INSTANT-7
Varill
INSTANT-0 INSTANT-5 INSTANT-7
Venill
INSITANT- INSTANTS
ResNet-50
ResNet-50 CIFAR100
ResNet-50 CIFAR100
CIFAR100
Accuracy (96)
Memory (MB)
MFLOPs
70
4.0
77
3500
3.0
75
74-
73.48
73,22
73.06
25
2500
2000
73
1..
72
100
1000
0.04
70
0.
0-
Varilaa
INSITANT-4 INSTANT-S INSTANT-7
Vaillu
INSTANT-O INSTANT-S INSTANT-7
Verilu
INSTANTE INSTANTS INSTANT--
MobileNetV2
MobileNetV2 CIFAR10
MobileNetv2
CIFAR10
CIFAR10
Accuracy (96)
Memory (MB)
MFLOPs
00
54
93.06
92.16
93.08
93.38
3.0
0.87
358
0.8
300
80
0.4
08
114
0.2
100
64
0.04
Verill
INSTANT-- INSTANTS INSTANT-7
Vinln
INSTANT-O INSTANTS5 INSTANT-7
Wella
INSTÄNT- INSTANTS INSTANT-7
MobileNetV2
MobileNetV2 CIFAR100
MobileNetV2 CIFAR100 MFLOPs
CIFAR100 Accuracy (96)
Memory (MB)
1.0
0.87
358
08.48
67.02
0.5
200
0.4
84
114
68
0.05
0.06
0.06
Verils INSTAANT-1 INSTANTTS INSTANT-7
Vini INSTANT-O RISTANT-S INSTANT-7
Velli
INSTANT INSTANTS INSTANT-7
Figure 9: Performance of INSTANT on convolutional architectures.
J.1 INSTANT WITHOUT CALIBRATION
To highlight the necessity and efficiency of our calibration scheme, we also provide extra baseline: Random which is INSTANT using random subspaces P, Q for compressing activation and activation gradient gy. The difference that in this Random scheme, we periodically update P, Q by random Gaussian matrices each N, 200 training iterations.
24

---

Published as a conference paper at ICLR 2026
Vanilla
INSTANT
LBP-WHT-4
GF
Backward Time
Vanilla
1.96
INSTANT
0.86
LBP-WHT-4
4.49
GF
0.50
2
3
4
6
Total Time
Vanilla
3.00
INSTANT
1.95
LBP-WHT-4
7.56
GF
1.59
2
4
6
8
Time (s)
Figure 10: Training time over epoch on CIFARIO using a Intel ES-2667.
Table 8: Time reported for training I epoch with V100 GPU when partially and fully finetuning ElficientFormerLl on CIFARIO. These reported numbers are averaged over 10 training epochs.
Partially Finetuning Last Layer
Full Finetuning
Method / Time(s)
Calibration
Backward*
Total epoch
Calibration
Backward**
Total epoch** ***
Vanilla
6.11
73.6
0
12.66
110.8
Gradient Filtering
7.77
74.0
0
15.07
94.2
IBP-WHT-2
7.99
79.8
0
19.6
146.4
LBP-WHT-4
9.02
123.8
0
18.55
246.0
LBP-WHT-8
10.45
291.4
0
20.5
625.0
INSTANT-0
2.16
8.8
74.2
4,99
17.55
116.6
INSTANT-S
2.2
9.01
74.0
4.98
17.68
120.0
INSTANT-7
2.22
8.73
74.2
4.98
17.17
120.2
(+)
Calibration time is total time used for creating periodically updated subspaces during training epoch
(2*) Backward time is total time of loss backward () during training epoch.
(***) Epoch the total running time training epoch, including calibration time, forward pass, activation
savings, loss calculation, backward pass, optimizer update....
The experimental results (Tab, 9 Tab. 10) demonstrate that INSTANT significantly outperforms the Random projection method, where the compression rank is set to π or & with representing the input dimension. This highlights that INSTANT effectively requires compact yet precise low-rank space to retain gradient information for accurate backpropagation through the network. Notably, in the case of full fine-tumi fine-tuning, the Random projection method fails to converge, as the gradients struggle to propagate through multiple layers, resulting in significant error accumulation due to the lack of well-defined low-rank space.
25

---

Published as a conference paper at ICLR 2026
Table 9: Partially finetuning last layer of ElficientFormer-LI on CIFARIO, CIFARIO0 with INSTANT- random (Appendix. J.1p and ESPACE (Appendix. K. methods.
Method
CIFARIO
CIFARI00
Forward MFLOPs
Backward MFLOPs
Activation Mem (MB)
ESPACE
95.23
414
0.96
ESPACE
N/4
94.21
76,26
212
414
0.48
Random
N/2
86.52
73.13
742
827
0,96
Random -N/4
64.97
46.02
742
414
0,48
INSTANT-0
94.66
77.64
742
270
0,16
INSTANI-5
95.07
78.65
742
475
0,38
INSTANT-7 95.23
79.01
742
544
0,45
Vanilla
95.23
79.28
742
1484
1.95
Table 10: Full finetuning EfficiendForerr-LI on CIFAR10, CIFAR100 with INSTANT-random (Ap-
pendix. J.T and ESPACE (Appendix. K.1 methods.
Method
CIFARIO
CIFAR100
Forward MFLOPs
Backward MFLOPs
Activation Mem (MB)
ESPACE N/2
93.16
74.56
2347
4694
4.29
ESPACE N/4
51.18
35.62
1173
2347
2.14
Random
N/2
13.47
3.66
2264
4694
4.29
Random N/4
10.00
9.26
2264
2347
2.14
INSTANT-5
96.29
82,41
2264
2107
1.98
INSTANT-10
96.48
83.05
2264
2491
2.73
INSTANT-15
96.85
83,56
2264
2834
3.45
Vanilla
96.99
84.84
2264
4528
18.46
K
ADDITIONAL RESULTS
K.1 COMPARISON OF INSTANT WITH ESPACE
ESPACE primarily focuses on compressing activations only, whereas our main objective is to jointly compress both activations and activation gradients. In trade-off to forward computations, ESPACE performance degraded compared to INSTANT due to error accumulations, even in the forward pass. Noticeably, using quite high dimension compressions - or N observes limited overhead reductions compared to INSTANT. More experimental results are shown in Tab. and Tab. I0
- ESPACE
K.2 RESULTS ON LARGE ARCHITECTURE FOR VISION TASKS
Vision Transformer achieves extraordinary performance on image classification. We conduct exper- iments with VIT-B/32, variant of Vision Transformer (Dosovitskiy et al. (021) with 88 million parameters, which 7x EfficientFormer-LI with 12 million parameters. In terms of accuracy, Gradi- Filtering fails, while LBP-WHT-4 performs their strategies are inappropriate for ViT-based architectures. Conversely, INSTANT with SVD-based compression preserve significant information while achieving good compression rate with negligible performance drop. Noticeably, INSTANT-7 outperforms Vanilla all reported metrics, including accuracy.
than EfficientFormer-L which suggests that
K.3 RESULTS ON COMPLEX PLACES-365 DATASET
We finetuned EfficientFormer-LI
Places-365 ZZou al. 2017) dataset, which contains 1.8M images and is more challenging than ImageNet, i.c models often perform worse on Places-365 than on the ImageNet dataset. The table above demonstrates that INSTANT remains highly effective even with large datasets such as Places-365. All three versions INSTANT show only a minor reduction in performance compared to vanilla training, despite achieving threefold reduction in
26

---

Published as a conference paper at ICLR 2026
Table 11: Experimental results on CIFARIO when fine-tuning last layers ViT-B/32. We report the MFLOPs and memory (Mem) required for training single sample.
Model
Method
MFLOPs 4
Mem
CF10 1
Vanilla
2831
2.20
96.56
Gradient Filtering
63
0.07
30.38
ViT-B/32
LBP-WHT-4
863
0.70
92.1
INSTANT-0
161
0.11
96.36
INSTANT-5
445
0.31
96.36
INSTANT-7
567
0.40
96.6
Table 12: We finetuned the last layer of EffficientFormer-Lll Places-365 dataset.
Method
Accuracy
Backward MFLOPs#
Activation Memory (MB)
Vanilla
55,30
1484
1.95
Gradient Filtering
9.2
24
0.04
LBP-WHT-2
50.92
95
0.12
LBP-WHT-4
53.27
335
0,40
LBP-WHT-4
54.67
1227
1.43
INSTANT-O
54.32
388
0.37
INSTANT-5
54.57
567
0.60
INSTANT-7
54.55
606
0.66
(+) The reported Backward MFLOPs includes forward compression cases INSTANT, Gradient Filtering, and LBP-WHT
both computational cost and memory usage. When compared to LBP-WHT, INSTANT offers slightly better performance while maintaining the same memory and computational budget. In contrast, Gradient Filtering fails to converge, yielding poor results.
K.4 RESULTS ON LARGE LANGUAGE MODEL (TINYLLAMA)
We conduct some additional experiments on TinyLlama (Zhag et al. 2024) on the BoolQ (Clark 2019) dataset with similar setup NLP tasks. The results reported in Tab. [3 It is noticeable that, with a bigger model, INSTANT only requires small rank to keep large amount of energy, thus reducing large amount of computational and memory consumption. However, this experiment, although INSTANT witnesses 3% drop in performance compared to vanilla training, INSTANT can save about 13x computational cost and 64x memory consumption. This makes training large model with more than billion parameters resource-constrrined devices possible.
Table 13: Liny results on BoolQ dataset
Fine-tuning the Last Block
Model
Method
MFLOPs I
Mem
BoolQ t
Vanilla
90194
35
67.71
LBP-WHT-
15083
5.47
63.94
TinyLLama
INSTANT-0
5517
0.07
63.88
INSTANT-5
7298
0.41
64.77
INSTANT-7
7465
0.55
64.89
27

---

Published as a conference paper at ICLR 2026
netwwrkk.32.mlp..
network.3.3.mp.rr2
network...5.tokoe mixer-pre)
Normaliead Energy
Nurmsleed Exarryy
Hurmaltan Essrroy
Sunple
Surple
Sungl
Soaple 1
Sxapl
02
Seaple
Eigenvalue
Eigenvalue
Elgesvalse
laver.6..tteation.co outee
layer.10. atteation.self.eyy
layer: Lattentto wet.tetttonsslifaaaee
Enwngy
No Mormallbed ELsooy
Sumpos
Semp4
Sesph
Sumple
Sumpe
Sespk
Semth
Cumpue
Seapee
Sasph
Eigensnton Index
Eigenvninn Index
Eigervatce Indwx
Figure 11: The normalized energy of eigenvalues of the gradient of many different layers in ElficrentFormer-LII and BERT.
INSTANT FLOPs an different datasets
INSTANT Peak Memory on different CV datasets
1800
LBP-WHHT
1000
1.2
Peak Memoryy (MBS)
1400
FLOP
1.000
LBP-WHTT4
0.8
1.000
800
LSP WMTT
LSPAWHT4
600
LBP-WIT-4
(P10) U:31 hootshowe Pes
(PIR) UFD POREPONTS PEB
cP000 0110
Taxts Plaves Pd
CFU0 0112 Pacde Fliess No
EftickntFomer--11
Swim2-Small
EfficienFForme-L
Swinv2-Small
(a) MFLOPs for different datasets
(b) Peak Memory different datasets
Figure 12: (a) MFLOPs of INSTANT when fine-tuning on 5 datasets with oversampling There are differences in MFLOPs of each dataset, however, all datasets, INSTANT has lower MFLOPs compared to LBP-WHT-8. (b) Peak memory of INSTANT when fine-tuning 5 datasets with oversampling p = The peak memory on different datasets slightly vary, and all datasets, INSTANT has lower peak activation memory than
5.
K.5 MORE SAMPLES ON LOW-RANK CHARACTERISTIC
Fig. II describes the low-rank characteristic of various samples through various layers of different architectures. In many layers, large amount of information be kept only few eigenvalues, proving that the compression tensor of INSTANT can work effectively on many blocks of the model.
K.6 EFFICIENT TRAINING- SUBSPACES
Fig. 12 and Fig. 13 illustrate the variation of FLOPs and peak memory between different datasets
when fine-tume tune with INSTANT. In CV tasks (Fig. 12 INSTANT with 0.95 and 5 has a comparable overhead to LBP- -HT-4, but our performance is better (Tab. 1). Compared to LBP. WHT-8, our overhead is extremely smaller in every dataset with trade-off of negligible accuracy.
28

---

Published as a conference paper at ICLR 2026
INSTANT Peak Memory on NLP datasets
750
0.83
Peak Memory (MB)
0.00
MFLOPs
650
0.55
0.50
0.45
500
400
VAU MADC OML
MIL MAPC QULLSATT
BERT
DistiBERT
DiStIBERT
MFLOPs for different datascts
(b) Peak Memory for different datasets
Figure 13: (a) MFLOPs of INSTANT when fine-tuning on 6 datasets with oversampling = 7. There are differences in MFLOPE of each dataset, however, all datasets, INSTANT has lower MFLOPs compared to LBP-WHT-2. (b) Peak memory of INSTANT when fine-tuning 6 datasets with oversampling p The peak memory on different datasets quite similar, and in all datasets, INSTANT has lower peak activation memory than LBP-WHT-2.
The higher overhead of Flowers is possibly due to its high variance samples, which require larger subspaces P, to capture sufficient meaningful information of tensors. In language tasks (Fig. 12). INSTANT with é = 0.95 and p 7 or = 15 outperform LBP-WHT-2 in every dataset. It is noticeable that activation storage in every dataset is similar. This possibly due the strongly low-rank characteristic of activation, especially in language tasks.
L LARGE LANGUAGE MODELS (LLMs) USAGE
We utilized an LLM support tool in preparing this paper. Its role was limited to:
polishing the clarity and flow of writing (but we do not use to generate new text)
assisting with retrieval and discovery, such as identifying relevant prior work and commonly used methods
All scientific design choices, methodological decisions, implementation, data analysis, and interpreta- tion of results were made solely by the authors.
The LLM did not contribute novel ideas or conduct experiments; it was used only as an assistant for writing and literature awareness.
29


Reviews:
[{"review_id": "d1KApzSLwv", "reviewer": "Reviewer_FBNs", "summary": "This paper proposes INSTANT, a training-time compression method that projects activations and gradients into low-rank subspaces and performs backpropagation in those compressed representations. The authors describe (i) an SVD-based calibration to build per-layer projectors, (ii) projection/truncation with an energy threshold + oversampling, and (iii) a low-rank backward algorithm. Experiments show large reductions in activation memory and FLOPs across Transformer and CNN models with small accuracy drops.", "questions": "* Were the authors aware of CompAct (NAACL 2025) at submission time?\n* What specific conceptual or methodological innovations distinguish INSTANT from CompAct beyond the choice of projection matrix?\n* Can you provide a direct empirical comparison to CompAct under matched conditions (e.g., same model, rank, dataset)?\n* How do the claimed FLOP reductions compare to a random projection baseline?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "* Solid empirical evaluation across modalities (vision & NLP) and architectures (Transformers, CNNs).\n* Clear exposition, reproducibility-minded appendices and pseudocode.\n* Practical relevance: reduces activation memory and backward FLOPs as demonstrated in their experiments, with sensible ablations (oversampling, calibration frequency, rank choices).\n* Sound theoretical guarantee for stable low-rank training. Their analysis shows that SVD-based projections minimize reconstruction error and that gradient approximation error remains bounded through depth, vanishing as the retained energy ε → 1.\n* Includes deployment-relevant experiments (edge device / Raspberry Pi).", "weaknesses": "1. **Severe overlap with prior work (CompAct, NAACL 2025, publicly available on arXiv since Oct 2024):**   \nConceptually highly similar and structurally parallel to CompAct [1], differing mainly in projection construction and calibration choices, without mentioning CompAct at all, despite the latter being published and publicly available months before ICLR submission. \nBoth papers:    \n* Compress activations via low-rank projections during the forward pass.\n* Compute gradients in the compressed subspace and decompress for weight updates.\n* Aim to reduce memory and optimizer overhead jointly.\n* Demonstrate scaling benefits on LLaMA and BERT-like models.\nThe algorithmic structure of INSTANT (forward compression → compressed backward → decompression for update) matches CompAct’s Algorithms 1–3. The overlap extends to terminology (“projected activations,” “reduced optimizer states”), theoretical justification, and empirical evaluation. \n\n2. **Novelty and contribution are overstated:**   \nWhile INSTANT adds implementation refinements, such as calibration-based rank selection and changes the choice of projection matrix, these are engineering extensions rather than conceptual advances. The claim of being the “first to jointly compress activations and gradients” is false, due to the overlap with prior work.\n\n3. **Lack of comparison:**   \nThe experiments do not compare with the most appropriate relevant works like GaLore [2] or any of the myriad of works that followed it (VeLORA [3] ,Grass [4] ,WeLore [5]...). Reported baselines are insufficient to establish novelty or superiority.\n\n\nReferences\n* [1] CompAct: Compressed Activations for Memory‑Efficient LLM Training – Shamshoum et al., NAACL 2025, arXiv:2410.15352v1. \n* [2] GaLore: Memory‑Efficient LLM Training by Gradient Low‑Rank Projection – Zhao et al., ICML 2024, arXiv:2403.03507.\n* [3] VeLORA: Memory Efficient Training using Rank‑1 Sub‑space Activations – Miles et al., NeurIPS 2024 Poster, algorithm for compressing activations into 1-D subspace. \n* [4] Grass: Compute Efficient Low‑Memory LLM Training with Structured Sparse Gradients – Muhamed et al., arXiv:2406.17660.\n* [5] WeLore: Weight Low‑Rank Projection for Memory‑Efficient Fine‑Tuning – Jaiswal et al., ICLR 2025, arXiv:2407.11239.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3SMPnWuqvm", "reviewer": "Reviewer_jz4Y", "summary": "This paper proposes a compression method called INSTANT. INSTANT is a method that projects the activations and gradients to lower ranks, based of analysis that is done at regular intervals during training. The authors show the benefits of compressing both the gradients and activations during training by showing reduction in FLOPs and training time (in some cases).", "questions": "1. Could the authors please provide the total wall-clock training time (not just backward time) for the main V100 GPU experiments in Tables 1 and 2? This is essential for evaluating the practical speedup of INSTANT in a standard training scenario.\n\n2. How is the computational cost of the periodic calibration step (Algorithm 1) factored into the total FLOPs reported? Could you provide an analysis of this overhead, for instance, as a percentage of total training time or FLOPs?\n\n3. Given that the subspace is recalibrated every $N_t$ steps, would \"periodically updated\" or \"cached\" be a more accurate description than \"static\"?\n\n4. The oversampling parameter $p$ appears critical for performance (Fig. 6, Fig. 8). How sensitive is the method to the choice of $p$ and the calibration frequency $N_t$? Does this introduce a significant hyperparameter tuning burden?\n\n5. Following up on the 17x FLOP vs. 12x time gap noted in Appendix I: What is the authors' hypothesis for the performance gap on a V100, where highly optimized dense matrix multiplication kernels are the standard?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is  tackling both memory and computational bottlenecks.  Activation compression for memory saving has been explored previously, but the idea of also compressing the activation gradient ($g_y$) to reduce backpropagation FLOPs is a useful contribution.\n\n2. The authors provide empirical evidence (Fig. 3) to support their intuition that activation gradients are inherently low-rank, justifying their compression approach.\n\n3. NSTANT's  SVD-based approach is data-driven. This allows it to generalize more effectively to other modalities, as demonstrated by its strong performance on NLP tasks (Table 2) where previous works (eg LBP-WHT) struggles.\n\n4. The authors validates the method's usability by reporting significant wall-clock speedups (2x to 12.5x) on resource-constrained edge CPUs.", "weaknesses": "I believe, currently this paper's primary weaknesses lie in the evaluation of its computational claims and the transparency of its overhead costs.\n\n**Omission of Wall-Clock Training Time on GPUs:**\n\n1. The paper's main results (Tables 1 & 2) were generated on an NVIDIA V100 GPU but only report FLOPs, not wall-clock training time.\n\na) The authors' justification (Section 4.1) that \"FLOPs... [are] unaffected by implementation details\" is a weak defense. A reduction in FLOPs does not guarantee a proportional reduction in training time, especially on GPUs. Modern deep learning libraries (like cuDNN) are highly optimized for large, dense matrix multiplications, and the low-rank operations introduced by INSTANT may not be as implementation-efficient, thus creating a bottleneck.\n\nb) The authors themselves admit a discrepancy in Appendix I, stating, \"The (12x) time reduction is not comparable to (17x) FLOP reduction,\" even on a CPU. This gap is likely to be even larger on a GPU, and the lack of this data makes it difficult to assess the practical speedup in a typical training environment.\n\n**Cost and Nature of the \"Static\" Subspace:**\n\n2. The term \"static\" subspace (used in the contribution list and abstract) could be misleading. The subspace is not fixed; it is \"periodically\" recalibrated every $N_t$ steps (e.g., $N_t=50$ or $N_t=200$).\n\na) The computational cost of this recalibration (Algorithm 1) appears to be non-trivial and is not accounted for in the reported per-step FLOP savings. This calibration requires running multiple batches and performing SVD for every layer being compressed. This overhead could significantly diminish the overall wall-clock time savings.\n\nb) The method's stability relies on an \"oversampling\" hyperparameter $p$, which is introduced to \"reduce information loss when the core bases change\" between calibrations (Section 3.2, Fig. 6). This suggests the \"static\" assumption is fragile and introduces another sensitive hyperparameter that must be tuned, adding to the method's complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BHK1EUWq8O", "reviewer": "Reviewer_9WbD", "summary": "This paper proposes a new distillation technique called grafting, aimed at improving the generalization of autoregressive language models. The grafting strategy integrates sequence trees generated at multiple temperatures into a single distillation target. The paper emphasizes the balance between model compression and generalization, addressing the mode-covering behavior crucial for building more generalizable models. The experimental validation demonstrates the method's effectiveness across several datasets, particularly CIFAR-10 and CIFAR-100.", "questions": "Could you provide a more detailed comparison between grafting and existing methods like Sakr & Khailany’s ESPACE or Yang et al.’s LBP-WHT, particularly in terms of accuracy and computational efficiency?\n\nHow do you plan to scale the grafting method for larger models or longer sequences, and what are the potential challenges?\n\nCould you provide further experiments on more complex datasets or real-world tasks to validate the method’s generalizability?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The grafting strategy is an original approach for balancing model compression and generalization in language model distillation.\n\nExperimental Validation: The method is validated across several datasets, and the results show some potential for improving model performance while reducing computational and memory costs.\n\nClarity: The paper is well-organized and clearly explains the experimental setup and methodology.", "weaknesses": "Lack of Comparison with Related Work: While the paper mentions the error accumulation problem in other activation compression methods, it does not provide a sufficient comparison with existing methods like Sakr & Khailany’s ESPACE or Yang et al.’s LBP-WHT in terms of accuracy and performance. Without this comparison, the claims about grafting’s effectiveness remain unclear.\n\nLimited Experimental Setup: The experiments are mainly focused on simple image classification tasks (e.g., CIFAR-10, CIFAR-100). These datasets may not fully demonstrate the method's potential in more complex or real-world scenarios.\n\nUnaddressed Limitations: The paper does not sufficiently explore some of the limitations of the proposed method, including the computational cost of SVD and its scalability to larger models or long sequences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)