TRAC: Tensor-Train based Across-layer Compression for Parameter-Efficient Fine-Tuning
Authors: Bangguo Ye
Yuanwei Zhang
Xiaoqun Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Parameter-efficient fine-tuning
Low-rank adaptation
Tensor decomposition
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose TRAC, a tensor-based extension of LoRA that enables extremely parameter-efficient fine-tuning while preserving strong model performance.

Abstract: Fine-tuning large pre-trained models under resource constraints remains challenging due to the massive number of parameters involved. Existing parameter-efficient tuning methods, such as low-rank adaptation (LoRA) and its variants, rely heavily on matrix factorization and often struggle in extremely low-parameter regimes. In this work, we propose TRAC, a novel fine-tuning framework that leverages Tensor-Train decomposition with Across-layer Compression. Specifically, TRAC represents each adaptation module as a compact sequence of tensor-train cores and allows certain cores to be frozen or shared across layers, thereby exploiting the inherent similarity and redundancy among layer weight matrices. To retain layer-specific flexibility, lightweight controllers are introduced, enabling shared tensor cores to adaptively modulate representations. We evaluate TRAC on diverse architectures, including Qwen, LLaMA, GPT, BERT, and ViT, across benchmarks covering text classification, text generation, and image classification. Experimental results demonstrate that TRAC achieves performance comparable to or better than LoRA and its variants, while substantially reducing trainable parameters and storage requirements.

Published as a conference paper at ICLR 2026
TRAC: TENSOR-TRAIN BASED ACROSS-LAYER COM- PRESSION FOR PARAMETER- EFFICIENT FINE-TUNING
Bangguo Ye', Yuanwei Zhang' ' Xiaoqun Zhang" 2,
School of Mathematical Sciences. Shanghai Jiao Tong University, Shanghai, China 2 Shanghai-Chongqing Institute Artificial Intelligence, Chongqing, China Institute of Natural Sciences, Shanghai Jiao Tong University. Shanghai, China (bgye, sjtuzyw, xqzhang @ ftu.tuu.. edu. en
ABSTRACT
Fine-tuning large pre-trained models under resouree constraints remains challeng- ing due to the massive number of parameters involved. Existing parameter-efficient tuning methods, such as low-rank adaptation (LoRA) and its variants, rely heavily matrix factorization and often struggle in extremely ww-parameter regimes. In this work, we propose TRAC, a novel fine-tuning framework that leverages Tensor-Train decomposition with Across-layed Compression. Specifically, TRAC represents each adaptation module as a compact sequence tensor-train cores and allows certain cores to be frozen shared across layers, thereby exploiting the inherent similarity and redundancy among layer weight matrices. To retain layer-specific flexibility, lightweight controllers introduced, enabling shared tensor cores to adaptively modulate representations. We evaluate TRAC on diverse architectures, including Qwen, LLaMA, GPT, BERT, and VIT, across benchmarks covering text classification, text generation, and image classification. Experimen- tal results demonstrate that TRAC achieves performance comparable to or better than LoRA and its variants, while substantially reducing trainable parameters and storage requirements.
INTRODUCTION
Large pre-trained models have demonstrated remarkable performance across natural language pro- cessing, computer vision, and multimodal tasks (Zhao et al. 2023 Khan et al. 2022 Yin al..
2023). However, their ever-increasing size presents significant challenger for fine-tuning and de- ployment on enoouuce-constrainee devices. To address this, parameter-effiiiet fine-tuning (PEFT) approaches (Houlsby et al. 2019; Li & Liang [02 Hu et al. 2022 Ding et al. 2023b) have been developed. Notably, low-rank adaptation (LoRA) (Hu et al. 2022) has become prominent due to its simplicity and efficiency in updating only small fraction of parameters (Mao et al. 2025).
Despite its effectiveness, LoRA leaves substantial room for further compression. Its parameter count is bounded by matrix factorization, which establishes a minimum footprint tied to the discrete rank r. As model sizes grow and service providers may need to store millions user-specific LoRA weights, this reduced parameter count becomes non-trivial (Kopiczko al.) 2024). Moreover, the dependence on rank limits the granularity of parameter configurations, restricting LoRA's applicability in extremely wwwpaarmettrr regimes (Koohpayegani al. 2024). These limitations highlight the need for approaches beyond matrix factorization.
Recent research has explored reducing LoRA's parameter count by introducing parameter freezing or sharing across layers, at either the vector or matrix level (Karimi Mahabadi et al. 2021: Kopiczko
al. 2024 Gao al. 2024 Li al. 2024 Liu al. 2024b Song al. (024). These approaches exploit the insight that LoRA matrices across different layers exhibit similarites and redundancies,
allowing further compression beyond standard low-rank adaptation. However, they remain confindd to the matrix decomposition paradigm, whose inherent structural constraints make it difficult to reach extremely small parameter budgets.
"Corresponinng Author
Code is available at httpp://ggttbb..cm/maaaa////////

---

Published as a conference paper at ICLR 2026
In parallel, tensor decomposition frameworks have recently been introduced for parameter--fffiient
fine-tuning Liu et al. [ Jie & Deng 2023 Bershatsky et al. 2024 Yang et al. 2024, Hounie
et al. 2024 Chen et a.. 2024). Compared with matrix factorization, tensor decompositions
more expressive and flexible, supporting broader range of parameter configurations while maintaining strong representational power. Yet, most existing tensor-based approaches, such LoRETTA (Yang) iet al. 2024) and QuanTA (Chen et al. 2 treat every decomposition module as an independent set of trainable tensors. By not fully exploiting cross-layer redundancies, these methods incur parameter growth that scales with model depth, thereby limiting their efficiency gains.
Motivated by these complementary gaps, we propose TRAC (Tensor- -Tensor-Train based Across-layer Compression), new fine-tuning framework that unifies the strengths of the above two directions. TRAC represents each adaptation module as compact sequence of Vensor-Trail cores (Oseledets) 2011), while strategically freezing and sharing certain cores across layers to capitalize on redundancy among layer weight matrices. To maintain layer-specifie flexibility, lightweight controllers are introduced to adaptively modulate the shared tensor cores. This design enables TRAC to achieve extremely low trainable parameter counts while preserving strong performance.
Our contributions can be summarized as follows:
We propose TRAC, a novel PEFT framework that integrates Tensor- Tensor-Train decomposition with cross-layer compression. TRAC substantially reduces trainable parameters while preserving or improving performance compared to prior methods.
2. We leverage cross-layer weight redundancy by sharing Tensor-Trae cores, enabling parameter savings beyond standard matrix-decom seerrxddccoomssttionnbeee bbssed PEFT. We validate the effectiveness of our design through controlled ablations and empirical comparisons.
3. Across diverse benchmarks, TRAC demonstrates competitive superior performance compared to LoRA and related variants while using significantly fewer parameters. TRAC achieves up 20x reduction in trainable parameters for ILaMA2-133 SuperGL.UE and up 14x reduction for ViT-Large on image classification tasks, while maintaining or improving accuracy.
2 RELATED WORK
LoRA and Its Variants. Low-Rank Adaptation (LoRA) (Hu et al. 2022) maintains a frozen pre- trained weight matrix Wo E Rmxx while expressing its update AW through low-rank factorization:
h Wax AWx Woz + BA x,,
where B E Rmxr and A Rexr are low-rank matrices with rank < min(m, n). This elegant design reduces the number of trainable parameters while preserving the frozen backbone. However, the decomposition structure imposes lower bound on trainable parameters, reached at 1, which limits its applicability in extremely parameter-constrained settings. Subsequently, a rich line of research has emerged pushing LoRA variants into more efficient regimes (Mao et al. 2025).
Baasd Fine-Turme Tuning Approaches. Beyond matrix factorization, tensor decompositions provide broader framework for parameter compression. Classical methods include Canonical Polyadic (CP) (Hitchcock] 1927). Tucker (Tucker) 1966), and Tensor-Train (TT) (Oseledets) 2011). each with different representational trade-offss (Grasedyck al. (201)). Recent works have explored these directions for fine-tuning large models. For example, LoRTA (Hounie et al.) (2024) applies CP- style low-rank tensor parametrization to further reduce LoRA parameters, while LoTR (Bershatsky et al. 2024) adopts fucker-inspired framework. However, CP faces the classic difficulties of rank selection and numerical instability, whereas Tucker requires dense core tensor whose size grows exponentially with order, which limits its scalability to high-order tensors.
Among the concurrent developments, many studies have adopted TT decomposition. By factorizing matrices into sequences of compact third-order cores, TT achieves linear parameter growth and allows explicit control of compression through TT-ranks. Representative TT-based approaches illustrate these advantages: LoRETTA (Yang al. (2024) applies TT decomposition directly to the low-rank matrices within LoRA; TT-LoRA (Anjum et al. 2024) instead represents the incremental update matrices AW in TT format; and Quan TA (Chen al. (2224) leverages wuantum-inspine TT structures for efficient high-rank adaptation. In line with these developments, our method built upon the TT framework the basis for parameter-efficien fine-tuning.
2

---

Published as a conference paper at ICLR 2026
Matrix Decompose
Example of tensor of size x x4:
f2
Reshape
P1
Decompose
P
c
AW gexx
E R
A" geaa
1/
Trainable Temso Core
Random
Trainable Controllers
Shared Trainable
Tensor Core
Tensor Core
Os S
Figure 1: Overview of our proposed method. TRAC reconstructs LoRA adaptation modules into
Tensor-Train form, represented by mixture of trainable, frozen, and shared tensor cores. Lightweight controllers are introduced to enhance the expressiveness of shared cores across layers. TRAC provides flexible control over parameter budgets by adjusting TT ranks, allowing, for example, more resources to be allocated to matrix B (Zhu et al. 2024).
Random Shared Weights Fine-Tuniin Tine-Tuning complementary research direction exploits frozen or shared structures rather than explicit tensorization. Early analyses highlighted the expressiveness of randomly initialized, frozen subnetworks (Frankle & Carbin 2019: Ramanujan et al. 2020), while
weight sharing exhibited effectiveness in compact Transformer designs (Lan et al. 20220.. Recent
PEFT methods extend these insights into LoRA-style tuning: VeRA (Kopiczko et al. (2024] shares
LoRA matrices across layers while training only scalars; NoLA (Koohpayegani et al. (2024) relies
random matrices combined via learned coefficients; FourierFT (Gao et al. (024) transforms updates
into the Fourier domain for cross-layer sharing of spectral components; VB-LoRA (Li al. 2024
trains a global pool of shared vectors to assemble low-rank adapters: AFLoRA (Liu et al. 2024b
progressively freezes LoRA matrices A and B during later training; ShareLoRA (Song al. enforces systematic cross-layer sharing of LoRA matrices; and COMPACTER (Karimi Mahabadi al., 2021) inserts task-specific adapter weights summing Kronecker products of shared weights and per-layer rank-one matrices. Collectively, these strategies demonstrate that randomization and sharing, when carefully controlled, can offer strong parameter efficiency and highlight the potential of exploiting cross-layer redundancy-a principle our method builds upon.
2024)
3 METHODOLOGY
When scaling to very large models, existing LoRA variants and tensor-based PEFT methods still impose notable parameter costs, limiting their applicability in severely resource-constrained scenarios. To address this, we propose TRAC (Tensor- -rain based Across-layer Compression). parameter- efficient fine-tuning framework that combines Tensor-Traie decomposition with across-layer sharing and freezing of selected tensor cores, enhanced by lightweight controllers (Figure [ In practice, each LoRA module replaced by a compact tensor-train structure consisting of trainable core, frozen core, and shared core modulated by the controllers, providing both efficiency and flexibility. This design exploits cross-layer redundancy to substantially reduce trainable parameters while preserving adaptability, enabling TRAC to achieve highly compact yet expressive fine-tuning modules.
3.1 MANSOR-TRAIE DECOMPOSITION AND TENSORIZED LAYER
201T) is widely employed for managing high- dimensional data. For given d-order tensor X' - the Tensor Train format expresses X as a collection of d three-order tensors GA E Th Xxm,Xra, 1.... ......d d, with rank (1, ri...., Td-1, 1, 1). Each tensor Gr is referred as the k-th lensor core. The elements of X can be represented in the following matrix -rrrduc form:
Tensor-Train (TT) decomposition (Oseledets
G Ga(ia))
where Ge (1x) R X is the i th slice of Gn along the second mode. Assuming T' = r, ny = n, then the effective dimension of X' in (2) dnr which scales linearly with tensor order, thus
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison on RoBERTa-Ban between the All Trainable baseline and our freeze-share strategy under identical TT-rank. Mean and standard deviation are reported.
# TRAIN. PARAM.
METHOD
CoLA
MRPC
RTE
STS-B
ALL TRAINABLE
411,648
64.83(2.21)
88.92(0.80
80.29(1.91)
91.21(0.19) 91.21(0.20)
OURS (HAREZE+SHARE)
26,304
64.84(2.89)
88.63(1.79)
80.14(1.80)
achieving exponential compression compared to the full ambient dimension n These properties render Tensor- Train decomposition highly efficient for representing high-order and high- nigh-dimensionll data, widely applied in quantum physics (I Lanyon et al. 2017). recommendation systems (Zhou et al. 2024), and many others (Kolda Bader 2009: et al. 2025).
We also define notation for the tensorization of matrices. For matrix W E mk with factorizations = (RI,...,7a.]. H
m = (m.,..., Md..); di my m and operator
n, we denote the reshape
W = Reshape(Ws m, m)) where the reshaped tensor W is of order (dm dn) with size (m, n)).
3.2 BASELINE: TRAINING ALL. TENSOR CORES
Motivated by the need to compactly represent the large parameter increment matrix AW in ① we employ TT format for parameter compression. Given LoRA's low-rank factorization AW BA with A E guxr, B E kmxrr, one straightforward approach is to apply TT decomposition directly on and and train all tensor cores. We refer to this approach the All Trainable baseline.
For instance, for A we consider:
A = Reshape(A, (A, (ni, n2, ns), r),
(3)
with A E Rui XnggsaxX Bingnn Following the TT representation format
A(i1, Gi (ii))a)))))
(4)
with rank (2, and cores Gi E x1xx,xrr, G2 E R Xmaxa, G3 E Brxxnaxr and all G; are trainable. An analogous decomposition is applied for B.
This strategy is similar in spirit to earlier tensorized PEFT methods such LoRETTA YYang et al., 2024) and TT-LoRA (Anjum et al. 2024).. Compared with vanilla LoRA, reduces parameters due to TT compression, yet the total parameter count still scales linearly with network depth since each layer has its own independent set of tensor cores.
3.3 LIMITATIONS OBSERVED: REDUNDANCY ACROSS LAYERS
Among the three cores in the TT representation, 91 is relatively small because it includes dimension of size 1, yielding parameter count G1 R1F1, In contrast, G2 and G3 are much larger with sizes |G2| = r172222 and Gal = 12f13,, and thus dominate the parameter budget. To reduce redundancy, our strategy keeps 91 trainable in each layer while freezing G2 and sharing G3 across layers.
Although the All Trainable baseline reduces parameters compared to LoRA, we empirically found that in deep transformers many layers exhibit redundant parameterization. especially within the large cores G2, G3- To corroborate this, we conducted controlled experiments on RoBERTa--Bae which consists of a 12-layer Transformer encoder with hidden size 768. Following common practice, we applied adaptation only to the self-attention Wo and Wy matrices, each of shape 768 x 768. Both the All Trainable baseline and our freeze-share scheme employed the same tensor shape [8, 8, 12] and TT-rank [16, 24] with LoRA rank r = 16, leading to
(Liu. 2019).
G1 ~ [1 8, 16], G2 ~ [16, 8, 24], Gy ~ [24, 12, 16).
As shown in Table even though the All Trainable baseline introduces nearly 15x more parameters than our freeze-share scheme (0.41M vs 0.026M), the performance improvement is marginal and often statistically comparable. This highlights the substantial redundancy across layers and supports design choice of freezing G2 and sharing G3 while keeping 91 layer-specific.
4

---

Published as a conference paper at ICLR 2026
3.4 TRAC: TENSOR TENSOR-TRAIN BASED ACROSS-LAYER COMPRESSION
Building upon the observations above, we propose TRAC. For each weight matrix W, we still represent LoRA matrices A, B in TT format, but assign distinct roles different cores:
91 (trainable), 92 (frozen);
G3 (shared across layers with lightweight controllers).
(5)
Concretely, within the Transformer, randomly initialized G2 G3 pair is allocated to each type of matrix (query, key, value, feed-forward), G2 is frozen, while G3 is shared across all layers of the same type. This sharing substantially reduces parameter usage while leveraging layer-wise redundancy.
To enrich the capacity of shared tensors, we introduce two lightweight vector controllers E IR"" and d E R per layer, gating the shared tensor 93 in multiplicative manner.
Ga(41, /2, la) = ((, d) (b)((.) o (d)(12) Ga(4,, (2);
where y is scaling factor (fixed trainable), and (-) is task-dependent activation (we use expo- nential for text classification, linear for text generation and image classification). These controllers act as lightweight gating signals, akin to MtE-style expert weighting (Cai et al. 2025 or DARTS-style continuous operator selection (Liu et al. 2019), thereby enriching layer-specific expressivity.
After applying the controllers, the vector-wise expression of i
A(i, 12 13. :) 91 (11)
We then take a compact notation denoted as A = 19, G2. Gs. b, dl. For matrix B E Rmxx, we similarly obtain B [O,, O2, O3, b'. d'). where O E lxm xs1 O2 E R xp.1xmax,,.0 O2 I222 x b' E =2 d' E Rmm,, and (1, 81, 82) is the rank.
At inference, TRAC, similar to LoRA, merges with pretrained weights without altering architecture.
3.5 IMPLEMENTATION DETAILS OF TRAC
Initialization. The initialization of tensor cores is crucial for model performance. For weight matrix W E Rm and its corresponding tensor core Gy E IR' Xn, xTa, we utilize an initialization formula adapted from the LoRETTA Yang et al. 2024) implementation:
Vkk
+
We term this approach Tensor-Traee normal initialization (TT Norm.), as it accounts for both tensor ranks and shape factors in the TT formulation. This initialization ensures variance stability in the reconstructed matrices, mitigating exploding or vanishing as tensor dimensions and ranks change.
Following LoRA convention (Hu et al.) 2022). initialize the last in B (i.e., O3) to zero, ensuring the initial B matrix is a zero matrix. Vector controllers are initialized based on the activation function (-) to yield identity operation at the of training. We also evaluate alternative approaches like Kaiming initialization (He et al. 2015) in our ablation studies (Appendix A.I.
Components. TRAC consists of two key components: shared tensor cores for parameter efficiency and vector controllers for layer-specific adaptability. This design balances efficiency with perfor- mance, validated by ablation studies on BERT and ViT models as well as parameter-count analysis in Appendix A.2 These results confirm that both components are essential: training all tensor cores independently drastically increases parameters but yields only marginal gains, whereas adding controllers offers clear performance improvements with negligible extra cost.
Asymmetry. In our framework, it is convenient to accommodate the asymmetric property of and B by assigning different tensor shapes and ranks for their representations. Here the asymmetry means that the importance of and B may differ during fine-tuning. Previous research (Zhu et al. 2024) suggests that matrix B is more influential and thus requires higher parameter capacity, which can be easily achieved in our approach by assigning larger tensor ranks. For example, we use ranks (71, r2) for A and (3r1, 3r2) for B across tasks, which shows better performance than using (2r1. 2r₂) for both A and B. Our detailed analysis in Appendix A.3 A.3 further confirms this finding, with experiments showing that allocating more parameters to matrix 7 consistently yields better results.
5

---

Published as a conference paper at ICLR 2026
Table 2: Orders of parameter and computational complexity for different fine-tuning methods.
FT
LORA
VERA
LORETTA
TRAC
PARAMETER ORDER
(
(
((Tm)
O(Tn1/8r9)
O(n!/3,2)
COMPUTATIONAL ORDER
O
((Lnr) O(LNrvERA)
O(Lnr)
O(Lar)
Hyperparameter Selection. Compared with standard LoRA (whose main hyperparameters are rank and learning rate), our lensor-based method introduces two additional factors: tensor order and tensor ranks. A 3rd-order tensor is the minimal configuration that accommodates trainable, frozen, and shared cores; higher orders only add frozen intermediate cores, slightly increasing cost without improving accuracy. For ranks, the trainable parameter count depends on tensor ranks, model depth, and dimensions, allowing us to set (rI, r2) under a target budget. See Appendix A.4 for details.
THEORETICAL ANALYSIS
4.1 APPROXIMATION AND SHARED STRUCTURES
We provide some theoretical analysis to understand why TRAC can achieve strong approximation ability under extremely compact parameter budgets. Our discussion is based on two key observations.
Approximation Error via TT ranks. As shown in classical results for Tensor-Train decomposi- tions (Oseledets) 2011), the approximation error of TT-SVD is controlled by the neglected singular values in intermediate unfoldings. Informally, larger TT ranks yield smaller approximation errors, and exact recovery is possible once the TT ranks exceed the true unfolding ranks. While this property well known in the general literature, we also derive in Appendix B. specialized analysis for our setting, which makes the role of the specific tensorized LoRA modules explicit.
Shared TT Structures Across Layers. Consider a family of matrices {A') 3/ with some shar- ing/redundancy, for example, where each can be factorized as A' = Qolt', 1= 1,2, 1,...... L. with common Qa and layer-specific, bounded Rt, In such case, the approximation error of jointly representing {A') } by sharing TT cores corresponding to Qo is bounded by a constant factor times the error of approximating Qo- We provide detailed formulation and derivation in Appendix [.2
Implication for TRAC. Taken together, these two results indicate that (i) approximation quality increases with TT ranks, and (ii) partially sharing TT cores across layers can well control the error for families of matrices with shared structure. Consequently, TRAC can afford to use larger tensor ranks under the same parameter budget- thanks to across-layer sharing- thus achieving smaller approximation error in practice and explaining its empirical effectiveness.
4.2 PARAMETER EFFICIENCY ANALYSIS
We analyze the parameter efficiency of TRAC in comparison with other fine- tuning methods. Consider a transformer with L layers and hidden dimension n. For LoRA, the parameter order is ((nrr); for VeRA it becomes O(Ln) due to parameter sharing; for LORETTA it is 03,n/33,2); and for TRAC it is 13,2)) by sharing tensor cores across layers. Since L and n are typically large in modern models, TRAC has an asymptotic advantage compared to VeRA and LORETTA. This parameter advantage, combined with the earlier analysis that larger TT ranks yield smaller errors, allows TRAC to use higher ranks under the same budget and achieve better approximation. Details in Appendix B.3
4.3 COMPUTATIONAL COMPLEXITY ANALYSIS
For a weight matrix W E Rax= as defined in ( we analyze the computational cost of AWa. excluding operations common to all methods, where the computational order refers to the dominant term. Full fine-tuning requires O(n²) operations, and LoRA needs O per layer. For VeRA, the effective rank P'ERA is typically very large (e.g., FVERA = in practice); thus its per-layer complexity approaches O(nrvRR))==0) (n²²) LoRETTA and TRAC can employ fast censor-train multiplications with a dominant cost of (nr) per layer. Table 2 summarizes these complexities across the entire
model with I. layers. Derivations are in Appendix B.4

---

Published as a conference paper at ICLR 2026
Table Performance comparison of fine-tuning methods on the GLUE benchmark using RoBERTa Base and Large. We report Matthews correlation for CoLA, Pearson correlation for STS-B, BB, and accuracy for other tasks. Higher values indicate better performance. Results marked with ()) are
from (Hu al. 2022).
METHOD
TRAIN.
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
AVG.
PARAM.
FT
124.69M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
BASE
LORA
0.295M
94.2 :0.1
88.0 =1.1
64.5
40.6
92.6 +0.2
82.2 #0.9
91.8 #0.1
85.6
VERA
0.043M
93.4 103
89.2
20.7
62.6 10.7
90.5
1033
77.1 #1.7
91.0
10.3
83.9
LORETTA
0.068M
20.5
87.7
=0.7
64.5
1:2.8
90.5
1o.6
73.6 22.7
89.7
10.8
83.2
TRAC
0.026M
93.7
+0.3
88.6 =1.8
64.8 +2.99
91.4
#0.4
80.1 #1.8
#0.2
85.0
FT
355.42M
96.4
90.9
68.0
94.7
86.6
92.4
88.2
LARGE
LORA
0.8M
95.8 10.3
88.1 20.8
69.2 11.3
94.7 10.2
88.8 11.5
92.5 10.1
88.2
VERA
0.061M
95.7 #0.4
88.0 =1.1
67.8 #1.6
93.9 10.2
86.8 #1.9
92.3 +0.2
87.4
TRAC
0.041M
95.4 20.4
89.7 =1.1
68.221.8
94.2 20.1
87.3 20.7
91.6 20.4
87.7
5 EXPERIMENTS
We conducted extensive experiments on models ranging from 86M to 13B parameters across diverse benchmarks, covering Natural Language Understanding (NLU), Natural Language Generation (NLG), commonsense reasoning, mathematical reasoning, code generation, and image classification. Our results demonstrate the effectiveness of TRAC across different architectures and task types, providing strong empirical support for our method. Additional comparisons and extended experiments are included Appendix C
Methods in comparison. We compare TRAC with several baselines: full fine-tuning (FT), which updates all model parameters, LoRA Hu et al. 2022), VeRA (Kopiczko et al. 2024), LoRETTA (Yang al. 2024). VB-LoRA (Li et al. 2024) and AdaLoRA (Zhang et al. 2023b). all of which were described in the previous section. All experiments were conducted using either the Hugging Face
PEFT library (Mangrulkar et al. 2022) or the open-source of the respective methods. Throughout all tables, bold and underlined values indicate best and second-best results, respectively.
5.1 NATURAL LANGUAGE UNDERSTANDING
5.1.1 GLUE BENCHMARKS
We evaluate our method on RoBERTa Base and RoBERTa Large models (Liu
2019) using the
General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2018) and compare it
with other baselines. Following prior work (Kopiezko et al. [2024] Gao et al.
2024 Li et al. 2024).
focus six GLUE datasets: SST-2 (Socher al. 2013). MRPC (Dolan & Brockett 2005).
CoLA (Warstadt et al. 2019), QNLI (Wang et al. 2018). RTE Wang et al. 2018). and STS-B (Cer
al. 2017).
Implementation Details. For each task, trainable parameters were initialized independently the test labels for GLUE are not publicly available, we report results the validation (see Appendix D.I for details). We compare our method to LoRA, VeRA and LORETTA. Following LoRA (Hu et 2022). apply all methods only the query and value projection matrices across all layers while fully fine-tuning the classification head. For all methods, we report the number of trainable parameters attributed to the fine-tuned layers, excluding the classification head, which is fully trained and remains consistent across all methods. We used a batch size of 64 across all experiments, while the number of epochs varied by task. Detailed settings for each method are provided in the Appendix E.1 Each task was run times with different random seeds, using the best epoch's performance for each run, and we report the mean value and standard deviation.
F As
Results. Table demonstrates that TRAC consistently outperforms VeRA and LoRETTA across both models while achieving comparable results to LoRA, despite reducing trainable parameters by up to 19x compared to LoRA (0.041M vs. 0.8M).
Rather than using the best weights fine-tuned the MNLI task to initialize MRPC, RTE, and STS-B tasks (Hu et al. 2022).
7

---

Published as a conference paper at ICLR 2026
Table Performance comparison on SuperGLUE tasks with LLaMA2-7B and LLaMA2-13B.
# TRAIN.
METHOD
CB
BooLQ
WSC
COPA
AVG.
PARAM.
LORA
2.10M
96.4 #1.
85.3 11.4
70.8 11.2
95.0 20.8
87.0
7B
VERA
0.279M
94.0 20.8
84.5
40.6
65.7
40.5
95.0
#0.0
84.8
LORETTA
0.315M
95.8 #1.7
83.9 40.8
67.0
#2.5
96.0
#2.4
85.7
TRAC
0.106M
95.8 #0.8
85.0 #0.6
68.3
#2.7
95.3
+0.5
86.1
LORA
3.28M
97.0 #0.8
87.0 10.6
73.4
13.0
95.7 =0.5
88.3
13B
VERA
0.430M
95.2 +1.7
85.7 +0.7
73.7 #1.2
94.7
#1.2
87.3
LORETTA
0.570M
94.0 #2.22
86.4 +0.7
73.7
43.99
96.3
=0.5
87.6
TRAC
0.166M
98.2 40.0
86.4
+1.1
74.0
95.7
88.6
Table 5: Performance of fine-tuning methods GPT-2 Medium and Large for the E2E benchmark. Higher values indicate better performance. Results marked with (2.2.3) are from previous work: (Hu
al. 2022). Zi al. 2023). "Kopiczko al. 2024). *(Li al. 2024).
TRAIN. PARAM.
METHOD
BLEU
NIST
METEOR
ROUGE-L
CIDER
FT
354.92M
68.2
8.62
46.2
71.0
2.47
ADPILI
0.37M
66.3
8.41
2.40
MEDIUM
LORA
0.35M
68.9
8.69
46.5
71.3
2.51
2
0.38M
68.2
8.58
44.1
70.7
2.35
VERA
0.098M
70.1
46.6
71.5
2.50
VB-LORA
0.076M
70.0
8.81
46.6
71.5
2.52
TRAC
0.054M
70.3
8.81
46.9
71.7
2.54
FT'
774.03M
68.5
8.78
46.0
69.9
2.45
LARGE
AD
0.88M
69.1
8.68
46.3
71.4
2.49
LORA
0.77M
70.5
8.88
46.7
72.2
2.53
VERA
0.17M
70.3
8.85
46.9
71.6
2.54
VB-LORA
0.13M
70.3
8.86
46.7
72.2
2.54
TRAC
0.099M
70.6
8.87
46.8
71.8
2.54
5.1.2 SUPERGLUE BENCHMARKS
We evaluate our method on LLaMA2-7B and LLaMA2-13B (Touvron et al. 2023) using four SuperGLUE classification tasks CB, BoolQ, WSC, and COPA (Wang et 2019).
Implementation Details. Since test labels are not publicly available in SuperGI.UE, we report results on the validation set. For large datasets, we sample both training and validation sets. See Appendix D.3 for detailed setup. Following (Malladi et al., (023). we reformulate all SuperGLUE classification tasks as language modelling problems. Both methods are trained for 10 epochs per task, with results reported from the final epoch. The LoRA matrix rank is set to 4 for both models. The tensor structure TRAC and additional byperparameters are detailed in the Appendix E.2
Results. Table demonstrates that for LLaMA2-7B, TRAC outperforms both VeRA and LoRETTA while maintaining comparable performance LoRA-despite despite using 20x fewer parameters. For LLaMA2-13B, TRAC achieves the best overall performance among all methods.
5.2 NATURAL LANGUAGE GENERATION
We evaluate method's performance generative tasks using the E2E benchmark (Novikoval ct al. 2017) with GPT-2 Medium and Large models (Radford al. 2019).
Implementation Details. Following (Hu et al. 2022). we fine-tune the query and key projection matrices for 5 epochs both LoRA and our method, reporting results from the final epoch. We adopt the hyperparameters from (Hu et al.) 2022) for LoRA implementations, and employ linear learning rate scheduler to determine the optimal learning rate for ours. Complete hyperparameter settings and tensor structure details provided in the Appendix E.3
8

---

Published as a conference paper at ICLR 2026
Table 6: Commonsense reasoning results of LoRA and TRAC on LLaMA3-8B and Qwen3-8B.
MODEL
МETHOD
TRAIN.
BQ
PQ
SQ
HS
WG
AB
AC
OB
PARAM.
LoRA
1.70M
69.39
85.31
76.00
90.77
77.27
89.44
76.37
79.00
80.44
LLAMA3-8B
TRAC
0.66M
70.52
85.69
75.90
91.29
81.45
87.84
76.28
78.40
80.92
TRAC
1.00M
71.77
85.69
77.02
91.81
81.29
88.05
76.02
79.00
81.33
LORA
1.92M
67.31
88.79
79.38
90.26
76.16
96.38
90.87
89.40
84.82
QWEN3-8B
TRAC
0.67M
66.67
87.49
79.63
92.16
80.27
96.17
90.44
88.40
85.15
TRAC
1.06M
70.95
88.74
79.99
92.37
80.98
96.34
90.61
89,20
86.15
Table 7: Results of LoRA and TRAC on GSM8K
accuracy and HumanEval pass @1 using the LLaMA3-8B model.
Accuracy (Kk
84-
LoRA
# TRAIN.
METHOD
GSM8K
HUMANEVAL.
PARAM.
200
400
600
800
1000
1200
LORA
1.70M
64.67
38.78
Trainable (x109)
TRAC
0.66M
63.31
38.29
Figure 2:
Accuracy Parameter Count of
TRAC
1.00M
64.90
38.90
LoRA and TRAC on ViT-Base (CIFAR-100).
Results. As shown in Table 5 TRAC uses only 14% and 12% of LoRA's trainable parameters on GPT-2 Medium and Large, and about half of VeRA's. Despite using nearly one order of magnitude fewer parameters than AdaLoRA, TRAC achieves higher scores. Compared with VB-LoRA-wihn reports stored rather than trainable parameters (see Appendix C.2 TRAC attains comparable or better performance with greater practical efficiency. Overall, it matches surpasses all baselines across most metrics, demonstrating strong performance under a drastically reduced parameter budget.
5.3 COMMONSENSE REASONING, MATH, AND CODE BENCHMARKS
We further evaluate our method on commonsense reasoning, mathematical reasoning, and code generation tasks with LLaMA3-8B (AI@ Meta 2024 and Qwen3-8B Team 2025) models.
Implementation Details. We follow the experimental settings DoRA (Liu et al. 2024a) and LoRA-GA (Wang et al. 2024). For commonsense reasoning, the models are trained Com- monsense 70K and evaluated on eight standard benchmarks (Hu et al. (023). For mathematical reasoning, the models trained on GSM8K (Cobbe et al. (021). For code generation, the models are trained on 100k subset of
100k subset of MetaMathQA (Yu al. (023) and evaluated
Code-Feedback (Zheng al. al.44 and evaluated on HumanEval (Chen) 2021). All experiments are conducted for 1 epoch, with performance evaluated at the end of training. For both models, the
LoRA matrix rank to while TRAC is tested under two parameter configurations. Detailed hyperparameter and configuration settings are available in the Appendix E.4
Results. Tables 7 summarize the results across the three benchmarks. Across all tasks, TRAC
achieves comparable or superior performance to LoRA while using only 35%-59% of its trainable parameters, demonstrating efficiency and robust generalization across modern models and tasks.
5.4 IMAGE CLASSIFICATION
We evaluate our method vision tasks using ViT Base and ViT Large models (Dosovitskiy al.
2020: Wu et al. 20200 pretrained on ImageNet-21k (Deng et al. 2009). The evaluation is conducted on four image classification benchmarks: CIFAR- CIFAR-1O/I00 (Alex 2009). Flowers102 (Nilsback &
Zisserman 2008), and CUB-200-2011 (Welinder et al. 2010).
Implementation Details. Following (Koohpayeganí et al. 2024 we use 10 labeled samples per class to simulate low-data regime (Appendix We compare TRAC with full fine-tuning (FT), classification head tuning (Head), LoRA, and LoRETTA. All methods update the final classification layer (excluded from parameter counts); PEFT methods also tune the query and key projections, with LoRA's rank set to 8 for both ViT-Base and ViT-Large. TRAC's tensor structure and other hyperparameters
given in Appendix E..) Models trained for 50 epochs with batch size of 16. We use four dlta-sampling and three iniiialization seeds (12 runs total) and report mean top-1 accuracy and standard deviation from the final epoch.

---

Published as a conference paper at ICLR 2026
Table 8: Accuracy of fine-tuned ViT Base and Large models on image classification benchmarks.
METHOD
# TRAIN.
CIFAR-10
CIFAR-100
FLOWERS
CUB-200-2011
AVG.
PARAM.
FULL
85.840M
87.03
76.49+1.71
99.04=0.05
82.07=0.44
86.16
BASE
HEAD
87.60=0.44
67.45 #0.52
98.60=0.04
81.20
83.71
LORA
0.295M
93.60-0-10
83.8440.27
99.38
84.25=0.24
90.27
LORETTA
0.068M
93.02 20.40
83.21 40.21
99.29 1:000
84.75 40.40
90.07
TRAC
0.026M
94.86
84.21
99.30 10007
85.04.0..4
90.85
FULL
304.088M
93. 91 =0.90
82.52 #0.91
99.25 #0.06
82. =0.25
89.64 64
LARGE
HEAD
92.86-0.25
75.71
98. 20.04
20.13
87.62
LORA
0.786M
95.87 =0.45
86,50
40.20
99.54 10.08
85.05 16.16
91.74
LORETTA
0.154M
96.24 +0.39
86.31
+0.36
99.49 +0.04
86,42 40030
92.12
TRAC
0.056M
96.81
+0.25
87.10 #0.28
99.49 +0.05
86.99 =0.35
92.60
Tralning Time Compatison: RoBERta Base
Paak CPC Mamory Usage: RoBERTa Large
Model Weight Storage Size: RoBERTa Large
Thininn Tima 6d
700
634-
497
CPC Monnon
2:793
3.701
146
2,33
Model Sooe U4111
1014
600
200
400
LORETTA
Training Comparisom: Large
Usage: HaMAA-2-13
Model Wiight Storage
2000
25
2000
12
Modee Siue (MB)
12-
7000
1332-3
2121.
3260.
1900
12
1000
1.6
3.000
LaRA
Vedx
LABETTX
TIAC
Lo03
VedA
LaBETTA
TRAC
148A
Seit
ESRITTA
TRAC
Figure 3: Comparison TRAC with main baselines on the RoBERTa and LLaMA-2- models, covering (a) training time, (b) peak GPU memory usage, and (c) model weight storage size.
Results. Table 8 demonstrates that TRAC outperforms LoRA three datasets with 11x fewer parameters for VIT Base and maintains comparable performance with 14x fewer parameters for ViT Large. Additionally, TRAC consistently surpasses LoRETTA across all configurations while using only 1/3 of its parameter count.
5.5 PARAMETER SENSITIVITY AND EFFICIENCY ANALYSIS
Parameter Sensitivity Analysis. evaluate how performance varies with parameter size, we conducted experiments the ViT-Base model by adjusting the number of trainable parameters for both TRAC and LoRA. As shown in Figure 2, performance for both methods improves steadily as pa- rameter capacity increases. However, TRAC consistently achieves higher accuracy with significantly fewer parameters, highlighting its more favorable parameter-efficiene trade-off.
Time and Memory Efficiency Evaluation. We conducted additional quantitative comparisons of training time, GPU memory usage, and storage cost relative to the main baselines. As shown in Figure 3 the additional training-time overhead of TRAC remains below 6% relative LoRA, indicating that the computational cost introduced by tensor decomposition is negligible. Meanwhile, the peak GPU memory consumption during training is almost identical across methods, and TRAC reduces model weight storage by more than 93% relative to LoRA. Overall, these results show that TRAC achieves strong parameter compression with minimal computational overhead.
CONCLUSION
In this paper, we propose novel fine-tuning method that combines parameter freezing and sharing with Tensor-Train decomposition, significantly reducing trainable parameters while retaining LoRA's advantages. Across various language and vision models and benchmarks, our method achieves comparable superior performance to LoRA and its variants while significantly reducing parameter counts. Several promising research directions emerge from this work. Theoretically, rigorous analysis is needed to understand how parameter freezing and sharing affect the representational capacity of Tensor- Train decomposition. Practically, developing efficient strategies to search for optimal configurations of tensor structures and ranks remains important direction for future research.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We confirm adherence to the ICLR Code of Ethics. This work proposes general-purposs parameter- efficient fine-tuning algorithm (TRAC) using only publicly available datasets and open-source models, involving no human subjects or personally identifiable information. While our method improves computational efficiency, we acknowledge that it facilitates model adaptation, potentially including malicious use cases. However, TRAC content-agnostic; any ethical risks (e.g., bias, toxicity) stem from the base models or downstream data. Responsibility for safe deployment lies with the practitioners. The authors declare no conflicts of interest.
REPRODUCIBILIT STATEMENT
We ensure the reproducibility of our work through detailed documentation of theoretical analysis, experimental setups, and open-source code. The derivations and discussions supporting our ap- proximation analysis and complexity comparisons (Section 4 are provided in Appendix B The experimental settings are described in Section 5 all evaluated architectures (e.g.. Qwen, LLaMA, GPT, BERT, and ViT) detailed in Appendix implementation of TRAC is available at https:t/////mm. /ttt://///tuubb// com/ BangquoYe/IRaa
with a comprehensive list of hyperparameters for
E Our
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported by the National Natural Science Foundation of China (Grant No. 12090024 and Grant No. 125B2026) and the Natural Science Foundation Chongqing, China (Grant No. CSTB2023NSCQ- LZX0054).
REFERENCES
AI@ Meta. Llama 3 model card. 2024. URL htttpp::///.ttubb.commmetttttt
blob/ma http:/maanm/M/DL.. CARD md
Krizhevsky Alex. Learning multiple layers of features from tiny images. https:///www. cs. toronto. sllubrri/heurinng-fetttse=--0
pdf. 2009.
Afia Anjum, Maksim Eren, Ismael Boureima, Boian Alexandrov, and Manish Bhattarai. Tensor train
low-rank approximation (tt-lora): Democratizing ai with accelerated 1lms. In 2024 International Conference on Machine Learning and Applications (ICMLA), pp. 583-590. IEEE, 2024.
Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, Aleksandr Mikhalev, and Ivan Oseledets. Lotr:: Low tensor rank weight adaptation. arXiv preprint 2.0Xi::2002.1376.. 2024.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions Knowledge and Data Engineering, 2025.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and cross- lingull focused evaluation. In The / Ithh International Workshop on Semantic Evaluation (SemEval-20017 pp. 1-14, 2017.
Mark Chen. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Zhuo Chen, Rumen Dangovski, Charlotte Loh, Owen Dugan, Di Luo, and Marin Soljacic. Quanta: Efficient high-rank fine-tuning of Ilms with quantum-informed tensor adaptation. Advances in Neural Information Processing Systems, 37:92210-922255 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 14168, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
11

---

Published as a conference paper at ICLR 2026
Ning Ding, Xingtai Lv, Qiaosen Wang. Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4133 1133-4145, 2023a.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-233 2023b.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (/WP2005), 2005.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (3):211-218, 1936.
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Ziqi Gao, Qichao Wang. Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter- efficient fine-tuning with discrete fourier transform. Forty-first International Conference on Machine Learning, 2024.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques. GAMM-Miteillngen, 36(1):53-78, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 1034, 2015.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, Mcasuring massive multitask language understanding. In International Conference on Learning Representationss 2021.
Frank L Hitchcock. The expression of tensor polyadic sum of products. Journal of Mathematics and Physics, 6(1-4):164-18,, 1927.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. efffmeter-efficient transfer learning for nlp. In International conference on machine learning. pp. 2790-2799. PMLR, 2019.
Ignacio Hounie, Charilaos Kanatsoulis, Arnv Tandon, and Alejandro Ribeiro. Lorta: Low rank tensor adaptation of large language models. arXiv preprint arXiv:2410.04060. 2024.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria. and Roy Lee. Lim-adapters: An adapter family for parameter-efficien fine-tuning of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 0254-5276, 2023.
Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 1060-1068, 2023.
Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in neural information processing systems, 34:1022-1035, 2021.
12

---

Published as a conference paper at ICLR 2026
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(105):1-41, 2022.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.
Soroush Abbasi Koohpayegani, Navaneet K I., Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. NOLA: Compressing loRA using linear combination of random basis. In The Twelfth International Conference on Learning Representations, 2024.
Dawid Jan Kopiczko. Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
Ben Lanyon, Christine Maier, Milan Holzäpfel, Tillmann Baumgratz, Cornelius Hempel, Petar Jurcevic, Ish Dhand, AS Buyskikh, Andrew Daley, Marcus Cramer, et al. Efficient tomography of a quantum body system. Nature Physics, 13(12):1158-1162 2017.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, 2021.
Yang Li, Shaobo Han, and Shihao Ji. VB-IoRA: Extreme parameter efficient fine-tuning with vector banks. In The eightt Annual Conference on Neural Information Processing Systems, 2024.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Zhi- Yuan Xie, Zhong- Yi Lu, and Ji-Rong Wen. Enabling lightweight fine-tuning for pre-trained language model compression based on matrix product operators. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 3388-5398, 2021.
Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang-Tieg Ting Cheng, and Min- Hung Chen. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning, pp. 32100-32121. PMLR, 2024a.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19Xiv:1907.11000 364, 2019.
Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, and Peter Beerel. Aflora: Adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 161-167, 2024b.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53077 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-effiiient fine-tuning methods. hhttps://githu/ comm/nuuggingffacc/peett 2022.
Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu. Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197005, 2025.
Leon Mirsky. Symmetric gauge functions and unitarily invariant norms. The quarterly journal of mathematics, 11(1):00-59 1960.
13

---

Published as a conference paper at ICLR 2026
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision. graphics & image processing, pp. 722-729. IEEE, 2008
Jekaterina Novikova, Ondřej Dušck, and Verena Rieser. The e2e dataset: New challenges for end-to- end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, 2017.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing. 33(5): 2295-2317, 2011.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 1-70))::6667220 2020.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari.
What's hidden a randomly weighted neural network? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11893-11902. 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Yurun Song, Junchen Zhao, Ian G Harris, and Sangeetha Abdu Jyothi. Sharelora: Parameter efficient and robust large language model fine-tuning via shared low-rank adaptation. arXiv preprint arXiv:2406. 10785, 2024.
Qwen Team. Qwen3 technical report, 2025. UBL_Ettp..//arrt..cc///////3/3
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31()): 279-311, 1966.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, pp. 353, 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. Advances in Neural Information Processing Systems, 17:54905-54993 2024.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641. 2019.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and
Pietro Perona. Caltech-uesd birds 200. 2010.
Bichen Wu. Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
Yifan Yang. Jiajun Zhou, Ngai Wong, and Zheng Zhang. Loretta: Low-rank economic tensor-train adaptation for ultra-low wor-meeaeeter fine-tur fine-tuning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3161-3176, 2024.
14

---

Published as a conference paper at ICLR 2026
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. arXiv preprint 1xXiv:2306.135499 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint wrXiv:2308.0330.3
2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiiiet fine-tuning. The Eleventh International Conference Learning Representations, 2023b.
Yuanwei Zhang, Fengmiao Bian, Xiaoqun Zhang, and Jian-Feng Cai. Preconditioned riemannian gra- dient descent algorithm for low-mulillnnca--rn rank tensor completion. In Focto- International Conference on Machine Learning. 2025.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreeer:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
Jie Zhou, Botao Hao, Zheng Wen, Jingfei Zhang, and Will Wei Sun. Stochastic low-rank tensor bandits for dialiiiammenion online decision making. Journal of the American Statistical Association, pp. 1-14, 2024.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In Forty-first International Conference on Machine Learning, 2024.
Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang, Delta-lora: Fine- tuning high-rank parameters with the delta low-rank matrices. arXiv preprint (xXx:2230.002111 2023.
15

---

Published as a conference paper at ICLR 2026
A SUPPLEMENTARY DETAILS OF TRAC
Note: For all ablation experiments in this section, we fix the lightweight vector controller's activation to linear function (x) = in order to control for the effect of activation choices. This setting differs slightly from the main experimental section, where task deeeddeenddent activations are used.
A.1 INITIALIZATION STRATEGIES
Tensor- Train Uniform Initialization. Based on the parameter defined in 7 we can consider another option that uses the uniform distribution within the range [-ok. Tkl, and we call this approach the Tensor-Train uniform initialization.
We use two types of Kaiming initialization: normal distribution and uniform distribution.
Kaiming Normal Initialization (He et al., 2015). Parameters are initialized following normal distribution (00, o.), where the variance of defined
2 Tlin.
0
(8)
where nin denotes the dimension of the input features. In LoRA, we apply variances of and 2 for AT E x and B E mmxx, respectively.
Kaiming Uniform Initialization (He et al. 2015). For uniform distribution, the initialization ranges are 11 v and [-V
2 for A Rg and R2 respectively.
We evaluate the impact of different initialization approaches on TRAC performance. Four initializa- tion strategies were tested:
TT Norm.: Tensor- Train normal initialization using the formula defined in Equation 7 which accounts for tensor ranks and shape factors.
TT Unif.: A uniform variant of Tensor- Train initialization, where values are sampled from uniform distribution scaled by factors similar to those in TT Norm.
Kaiming Norm.: Standard Kaiming normal initialization (He et al. 2015 commonly used in deep neural networks.
Kaiming uniform initialization (He et al. 2015 providing uniform distribution alternative to Kaiming normal.
Kaiming Unif.:
As demonstrated in Table 9 the Tensor-Train normal initialization consistently outperforms other strategies across all three evaluation tasks. This confirms the importance of properly accounting for tensor structure during initialization ensure stable training and optimal performance in tensor-based adaptation methods.
Table Ablation experiments comparing different initialization strategies on MRPC, CoLA, and RTE datasets. The corresponding model is RoBERTa Base.
INIT. STRATEGY
MRPC
CoLA
RTE
TT NORM.
89.0 20.7
65.2 +2.4
79.9 =3.6
TT UNIF.
87.7 #1.1
62.4
+1.5
77.7 =2.2
KAINMING NORM.
87.9 21.3
63.8
+2.6
79.6
=2.0
KAINMING UNIE.
87.8 40.5
63.3
12.8
79.9
16

---

Published as a conference paper at ICLR 2026
Table 10: Ablation experiments comparing TRAC with its variants on CoLA and CIFAR-100 datasets. The corresponding models are RoBERTa Base and VIT Base.
# TRAIN. MODULE
# TRAIN. PARAM.
CoLA
CIFAR-100
TRAC (OURS)
0.026M
65.2
84.21
SHARED IT FROZEN
0.025M
61.63 (L 5.4 %)
84.14 0.1%)
0.008M
38.98 (1
%)
67.47 ( 19 %)
ALL TRAINABLE
0.41M
66.17 (1 1.6 %)
84.44 (1 0.3%)
A.2 COMPONENT AND PARAMETER ANALYSIS
We systematically investigate the impact of tensor and vector components in our method, and further analyze the parameter efficiency of different designs.
Component Study. We compare four configurations:
TRAC (Ours): The complete TRAC method as described in Section 3.4
Shared TT: Removal all vector controllers, keeping only shared tensor cores.
Frozen: Replacing trainable tensors with frozen tensors assess the impact of adaptability.
All Trainable:
Converting all frozen and shared tensors independent trainable parameters.
As shown in Table 10 both shared tensor cores and vector controllers make substantial contributions model performance. Removing controllers (Shared TT) using frozen tensors (Frozen) leads to notable performance degradation. While training all tensor cores independently (All Trainable) provides marginal performance gains, requires drastically more parameters, making it significantly less parameter-efficientt. These results validate our design decisions, confirming that TRAC achieves an effective balance between performance and parameter efficiency.
Parameter Efficiency Analysis. We now provide detailed analysis of the trainable parameter counts for the three approximation methods (All Trainable, Shared TT, and TRAC), complementing the performance study above.
All Trainable: For each adaptation matrix A E R*xT, this method uses three independent trainable tensors:
E R Xxn1xr,
c Rraxa, (()) R
The parameter count for each component is:
1x X f171 parameters
X 12/1 parameters
12 2a X r 1'322" parameters
For all L matrices, the total number of trainable parameters is:
L(nir 727172 + n2f2r).
Shared TT: This method utilizes:
L independent tensors (((() J each with niri parameters
One frozen tensor G2 Rn (krx()r:) (initialized randomly but kept fixed, hence not counted trainable)
One shared trainable tensor G3 ((r2))xnxx, contributing knyr2? parameters
The total number of trainable parameters is:
L RAFi kngr2r Lnyri kngrar.
17

---

Published as a conference paper at ICLR 2026
Trainable parameter count versus TT-rank for All Trainable, Shared TT, and Shared TT with Controllers (TRAC)
All Trainable
Shared TT
Parameters eetee (K)
Shared TT with Controllers (TRAC)
Trainable Parametee
30
20
10
10
11
13
16
TT-Rank (r1
Figure 4: Trainable parameter count versus TT-rank for All Trainable, Shared TT, and Shared TT with Controllers (TRAC) (Set L 12, 8, 12 ns 12, k 3).
Note that the parameter saving comes primarily from sharing G3 across all matrices and using frozen G2-
Shared TT with Controllers (TRAC): This method enhances the Shared TT by using:
L tensors (6(")] L n17 parameters
One shared tensor G3: knyr2r parameters
L controller vectors (b() M L x kr2 parameters
L controller vectors (d) | L x nı parameters
The total number of trainable parameters is:
Lnirı + kn + Lkry + Lng = L(nir1 + kr + ns) + kngrer.
While this method adds more parameters compared to the Shared TT approach, the additional controllers provide layer-specific customization of the shared components, significantly enhancing expressivity with only modest increase in parameter count. Specifically, the controllers add only L(kr + 713) parameters, which is typically much smaller than the L x 722172 parameters saved by freezing G2 and the - x nar2r parameters saved by sharing Gs. thereby confirming that the additional cost of controllers is negligible compared to the savings.
As illustrated in Figure 4 even though the TT-rank is enlarged by a factor of in shared methods, both Shared TT and TRAC require far fewer trainable parameters than the All Trainable TT approach across rank settings. TRAC particular achieves this efficiency while offering additional flexibility through controllers, making it highly effective and scalable solution.
A.3 IMPACT OF PARAMETER ALLLOCATION ASYMMETRY
We present additional experiments that demonstrate how asymmetric parameter allocation between matrices A and B affects model performance.
Previous work, such as LoRA-FA (Zhang et al. 2023a), indicated that the B matrix plays a more critical role than the A matrix in LoRA adaptation. While LoRA-FA completely freezes the A matrix, our tensor- based approach enables more fine-grained control over parameter allocation between matrices.
Symmetry Analysis Tensor Structure. Across different models and tasks, matrices A and B in LoRA may exhibit varying levels of importance, which is reflected in the tensor ranks used in
18

---

Published as a conference paper at ICLR 2026
Performance vs. Parameter Asymmetry
Performance vs. Parumeter Asymmetry
68
84.5
Matthew's is Correlation
84.0
03.5
Acruracy
62
83.00
82.3
58
81.5
56
81.0
1:7
2:6
3:5
4-
1:7
2:6
3:5
4.1
7:1
Parameter Allocation Ratio (#A:#B)
Parameter Allocation Ratio (#A:#B)
(a) RoBERTa-Base CoLA
(b) ViT-Base on CIFAR-100
Figure 5: Performance comparison with different parameter allocation ratios between A and B matrices. Both plots demonstrate that asymmetric configurations favoring more parameters in the B matrix (ratios 2:6 and 3:5) yield better performance than symmetric (4:4) or A-heavy allocations.
approach. For asymmetric structures, we set tensor ranks (r1, for matrix A, while larger tensor ranks 3r²)) are allocated to matrix B, resulting in 3x parameter count for B compared to A. For symmetric structures, we use tensor ranks (2r1, 2r2) for both matrices. Table 11 11 shows that for various models and tasks, asymmetric tensor structures consistently perform better.
Table 11: Performance comparison of symmetric versus asymmetric tensor structures across CoLA (Matthew's ' correlation), CIFAR-100 (accuracy), and E2E (BLEU). The corresponding models RoBERTa-baa base, ViT-base and GPT2-base.
TASK
CoLA
CTFAR-100
E2E
S SYMMETRY ASYMMETRY
62.6 #2.3
84.10 20.18
69.4
65.2 65.2+2.4
84.21 #0.40
70.3
To further investigate the optimal degree of asymmetry, we conducted more fine-grained experiments on two representative tasks: RoBERTa-Bael Base CoLA and ViT-Base CIFAR-100. In each experi- ment, we maintained constant total number of trainable parameters while varying the allocation ratio between matrices A and B. Figure visualizes these results.
The experiments reveal a clear pattern: asymmetric configurations that allocate more parameters to the B matrix and fewer to the A matrix (e.g., ratios 2:6 and 3:5) consistently outperform both symmetric allocations (4:4) and configurations that favor the matrix (e.g., 6:2, 7:1). Specifically, the optimal ratio appears to be 2:6 for RoBERTa-Bars on CoLA (65.2 Matthew's correlation) and 3:5 for ViT-Base on CIFAR-106 100 (84.22% accuracy).
In contrast, when more parameters allocated the A matrix (ratios 6:2 and 7:1), observe noticeable performance degradation. This empirical evidence confirms that:
1. The B matrix requires more trainable parameters than the A matrix, consistent with previous research suggesting its greater importance LoRA-based adaptation (Zhu et al. 2024).
2. Our approach's 's ability to flexibly distribute parameters between matrices enables more effective exploitation of this inherent asymmetry.
Architectural flexibility provides tangible performance gains by allowing practitioners optimize parameter allocation based on the specific characteristics of different models and tasks.
These findings validate claims about the benefits of parameter granularity and architectural flexibility our proposed method, demonstrating concrete performance improvements through tailored parameter allocation. The results consistent across multiple models (RoBERTa, ViT,
19

---

Published as a conference paper at ICLR 2026
GPT2) and tasks (CoLA, CIFAR-100, E2E), indicating that the asymmetric nature of LoRA adaptation a general phenomenon that our tensor- based approach effectively leverages.
A.4 HYPERPARAMETER SENSITIVITY AND RECOMMENDED CHOICES
We provide additional details on hyperparameter selection, which were also clarified during the rebuttal stage.
Tensor order. 3rd-order tensor suffices capture the design of trainable, frozen, and shared cores. If the tensor order is increased to >> 3, then in TT decomposition most middle cores have shape with parameter count Or²)). To maintain total budget of Or)),
r X n, x r for ,,..., these middle cores must remain frozen, leaving little benefit while increasing computational cost. We compare 3rd-, 4th- and 5th-order settings under the same trainable budget in Table 12 confirming that higher orders degrade efficiency without improving performance.
Table 12: Comparison of different tensor orders under the same trainable parameter budget. Perfor- mance is reported on the RoBERTa Base setting.
Tensor Order
Train. Param.
CoLA
MRPC
RTE
STS-B
3RD-ORDER
26,304
64.8
88.6-1.8
80.1
91.2=a02
4TH-ORDER
26.784
64.5
87.5
75.3
90.8
-RDDER
28,560
63.7+1.5
88.6-u.6
76.4
90.7
Training efficiency results are provided in Table 13 training time, leading to slightly reduced efficiency.
shown, higher-order tensors require longer
Table 13: Training time per epoch with different tensor orders.
Tensor Order
3rd-order
4th-order
5th-order
TRAINING TIME /
776.3
812.5
873.7
Tensor ranks. The number of trainable parameters can be written
#p = L(nir1 + kr ns) + kngP2?,
where I. is the number of layers, the LoRA rank, and A a scaling factor related to controller vectors. This allows us directly solve for (r1,72) given target budget #p. In practice, setting ri r2 allows us to directly solve for their values under given parameter budget, providing convenient starting point. Subsequently, the asymmetry property (cf. Section 3.5) can further reallocate ranks between matrices A and B, typically assigning larger ranks to B when beneficial.
In conclusion, these results confirm that (i) increasing tensor order beyond 3rd is unnecessary, and (ii) tensor ranks can be systematically set according parameter budgets, rather than arbitrary tuning.
20

---

Published as a conference paper at ICLR 2026
B THEORETICAL ANALYSIS AND DETAILS
B.I DERIVATION OF THE ERROR BOUND FOR TT-SVD
In this section, we present the detailed derivation of the observations discussed in Section 4.1
First, we analyze the approximation emro under given TT ranks. Note that while the general error analysis of TT-SVD is established in (Oseledets) 2011), we specialize the analysis here to the exact tensorized structure arising in our framework. This concrete instance explicates the roles of the particular tensor shapes and unfoldings involved in TRAC.
Setup Recall, We briefty recall the relevant structure introduced Section A I "xx is reshaped into fourth-order tensor
3
The weight matrix
A = Reshape (A, (n1, n2, na), E X
11 12g2g
Its TT approximation A follows the tensor-train format
A(t1, 82, 23, (11) G2((2) Gu(is),
with TT-rank (1, r1, 12) and cores 9i E 18 G2 E Rrxnaara, 93 E R The derivation below characterizes the error ||A - Áll when these cores are obtained via TT-SVD.
Specialized TT-SVD Error Analysis. Let A be the TT approximation of A E B2 produced by TT-SVD with intermediate ranks rı and
described above. Let A(1) E R (nanar) denote the first-mode unfolding of A, and let A(2) ((rrna)) (nar) denote the reshaped coefficient matrix arising after the first truncation step: their precise definitions are given in Steps 1-2 below (note in particular that A(2) depends r.). The approximation error satisfies:
min(n1. Ronsr)
min(ring, (ar)
IA ÁLL
Σ kmr++1
(A
Σ
Th (A(22))
1=
where (+)-) denotes the k-th largest singular value. Consequently, exact recovery is achievable when ri 2 ran)(A(1)) and r2 2 rank(A(2)).
Derivation Details. The result is derived by two successive SVD truncations on the unfoldings A(1) and A(2). invoking the classical Eckart-Young Young-Mirsky theorem. For completeness, we provide the step-by-step procedure below.
Lemma B.1 (Eckart- Young Mirsky). Let M RPxq with singular values 01 (M) 2 62(M) 2
O. For any 0) < S min(p, q). let (s) denote the rank-s approximation obtained by retaining the s largest singular values. Then
min(p.g)
| (
Σ
ox(M)²
This lemma asserts the optimality of SVD the Frobenius norm (Eckart & Young 1936 Mirsky 1960: Golub & Van Loan 2013).
Step 1: First-Mode Unfolding and Truncation. Form the first-mode unfolding
Aunli (i2 1)nar + (is 1)r + )] 12,
giving A() IR (namar) with full SVD A() = V.E.V. Let Ü, E Ruixr, Σn RI and V. E (maass)) denote the leading 71 SVD factors. Define the rank-r1 approximation and the residual coefficient matrix:
(m) - = VT. C -= E x
Set the first TT-core (1(1ú), -i, kn). By the Eckart- Moooung-Mirsyy lemma,
min(n1, R2ngr)
AAa) 11
Σ
(A
21

---

Published as a conference paper at ICLR 2026
Denote by Ai the intermediate tensor whose first-mode unfolding equals A0) U1; then
||-A A1 equals the right-hand side of (
Step 2: Second-Mode Unfolding and Truncation. Reshape C₁ (rix((2nar) into A(a) (((ar) via
Ak2)((1 1) i2, (is 1)r j] Ci (i2 1)nar + (ig 1)r + 5
This operation is bijective re-indexing of entries, so the Frobenius norm is preserved: (|A ||C ll.. Perform the SVD A(2) = V2 with leading 72 factors U2 E (( Eg E R2
V/2 E ((arr)xr,, Define:
A/3 = U2 V2
C2 = L2=22 E (r2x((ar),
Set the second TT-core G2(N1, k2) Ü2(((1 1) + i2, 12), and let C1 E X be the matrix obtained from A by the inverse of the above re-indexing, Since the reshape is an isometry and by the Sekart-Young-Minng lemma,
min(rrns, nar)
|| - 11112 AA(2)
1
Σ
(A(2)))
(10)
k=
Step 3: Third Core. Reshape C2 into the third TT-core: Gs 13 = C2lk2. (is 1)r + jl.
giving G3 E Rraxnaxr, No further truncation is applied, no additional error is introduced.
Combining the Two Error Terms. Let Ej = A - Ai and E2 Ai A, so that A - - = F₁ Ez. Their first-mode unfoldings are
unfold;(E)) A(1) (
(n)
Σ kDT
oR (o((A())
unfold, (F2) = U, (C1 - CT).
where ut and e2 are the k-tth left and right singular vectors of A(1)- The column space of unfold, (Fr) span (onn(u [ux
ri}, while the column space of unfold, (F2) is contained in range(U;) span(us ux k < r1). These subspaces are mutually orthogonal by the orthonormality of the left singular vectors of A(1). so
(E1, Eg) = (unfold,(E) (Ez), unfold, (Ez)) F = 0.
By the Pythagorean theorem for the Frobenius norm:
||A - Ál? = || + |
(11)
For E211 note that left-multiplication by matrix with orthonormal columns is isometry,
| - - C0 = ||Cr C.
Substituting
and 100 into yields:
min(as, Rgnsr)
min(rin2, mgr)) o% (A(2))))
LA 2
Σ
(A(A
Σ
k=i++
k=
This shows that the total approximation error equals exactly the sum of the squared tail singular values the two successive unfoldings. Exact recovery follows when both sums vanish, which holds if and only if ri 2 ra))(A(r) and ra 2 ra))
22

---

Published as a conference paper at ICLR 2026
B.2 ANALYSIS ERROR BOUND FOR SHARED STRUCTURES
We now extend the analysis to Vensor-Train structures shared across layers within a specific family of matrices.
Shared-core approximation for structured matrix families. Consider a family of matrices {A R Roxe, Assume that each A' can be factored as
A' = Qo R', 1=1,...,44 L,
where Qo E R is shared across layers and each R' R= is bounded. Let eQo be the TT approximation error of Qo. and {(A)) be the collective approximation error of {A'} under a partially shared TT representation. We observe that there exists constant M depending only on (R*) such that
E(A') S M EQU
Derivation. Reshape Qo E RuxT into a tensor Q. E IR Xaxxx2aXaxxr with R1 = n. Suppose Qo admits TT approximation
20) = 91 (212 Ca((a)))
For each layer write
A 211, (3)
Σ
2o R (k)
Define an adjusted third core
93 (1s,
Galis, k) R (k.
Then
13, i2
~ (o1)) T3 (ia.
Therefore,
Σ
A' A'
1=1
Σ
1QoRt - (Q6))²11
1=1
Σ [1r2(12) 11Qo -
Thus, the error is controlled by M EQu with M = Σt ||122112.
This generalized result shows that whenever family of matrices admits a shared component Qo multiplied by bounded layer-speeific factors R'. one can safely reuse the first two TT cores across all layers, while only adjusting the third core cheaply. This idea directly corresponds to the lightweight vector controllers in TRAC: by employing multiple such controllers, our method flexibly adapts the shared third core to better capture the characteristics of each individual layer.
B.3 DERIVATION OF PARAMETER ORDERS
For completeness, we provide the detailed setup and derivations of the parameter complexity orders in Table 2 We consider transformer with layers and hidden dimension m n. Each weight matrix W m is tensorized, with factors 1/3 and m; m for i 2, We focus on the query and value matrices, though the analysis generalizes to other components.
LoRA. The rank-r factorization adds ((r) parameters per layer, yielding total of O
23

---

Published as a conference paper at ICLR 2026
VeRA. Following (Kopiczko al. 2024), VeRA shares single pair of frozen random matrices across all layers and only updates Mayer-specific scaling vectors. Consequently, the number of trainable parameters is governed by L(n r). Since typically > VeRA's parameter complexity is dominated by O(Ln).
LORETTA, Applying TT with ranks (ri, r2) ~ r yields / parameters per layer, hence O(Ln overall.
TRAC. By sharing tensor cores across layers, TRAC requires only 0(12/3/2) parameters in total, not scaling with L.
B.4 DERIVATION OF COMPUTATIONAL ORDER
This section provides the detailed derivations of the computational complexity in Table Throughout, the input dimensionality per layer is denoted by n. the low-rank (or tensor-train) dimension by and the model contains L layers. We focus on the additional computations introduced by the PEFT modules, excluding identical forward operations shared with the pretrained weights.
LoRA. The LoRA update is expressed as AW = B.,, where A d and B Raxr, During the forward pass, y = Wox + a (Ax). Computing Axr costs (nr)., and subsequently B(Ax) costs another (nr). Including linear transformations and activation scaling yields a total of approximately O(4nr) operations per layer, i.e., an overall complexity of ([nr)) for the entire model.
VeRA. VeRA approximates LoRA adapters by learning a set of shared vector representations and layer-specific scaling coefficients. Although the formula resembles LoRA's BAx, its effective rank TVeRA is several orders higher (e.g., 1VeRA=1024 for BERT-Base and GPT-2). Consequently, TVeRA R n, giving a per-layer complexity O(NrVeRA) R= (n²) and thus (LLnrvRA) ss O(L??) for the entire model.
LoRETTA and TRAC. Both methods apply tensor-train (TT) decomposition to represent AW as a product of smaller TT-cores, enabling structured low-rank operations. Letting each TT-rank be the forward multiplication y = AWr x can be carried out using a fast TT-matrix-vector multiplication algorithm. Following this, the per-layer complexity is approximately O(nr + 2/3,22 + n 133,2) so O(r)). Across L layers, the total computational complexity is ((nr), comparable to that of LoRA.
Fast -eeetor multiplication. For the computation of AWa, we present the following efficient procedure for our method:
Perform vector controller operations to construct Ga X and Og E Rexxmaxr, requiring computational complexities ((nrrr)) and (mm32r)), respectively.
2. Reshape the vector E I= into three-order tensor X E R Xmgxnn
Contract X with G1 along the n, dimension, resulting in a tensor Xi E Rrixnn X with com- plexity O (127gg)))))
4. Contract Xi with G2 along the dimensions n2 and ri, yielding tensor Xs E Raaxx2, This procedure requires complexity O(n2nar172).
5. Contract X with G3 along the dimensions ny and r2, resulting in vector y R with complexity O(ny22r).
6. Similarly, sequentially contract the vector y with tensors O3, O2, and Oi, with corresponding computational complexities of O(m382r), ((m3m12825)) ), and Ommgmmm 81 respectively.
Under the assumptions that n, m, n n; = nt, and 8; = for 1,2, the overall computational complexity of our method for computing AWr is ((r)).
24

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTS
In this section, provide additional experiments that further compare method with recent approaches. Beyond the main experiments, we place particular emphasis on tensor-based approaches such as LoTR (Bershatsky et al. 2024). parameter- parameter-sharing sharing methods such VB-LoRA (Li et 2024), and adapter-based techniques such as COMPACTER (Karimi Mahabadi et al. 2021). which inserts task-specific adapter weights composed sums of Kronecker products between shared weights and per-layer rank-one matrices. Furthermore, we extend our evaluation to additional benchmarks such as
MMLU (Hendrycks et 2021 provide broader view of TRAC's generalization and efficiency.
C.1
COMPARISON WITH LOTR
To better situate TRAC among tensor-based PEFT variants, we compare against LoTR (Bershatsky 2024) on the GLUE benchmark (Wang al. 2018) using RoBERTa-Base and RoBERTa- Large Luu 2019), where LoTR employs Tucker decomposition. This is representative alternative tensorization strategy distinct from our use of the Tensor Train. As shown in Table 14] TRAC consistently matches or outperforms LoTR on both RoBERTa-Bal Base and RoBERTa-Laue while requiring fewer trainable parameters. These results highlight the distinctive advantage of the Tensor Train decomposition adopted in our framework: the ability to flexibly adjust parameter counts while maintaining strong accuracy.
settings,
Table 14: Comparison between TRAC and LoTR on the GLUE benchmark using RoBERTa Base and Large. We report Matthews correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. Higher values indicate better performance. Results marked with (+) are from (Bershatsky
ct al. 2024).
METHOD
# TRAIN. PARAM.
SST-2
MRPC
CoLA
QNLI
RTB
STS-B
AvG.
BASE
LOTR! TRAC
0.026M
0.32M
93.3
93.7
42.0
88.0 #9.0
88.6
61.3 #6.0
64.8
91.4
92.0 +4.0
67.0 413.0
80.1
91.2 10.2
91.0 #1.0
85.0
82.1
LARGE
LoTR!
0.33M
#1.0 95.4
89.0 +5.0
61.3 +o.0
94.8 $1.0
84.0 #2.0
91.6 11.0
86.1
TRAC
0.041M
89.7 11.1
68.2
94.2
87.3 20.7
91.6 20.4
87.7
C.2 COMPARISON WITH VB-LORA
VB-LoRA (Li et al. 2024 is a recent variant designed around a "vector bank" mechanism combined with low-rank adapters. Unlike most PEFT approaches, VB-LoRA reports the number of stored parameters that remain after training, whereas our method reports the number of trainable parameters that actually require gradient updates during training. For this reason, present detailed, separate discussion of comparisons with VB-LORA.
On text generation tasks with GPT2 (Radford al. 2019) the E2E benchmark (Novikova et al.) 2017), Table shows that TRAC achieves comparable or superior results across BLEU, METEOR, ROUGE-L, and CIDEr, while using fewer trainable parameters.
Table 15: Comparison between TRAC and VB-LoRA on text generation tasks with GPT2 Medium and Large. Higher values indicate better performance. Results marked with (?) are from (i al. 2024
PARAMETERS (TRAIN./STORED)
METHOD
BLEU
NIST
METEOR
ROUGE-L
CIDER
VB-LORA TRAC
0.076M (STORED)
70.0
8.81
46.6
71.5
2.52
MEDIUM
0.054M (TRAINED)
70.3
8.81
46.9
71.7
2.54
VR-LORA
0.13M (STORED)
70.3
46.7
72.2
2.54
LARGE
TRAC
0.099M (TRAINED)
70.6
8.87
46.8
71.8
2.54
25

---

Published as a conference paper at ICLR 2026
On GLUE (Wang al. 2018 with RoBERTa- Base Liu, 2019), ATM-LORA performs par with slightly ahead of Vo-LoRA across most tasks (Table 16). To ensure faimesss, in these comparisons aligned the number of training epochs with the settings reported for VB-LoRA (c.g.. 160 epochs for RTE), which differs from the 20-epoch configuration used in our main experiments. Under these aligned settings, ATM-LoRA achieves comparable or better accuracy while maintaining significantly fewer trainable parameters.
Table 16:
Comparison of TRAC and VB-LoRA (qv) on GLUE tasks using RoBERTa-Baae We report Matthews correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. Higher values indicate better performance. Results marked with (+) are from (Li et al. 2024).
METHOD
MRPC
CoLA
RTE
STS-B
AvG.
VB-LORA
89.5
63.3
82.3
90.8
81.5
BASE
TRAC
89.210.8
65.2.2.1
82.7
91.0=0.2
82.0
Finally, Table 17 dissects the difference between trainable and stored parameter counts. For VB- LoRA, the number of trainable parameters during optimization is much larger than the number of stored parameters reported for inference. Consequently, although the stored parameter count of VB-LoRA appears smaller, the training-time parameter burden is considerably heavier than that of TRAC. In contrast, TR maintains a low trainable parameter count that directly translates to reduced training cost, while achieving nearly identical stored parameter size to VB-LoRA.
Table 17: Comparison of trainable and stored parameter counts for LoRA using RoBERTa-Bass Base.
METHOD
TRAINABLE PARAMETERS
STORED PARAMETERS
VB-LORA TRAC
74,880 26,304
24,768
BASE
26,304
C.3 COMPARISON WITH COMPACTER
To broaden our comparison with parameter-efficient fine-tuning (PEFT) methods, we evaluate TRAC against COMPACTER (Karimi Mahabadi al.) 202T), an adapter-based approach that inserts task- specific adapter weights represented as sums of Kronecker products between shared and rank-one matrices. Unlike LoRA Abased methods such as TRAC, COMPACTER adds nonlinear adapter modules active during inference, introducing small but persistent overhead.
We use the official COMPACTER configuration with T5-Base (Raffel et al. 2020 on the GLUE benchmark (Wang et al. 2018), matching its batch size, learning rate, optimizer, and training epochs for fair comparison. Both methods are trained under identical settings using the released code of COMPACTER.
Table 18 shows that TRAC achieves similar or better performance across most GLUE tasks while using only about 33% of COMPACTER's trainable parameters. On average, TRAC improves scores by +00.66 with just 34K parameters (0.015% of model size), demonstrating its higher degree of parameter efficiency and compression.
These results highlight that TRAC complements adapter- based techniques by extending LoRA-style efficiency through tensorized parameter sharing without modifying the model architecture.
C.4 EVALUATION ON MMLU BENCHMARKS
To broaden the evaluation scope, we additionally tested TRAC on the MMLU benchmark (Hendrycks et al. 2021). All methods were first fine-tuned on Alpaca and then evaluated on MML subjects. Representative results selected MMLU domains shown in Table 19 On LLaMA2-7B (Touvron) ct al. 2023). TRAC achieves a higher overall average accuracy, with clear improvements also observed in representative subjects such as chemistry, statisties, and psychology. These experiments
26

---

Published as a conference paper at ICLR 2026
Table 18: Comparison between COMPACTER and TRAC on GLUE using T5-Base. We report Matthews correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. Higher
values indicate better performance. Results marked with (!) are from (Karimi Mahabadi al. 2021
METHOD
# TRAIN. PARAM.
RATIO
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
AVG.
COMPACTER
104,704
0.047%
93.81
90.69
61.27
93.08
74.82
90.46
84.02
COMPACTER
104,704
0.047%
94.04
89.71
58.56
93.30
82.01
90.33
84.66
TRAC
34,176
0.015%
94.61
90.69
59.85
93.10
82.01
91.06
85.22
confirm that TRAC remains effective in knowledge-heavy and reasoning-focused tasks, while training substantially fewer parameters compared to vanilla LoRA.
Table 19: Comparison between TRAC and LoRA on selected MMLU subjects and average accuracy using LLaMA2-7B. Higher values indicate better performance.
TRAIN. PARAM.
METHOD
HS CHEMISTRY
HS STATISTICS
HS PSYCHOLOGY
MMLU Acc.
LORA
2.10M
30.54
27.31 38.89
59.82 64.22
45.17
7B
TRAC
0.106.M
34.48
47.38
27

---

Published as a conference paper at ICLR 2026
D DATASETS
D.1 GLUE BENCHMARK
Our experiments on RoBERTa Base and RoBERTa Large (Liu 2019) utilize six datasets from the GLUE benchmark WWag et al. 2018): SST2, MRPC, CoLA, QNLI, RTE, and STS-B. Due to the unavailability of test labels, we conduct evaluations using the validation set. For datasets containing fewer than 10,000 training examples, the validation set is equally divided into two portions: one for model selection during training, and the other for final evaluation. For larger datasets (more than 10,000 training examples), we randomly sample 1,000 examples from the training set for validation and employ the complete validation set for testing. This splitting strategy adapted from the SoRA codebase (Ding et al. 2023a). Table 20 presents the detailed statisties of these datasets.
Table 20: Task descriptions and statistics of the datasets used in GLUE benchmarks.
DATASET
TRAINING
VALID
TEST
TASK TYPE
#LABELS
METRIC
SST-2
67,349
872
1,821
SENTIMENT
AcC.
MRPC
3,668
1,725
PARAPHRASE
AcC.
COLA
8,551
1,043
1,063
ACCEPTABILITY NLI
MCC
QNLI
104.743
5,463
5,463
ACC.
RTE
2,490
277
3,000
NLI
ACC.
STS-B
5,749
1,500
1,379
SIMILARITY
1 (REGRESSION)
PEARSON
D.2 E2E BENCHMARK
For the GPT2 (Rafford et al. 2019) experiments, we utilize the E2E dataset (Novikova et al. 2017
benchmark for natural language generation tasks. The models (GPT2 Medium and GPT2 Large) are trained on the complete E2E training set and evaluated on the full test set. Table presents the statistical characteristics of the E2E dataset.
Table 21: Task descriptions and dataset statistics of the E2E benchmark.
DATASET
TRAINING
VALID
TEST
METRICS
NLG
42,061
4,672
4,693
BLEU, NIST, METEOR, ROUGE-L, CIDER
D.3 SUPERGLUE BENCHMARK
The experiments with LLaMA2-7B and LLaMA2-13B (Touvron et al. 2023 are conducted on four datasets from the SuperGLUE benchmark (Wang al. 2019): CB, BoolQ, WSC, and COPA. Due to the unavailability of test set labels, we perform evaluations using the validation set. For BoolQ, we randomly select 1,000 examples from both the training and validation sets. For CB, WSC, and COPA, we use their complete training and validation sets. This data split is adapted from the LoRETTA codebase (Yang al. 2024). Table 22 presents the statistical details of these datasets.
Table 22: Task descriptions and statistics of the datasets used in SuperGLUE benchmarks.
DATASET
TRAINING
VALID
TEST
TASK TYPE
#LABELS
METRIC
CB
250
250
NLI QA
Acc.
BoolQ
9,427
3,270
3,245
ACC.
WSC
554
104
146
CORFFERENCE
ACC.
COPA
400
100
500
CAUSALITY
Acc.
28

---

Published as a conference paper at ICLR 2026
D.4 IMAGE CLASSIFICATION BENCHMARKS
The dataset configuration for VIT is shown in Table 23 To simulate few-shot learning scenarios, we randomly sample 10 instances per class from the training for each experiment. For datasets with public validation or test sets (CIFAR 10/100 (Alex 2009) and Flowers 102 (Nilsback & Zisserman 2008)), evaluation is performed on the entire test sets, For datasets without given validation test sets CUB-200-2001 (Welinder et al. 2010)), we evaluate on the remaining training samples after sampling.
Table 23: Task descriptions and dataset statistics of the image classification benchmarks.
DATASET
TRAINING
VALID
TEST
# CLASSES
METRIC
CIFAR 10
50,000
10,000
10
TOP-1 ACC.
CIFAR 100
50,000
10,000
100
TOP-1 ACC.
FLOWEERS 102
1,020
1,020
6,149
102
TOP-1 ACC.
1UB-200-2011
11,788
200
TOP-1 ACC.
29

---

Published as a conference paper at ICLR 2026
E HYPERPARAMETERS
E.I ROBERTA
We fine-tune RoBERTa Base and RoBERTa Large Liu
2019 on six GLUE benchmark datasets (Wang et al. 2018). The learning rates and other hyperparameters used in our experi- ments are presented in Table
Table 24: Hyperparametens fine-tuning RoBERTa Base and RoBERTa Large models using LoRA, VeRA (with separate learning rates for classification head and tuning modules), and TRAC methods across different tasks.
METHOD
SST-2
MRPC
COLA
QNLI
RTE
STS-B
OPTIMIZER
ADAMW
LR SCHEDULE
LINEAR
WARMUP RATIO
0.06
BOTH
SEEDS
{0. 21, 81, 100)
BATCH SIZE
MATRIX RANK (LORA)
MATRIX RANK (TRAC)
16
ACTIVATION (TRAC)
(x) softmax((1)
EPOCH
10
20
20
10
20
20
LORA
5E-4
4E-4
4E-4
4E-4
5E-4
4E-4
VERA
4E-3
1E-2
1E-2
1E-2
4B-3
1E-2
LORETTA
5E-4
1E-3
2F-3
5E-4
1E-3
2E-3
BASE
TRAC
5E-3
5E-3
3E-3
5E-4
26-3
1E-2
TRAC (CONTROLLER)
5E-3
5B-3
3E-3
SE-4
2E-3
1E-2
TENSOR SHAPE
768
[8, 8, 12]
TENSOR RANK (A)
[1, 12]
TENSOR RANK (B)
[1, 21, 36]
EPOCH
20
20
20
LORA
4E-4
3E-4
2E-4
2E-4
4E-4
2E-4
VERA
1E-2
3E-3
IE-2
2E-4
2B-3
2E-2
LARGE
TRAC
5E-3
5B-3
5B-3
SE-3
5E-3
2E-3
TRAC (CONTROLLER)
5F-4
5E-3
5F-4
5F-3
5F-3
2F-3
TENSOR SHAPE
1024 18,8, 16]
TENSOR RANK (A)
[1,8, 12]
TENSOR RANK (B)
[1, 24, 36)
30

---

Published as a conference paper at ICLR 2026
E.2 LLAMA2
We fine-tune LLaMA2-7B and LLaMA2-13B (Touvron et al. 2023) on four SuperGLUE benchmark datasets (Wang et al. 2019). The learning rates and other hyperparameters used in experiments are shown in Table 25
Table 23; Hyperparameters for fine-tuning LLaMA2-7B and LLaMA2-13B models with LoRA and TRAC methods across different tasks.
METHON
CB
BOOLQ
WSC
COPA
OPTIMIZER
ADAMW
LR SCHEDULE
LINEAR
SEEDS
[0, 42, 100)
BOTH
EPOCH
BATCH SIZE
MATRIX RANK (LORA)
4
MATRIX RANK (TRAC)
16
ACTIVATION (TRAC)
((x)=
LORA
5E-4
1E-4
1E-4
5E-4
VERA
5E-3
1E-3
1E-3
1E-3
LORETTA
2E-3
5E-4
1E-3
2E-3
TRAC
4E-3
2B-3
2B-3
4E-3
TENSOR SHAPE
4096
[16, 16, 16]
TENSOR RANK (A)
[1, 48)
TENSOR RANK (B)
[1, (1,21,
LORA
5E-4
1E-4
IE-4
5E-4
VERA
1E-2
1E-3
2E-3
5E-3
LORETTA
16-3
5F-4
1T-3
1F-3
TRAC
3F-3
1E-4
1E-3
4F-3
TENSOR SHAPE
5120: : [16, 16, 20]
TENSOR RANK (A)
[1, 30, 60)
TENSOR RANK (B)
[1, 30, 60)
E.3 GPT2
For GPT2 Medium and GPT2 Large (Radford et al. 2019) on the E2E dataset (Novikova et al. 2017 ) perform grid search find the optimal learning rate with a step size of x 10 The optimal learning rates and other hyperparameters are shown in Table 26
Table 26: Hyperparameter configurations for TRAC on the E2E benchmark, for GPT2 Medium and Large models.
HYPERPARAMETER
MEDIUM
LARGE
GPUS
OPTIMIZER
ADAMW LINEAR
LEARNING RATE SCHEDULE
WEIGHT DECAY
0.01
BATCH SIZE
EPOCHS
WARMUP STEPS
500
LABEL SMOOTH
0.1
MATRIX RANK
16
ACTIVATION (TRAC)
(x)
LEARNING RATE
1E-2
8E-3
TENSOR SHAPE
1024 : [8, 8, 16]
1280 [8, 8, 20)
TENSOR RANK A
[1, 8, 18
[1, 12, 24
TENSOR RANK B
[1. 24, 54)
[1, 36, 72
31

---

Published as a conference paper at ICLR 2026
E.4 LLAMA3 AND QWEN3
We present the hyperparameters used for fine-tuning LLaMA3-8B @ Meta
2024) and
mathemati- cal reasoning Yuu et al. 2023] Cobbe et al., 202T), and code generation (Zheng et al. 2024 Chen 2021) benchmarks. Some of our hyperparameter settings are adapted from the experimental setups of DoRA (Liu et al., 2024a) and LoRA-GA (Wang et al. 2024). TRAC is tested under two parameter configurations; for clarity, hyperparameters used in our experiments are shown in Table 27
Qwen3-8B (Team] 2025) models the commonsense reasoning (Hu 2023).
label them as SMALL and LARGE. The learning rates and other
Table 27: Hyperparameters for fine-tuning LLaMA3-8B with LoRA and TRAC across commonsense reasoning (CR), mathematical reasoning, and code generation benchmarks.
HYPERPARAMETER
COMMONSENSE REASONING
MATH
CODE
OPTIMIZER
ADAMW COSINE
LR SCHEDULE
EPOCHS
1
BATCH SIZE
32
MATRIX RANK (LoRA)
4
MATRIX RANK (TRAC)
64
ACTIVATION (TRAC)
(zz
LEARNING RATE (LORA)
5E-5
5E-5
2E-5
LEARNING RATE (TRAC-SMALL)
5E-3
4g-3
6E-5
LEARNING RATE (TRAC-LARGE)
8E-3
5E-3
8E-5
TRAC-SMALL TENSOR SHAPE
4096
[16, 16, 16]
TRAC-LARGE TENSOR SHAPE
1096 664,4 16]
TENSOR RANK (A)
[1. 64, 128
TENSOR RANK (B)
[1. 64, 128
Table 28:
Hyperparameters for fine-tuning Qwen3-8B with LoRA and TRAC on commonsense reasoning benchmarks.
HYPERPARAMETER
COMMONSENSE REASONING
OPTIMIZER
ADAMW COSINE
LR SCHEDULE
EPOCHS
1
BATCH SIZE
16
MATRIX RANK (LORA)
4
MATRIX RANK (TRAC)
64
ACTIVATION (TRAC)
(xx) = Ø
LEARNING RATE (LORA)
SE-5
LEARNING RATE (TRAC-SMALL)
SE-3
LEARNING RATE (TRAC-LARGE)
5E-3
TRAC- TENSOR SHAPE TRAC-LARGE TENSOR SHAPE
4096: : [16, 16, 16)
1096 164, 161
TENSOR RANK (4)
[1, 64, 128]
TENSOR RANK (B)
[1, 64, 128]
32

---

Published as a conference paper at ICLR 2026
E.5 VIT
We conduct image classification experiments (Alex 2009; Nilsback & Zisserman 2008 Welinder
al., 2010) ViT-Base and ViT-Large (Dosovitskiy et 2020 Wu al. 2020), and report the corresponding hyperparameter settings below. For Head, LoRA, and TRAC, we search for the optimal learning rate among x 10 1 10 3 and x 10 For FT (Full Fine-tuning), we search for the optimal learning rate among 1 x 10 10-5, 5 x 10T and 1 x 10 The optimal learning rates
and other hyperparameters are shown in Table 29
Table 29: Hyperparameters for fine-tuning VIT Base and ViT Large models with different methods across different tasks.
METHOD
CIFAR-10
CIFAR-100
FLOWERS102
CUB-200-2011
OPTIMIZER
ADAMW
LR SCHEDULE
COSINE
INIT. SEEDS
[0, 42, 100]
BOTH
DATA SEEDS
[0, 1,
EPOCH
50
BATCH SIZE
16
MATRIX RANK (LORA)
MATRIX RANK (TRAC)
16
ACTIVATION (TRAC)
(x)
FT
IE-4
5B-5
5B-5
5E-5
HEAD
2B-3
2B-3
2B-3
2E-3
LORA
IE-3
SB-4
5B-4
5E-4
VIT-B
LORETTA
2B-3
IB-3
2B-3
1E-3
TRAC
2B-3
IB-3
2B-3
1E-3
TENSOR SHAPE
768
[8, 12]
TENSOR RANK (A)
[1, 8, 12]
TENSOR RANK (B)
[1, 24, 36]
FT
IB-5
IB-5
1B-S
IE-5
HEAD
5E-4
5B-4
2B-3
1E-3
LORA
5B-4
5B-4
5B-4
5E-4
VIT-L
LORETTA
5B-4
5B-4
2B-4
5E-4
TRAC
2B-3
IB-3
5B-4
2E-3
TENSOR SHAPE
1024 [8, 8, 16]
TENSOR RANK (A)
[1, 12, 16
TENSOR RANK (B)
[1, 36 48
33

---

Published as a conference paper at ICLR 2026
F USE OF LARGE LANGUAGE MODELS
In accordance with the ICLR 2026 policy on the use of Large Language Models (LLMs), we state the following. During the preparation of this paper, we used LLM-based tools exclusively to aid in language polishing and stylistic refinement of the manuscript (e.g., improving clarity, grammar, and readability). No parts of the research process itself including problem formulation, theoretical derivation, algorithm design, experimental setup, implementation, or analysis of results were conducted by or delegated to LLMs. Furthermore, LLMs were not used for literature retrieval, ideation, or to generate any substantive scientific content.
All technical contributions, proofs, models, and experiments presented in this work originate entirely
from the authors.
34


Reviews:
[{"review_id": "oj9s1NLvod", "reviewer": "Reviewer_vXG5", "summary": "This paper proposed a new parameter-efficient adaptation method for transformers. It uses tensor decompositions to target the extreme end of low-paramaters and further reduces the number of necessary parameters by sharing across layers.", "questions": "* given a thorough hyperparameter evaluation protocol, can the authors provide results on more recent architectures such as llama-3? \n* provide clarity on the use of the theoretical results and why they are included\n* Provide clarity on \"computational order\"\n* Provide insights on training time & memory required for training experiments.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* easy to understand and simple new PEFT method\n* compares well to SOTA methods on several benchmarks while being more parameter-efficient\n* releted works section is thorough and gives good overview", "weaknesses": "* the computational complexity analysis is a bit unclear. I assume this is for a) for training time and b) for the number of adapted parameters (this should be made more clear). But then VeRA's complexity seems wrong, as there's no n^2 complexity in its decompositions. How is this arrived at and what even is \"computational order\"?\n* The importance of the theoretical analysis is unclear. The fact that more parameters -> more expressivity is trivial and the fact that sharing across layers can be used too. Does this yield any insights into how to use the adapter or how it should be designed? Otherwise this seems like an appendix without a use. \n* the PEFT method builds on existing works such as LoReTTA and several works that rely on frozen random matrices\n* old models. By now, llama-2 and GPT2 benchmarks have become irrelevant and newer models should be used with correspondingly adapted benchmarks. examples include qwen, llama-3 and or vision-language models. \n* No mention of training times\n* minor: use citations with \\citep when referring to a paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0TSiqgaq0L", "reviewer": "Reviewer_kh71", "summary": "This paper proposes TRAC (Tensor-Train based Across-Layer Compression), a parameter-efficient fine-tuning (PEFT) framework that combines Tensor-Train (TT) decomposition with cross-layer core sharing and freezing. Each LoRA module is represented as a sequence of TT cores (trainable, frozen, shared), and lightweight controllers are added to modulate shared cores per layer. The method is evaluated on BERT, LLaMA2, GPT-2, and ViT models across GLUE, SuperGLUE, E2E, and image classification benchmarks. The authors report comparable or slightly better performance than LoRA and LoRETTA while using 10–20× fewer trainable parameters.\n\nWhile the method is clearly presented and theoretically consistent, the core idea is NOT novel. Similar hybrid parameterizations have been explored extensively in prior work-most notably COMPACTER (Karimi Mahabadi et al., NeurIPS 2021)—which achieves comparable or better results than full finetuning while training less than 0.05% of parameters, versus This approach substantially higher ratio (12-14%). Moreover, the evaluation is also limited to simple benchmarks, without results on reasoning, math, or code tasks that are now standard for assessing LLM generalization, benchmarks like MMLU, MMLU-pro, humaneval, MBPP, GSM8k, ... \n\nOverall, the contribution is not novel and not more efficient than prior work (compared to Neurips 2021), and the empirical evidence is insufficient to establish this method as a meaningful advance over existing parameter-efficient training methods.", "questions": "* Comparison with COMPACTER, and GaLore: how does the method compare in terms of performance and also #trainable parameters compared to existing work of compacter and GaLore? Without these, it is difficult to assess real performance advantages.\n\n* Parameter efficiency vs. compute efficiency: While TRAC reduces trainable parameters, how does it affect FLOPs, memory usage, or wall-clock training time? Can you provide quantitative data on actual resource savings, if any?\n\n* Evaluation scope: Do you plan to evaluate TRAC on modern reasoning, math, and code benchmarks (e.g., MMLU, GSM8K, MBPP, HumanEval)? Such tasks are now standard for assessing LLMs and would strengthen the empirical contribution.", "limitations": "", "rating": null, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is well-written with detailed method descriptions, and consistent notation.\nThe proposed TRAC framework is technically sound.\n\nThe authors conduct comprehensive experiments on multiple architectures (BERT, GPT-2, LLaMA2, ViT) and datasets (GLUE, SuperGLUE, E2E, and image classification), demonstrating that TRAC achieves similar or slightly better performance than LoRA and LoRETTA with fewer trainable parameters. However, these benchmarks are rather outdated and evaluations on more recent benchmarks such as MMLU, MMLU Pro, humaneval, MBPP, GSM8k, ... are needed.", "weaknesses": "* Comparison to prior work. The paper fails to acknowledge or compare against COMPACTER (Karimi Mahabadi et al., NeurIPS 2021), which is a foundational and much stronger baseline. COMPACTER achieved similar or better results while fine-tuning only 0.047% of parameters, whereas TRAC trains several orders of magnitude more parameters to reach comparable performance. This omission significantly weakens the contribution and demonstrates limited awareness of prior literature.\n\n* Lack of novelty and missing citations. The paper provides no substantial innovation beyond prior PEFT work. Its main components—Tensor-Train (TT) decomposition and cross-layer parameter sharing-have already been introduced in TT-LoRA (Anjum et al., 2024), LoRETTA (Yang et al., 2024), VeRA (Kopiczko et al., 2024), NoLA (Koohpayegani et al., 2024), and SharedAdapter (Poth et al., 2023). TRAC simply combines these established ideas without offering new theoretical insights or mechanisms.\n\n- Questionable efficiency gains. Although TRAC reduces trainable parameters relative to LoRA, it is unclear whether this translates into any real compute or time savings. The paper does not report FLOPs, memory footprint, or wall-clock improvements.\n\n- Weak empirical results. Gains over LoRA and LoRETTA are small and often within noise. Important baselines such as COMPACTER, and GaLore (https://arxiv.org/pdf/2403.03507) are missing, leaving the evaluation incomplete.\n\nLimited experimental scope. The paper only tests on small or mid-size models (BERT, GPT-2, LLaMA2-7B) and saturated benchmarks (GLUE, SuperGLUE, ImageNet). There is no evaluation on modern reasoning, math, or code benchmarks (e.g., MMLU, GSM8K, MBPP, HumanEval), which are standard in current LLM research.\n\nOverall assessment. TRAC appears to be an incremental reimplementation of prior TT-based and cross-layer methods, presented without proper acknowledgment of earlier work or competitive results. Its contribution is neither novel nor impactful compared to established baselines, particularly COMPACTER (Karimi Mahabadi et al., 2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qVq1ne1S7F", "reviewer": "Reviewer_Afac", "summary": "The paper studies a new PEFT adapter based on tensor-train format initialization. Compared with widely used PEFT methods like LoRA, the proposed method greatly reduces the number of trainable parameters. Compared with existing tensor-train–based adapters, the proposed method further includes freezing and sharing designs to achieve better performance without compromising parameter efficiency. Experimental results on the BERT, LLaMA, and ViT model families demonstrate the effectiveness of the proposed method in balancing efficiency and performance.", "questions": "- In the experiments, the authors set the LoRA rank to 4. However, I recall that the optimal setup for LoRA rank is typically 8 or 16. Would the performance gap between LoRA and TRAC increase if a higher LoRA rank were used in Tables 3 or 4?\n- I wonder if the tensor contraction in TRAC leads to higher latency compared with the LoRA adapter. I noticed an increase in training time with larger tensor orders in Table 11.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is well-written and provides sufficient background knowledge on the tensor-train format, making it easy to follow.\n- In addition to NLP tasks, the paper also considers vision classification tasks to demonstrate the effectiveness of the proposed method.\n- The paper provides theoretical analysis on the effectiveness of the TRAC method.", "weaknesses": "- Even though the proposed method has not been explored before, there is sufficient prior work on freezing (e.g., AFLoRA) or sharing (e.g., ShareLoRA) parts of adapters. This work appears to mainly combine these ideas, which limits its overall contribution.\n- I wonder if the authors could provide a more direct explanation for the motivation behind studying an even more parameter-efficient method than LoRA. In my opinion, further reducing parameters may not significantly decrease the training cost compared with standard LoRA methods.\n- I am satisfied with the experiments on the BERT and ViT models. However, for the experiments on the LLaMA-2 model, the tested tasks seem too limited. I wonder if the authors have evaluated the proposed method on more challenging reasoning tasks, such as those used in the DoRA paper. These tasks might better align with today’s real-world PEFT setups. Additionally, all models considered in this paper are somewhat outdated; I would expect some experiments on more recent models.\n- I am quite interested in the results regarding the symmetry analysis in Appendix A.3. Could the authors provide more intuition behind these results? Are there similar findings in LoRA-style works?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o1Dsi3023o", "reviewer": "Reviewer_Sk7c", "summary": "This paper introduces TRAC (Tensor-Train based Across-layer Compression), a framework for parameter-efficient fine-tuning (PEFT) of large pre-trained models using Tensor-Train (TT) decomposition. TRAC combines tensor decomposition with across-layer parameter sharing and freezing, incorporating lightweight controllers to enhance adaptability. The paper evaluates TRAC on diverse architectures, including LLaMA-2, GPT, BERT, and ViT, across tasks such as text classification, text generation, and image classification. The results demonstrate that TRAC achieves comparable or superior performance to existing methods (e.g., LoRA, VeRA, and LoRETTA) with significantly fewer trainable parameters. The paper claims a reduction of up to 20× fewer trainable parameters for LLaMA-2-13B on SuperGLUE tasks while maintaining or improving performance.", "questions": "1. Could the authors provide more intuition about why gating shared cores with vectors (vs. low-rank residuals or elementwise modulation) is expressive enough? Have you explored alternative controller architectures (e.g., learned affine transforms, attention-based modulation)?\n2. Could the authors include a more explicit guideline or heuristic for selecting tensor shapes under a fixed parameter budget? How sensitive is the performance to suboptimal tensorization choices?\n3. Have the authors identified any regimes (e.g., early/late transformer blocks, deeper MoE architectures) where sharing harms performance? Would allowing partial sharing (e.g., blockwise) improve robustness? If not tested, can the authors comment on expected behavior?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method achieves up to 20× parameter reduction compared to LoRA in certain tasks while maintaining comparable or superior performance. This is particularly impactful for resource-constrained scenarios.\n2. TRAC effectively combines Tensor-Train decomposition with cross-layer parameter sharing and freezing, which is a novel and well-motivated design. This approach directly addresses the redundancy in deep transformer layers.\n3. The experiments are conducted across various architectures (e.g., BERT, LLaMA, GPT, ViT) and tasks (e.g., GLUE, SuperGLUE, E2E, image classification). The paper includes comparisons with strong baselines like LoRA, VeRA, and LoRETTA, supported by statistical analysis.", "weaknesses": "1. While the paper emphasizes the importance of lightweight controllers, the experiments do not deeply explore the trade-offs of different controller designs or activation functions. This limits the understanding of the controller's impact on performance.\n2. The paper does not sufficiently discuss some related PEFT  methods, such as AdaLoRA[1], MPOP[2]. Including a broader discussion would enhance the positioning of TRAC in the literature.\n3. While the paper claims that higher-order tensors do not improve performance, the exploration of tensor orders beyond third-order is limited to one or two configurations. A more detailed analysis would help justify this design choice.\n\nRef:\n[1]. Zhang Q, Chen M, Bukharin A, et al. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning[J]. arXiv preprint arXiv:2303.10512, 2023.\n[2]. Liu P, Gao Z F, Zhao W X, et al. Enabling lightweight fine-tuning for pre-trained language model compression based on matrix product operators[J]. arXiv preprint arXiv:2106.02205, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)