LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
Authors: Ruijie ZHANG
Ziyue Liu
Zhengyang Wang
Zheng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Pre-training
Efficient Fine-tuning
Adapters
Low-rank Training
Vision Transformers
Large Language Models
Primary area: deep_learning

Abstract: Training foundation models such as ViTs and LLMs requires tremendous computing cost. Low-rank matrix or tensor factorization offers a parameter-efficient alternative, but often downgrades performance due to the restricted parameter space. In this work, we introduce ${\textbf{Latent Crossing (LaX)}}$ -- a simple yet effective plug-and-play module that enhances the capacity of low-rank models by enabling information flow across low-rank subspaces. We extensively validate the benefits of LaX on pre-training tasks with ViT-Base/Large and LLaMA-like models ranging from 60M to 1B parameters. LaX boosts low-rank model performance to match or exceed the full-rank baselines while using 2-3$\times$ fewer parameters. When equipped with low-rank adapters (i.e., LoRA) for fine-tuning LLaMA-7/13B, LaX consistently improves performance on arithmetic and common sense reasoning tasks with negligible cost.

LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
Ruijie Zhang', Ziyue Liu", Zhengyang Wang, Zheng Zhang?
University of California at Santa Barbara
(ruijiezhang, ziyueliu, zhengyangvang)Juess) edu,
zhengzhang@ece.ussie
Abstract
Training foundation models such as ViTs and LLMs requires tremendous comput- ing cost. Low-rank matrix tensor factorization offers a parameter-efficienl alter- native, but often downgrades performance due to the restricted parameter space. this work, introduce Latent Crossing (LaX) simple yet effective plug-and- play module that enhances the capacity of low-rank models by enabling informa- tion flow across low-rank subspaces. We extensively validate the benefits of LaX on pre-training tasks with ViT-Base/Laarg. and LLaMA-like models ranging from 60M 1B parameters. LaX boosts low-rank model performance match ceed the full-rank baselines while using 2-3x x fewer parameters. When equipped with low-rank adapters (i.e., LoRA [23]) for fine-tuning LLaMA-7/13B, LaX con- sistently improves performance on arithmetic and common sense reasoning tasks with negligible cost.
Introduction
Following neural scaling laws [28 Pz. 33), the size and training data of foundation models have grown rapidly, exemplified by models such ViT-22B [ GPT-3 (175B) [3], LLaMA-3 (405B) [[3]] and PaLM (504B) [S]]. These large-scale foundation models have achieved remarkable suc- cess in diverse applications such as language and vision. However, their success comes at immense computing cost, typically on the scale of multi-million GPU hours per pre-training run. As the unsustainable trend continues, training or even deploying such foundation models has become pro- hibitively expensive for most research institutions and organizations around the world.
To address these challenges, the community has become increasingly interested low-rank approx- imation techniques. This is largely motivated by the empirical observation that weight matrices in deep neural networks often exhibit low effective ranks [II, [T], An 5e. 93). Classical matrix compres- sion techniques (such as singular value decomposition (SVD)
and tensor decomposition meth- ods (c.g. Canonical Polyadic (CP), Tensor Train (TT) [TG gl 2 and Tucker decomposition [[]]]] have been widely applied reduce the number trainable parameters by instantiating and updating the lightweight low-rank factors [31,, 83. 12, 557, 1001 566 39, 655, B4]] These approaches exemplify the paradigm of "low-rank training" and have achieved varying degrees of success. In particular, parameter-efficies fine-tuning (PEFT) [23 63, ATL 17 62 has drastically reduced the barrier to fine- tuning large language models while producing competitive results. Recent efforts [[S] 66. ISS 60] have extended similar concepts to pre-training. Although low-rank methods typically reduce the model size and computing cost, they introduce critical trade-off: smaller ranks yield lower ca- pacity and often harm performance, whereas larger ranks incur additional cost, undermining the intended efficiency (see Fig. (a)).
Equual contribution Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
(c)
Impact of Rank (SVD)
Low-rank Models +La
Reasoning Tasks (LoRA +LaX)
78
78
LoRA
52-
76
75
75
Arcuracy
74-
74
78
73-
73
72-
74
VIT-8
VIT-0
71
71
Models
+LaX
+LaX
64
128 256
Tansor Train
SVD
CoLA
PIDA
90AA
HellaSwag
WinoCrande
e
QBQA
Figure 1: LaX boosts the performance of low-rank training methods. (a) SVD-based pre-training ViT-B on ImageNet-1K IK with different matrix ranks: lower-rank leads to greater performance drop: LaX consistently improves the performance in all settings. (b) Pre-training ViI-B on ImageNet-IK with different low-rank methods. LaX sigmificantly improves performance for all low-rank methods, even surpassing the full-rank pre- training. (c) Fine-tuning LLaMA-7B on commonsense reasoning tasks using LoRA, with and without LaX respectively, LaX improves LoRA's fine-tuning performance in all tasks.
In this work, we propose Latent Crossing (LaX), lightweight, drop-in module designed to enhance the capacity of low-rank models without explicitly increasing matrix/tensor ranks. By allowing infor- mation flow across low-rank subspaces via residual connections, LaX improves model performance while keeping the parameter budget nearly unchanged. Importantly, LaX can be seamlessly inte- grated with existing low-rank modules such LoRA [23]], SVD, CoLA | and TT [A5]], serving as a plug-and-play performance booster that significantly narrows or eventually closes the gap between low-rank and full-rank models.
We summarize our contributions as follows:
We propose LaX, lightweight module that increases the capacity of existing low-rank structures without compromising efficiency. By allowing information flow across low-rank subspaces via residual connections, LaX consistently boosts performance in both pretraining and fine-tuning settings (Fig. M)..
2. We design LaX Gate to align mismatched bottlenecks for low-rank models. To support diverse ar- chitectural and computational constraints, we introduce several variants of the LaX gating mecha- nism, each balancing expressiveness and efficiency under different deployment settings. We also provide practical guidelines for adapting LaX gating to a variety of tasks.
We evaluate LaX in of low-rank pre-training and fine-tuning experiments for both language and vision foundation models. In ViT pre-training, LaX improves accuracy by up to 4.32% on ImageNet- 1K. For LLM pre-traine pre-training. LaX shows consistent gains across various model scales and different low-rank architectures. When combined with LoRA for fine-tuning, LaX enhances reasoning capabilities of I.LMA-7B/113B on both arithmetic and commonsense reasoning tasks.
2
Related Works
2.1 Low-rank Factorization for Neural Networks
To mitigate the high computational and storage costs associated with large models, low-rank factor- izations have been widely explored as an effective strategy [ELZ, M. Early efforts focused on applying low-rank matrix factorization, such SVD, to compress layers [. 26. 3II 50. 61]]. More recently, LoRA-style adapters [23] T A 17 62 extend this idea by adapting SVD-like modules onto frozen pre-trained weights, enabling efficient fine-tuning of large foundation models. Besides, low-rank ten- sor factorization, including tensor train (TT) [40, 16, 554], Canonical Polyadic (CP) [IG, B. 2%], Tucker decomposition [[5]], and other tensor-based formats [67 32, QL 50 have shown promise for reducing complexity of models [HI,, 3R. 52, 42
39
65.
64
60]. In this paper, we focus on representative
methods from both directions:
We provide code here

---

Low-rank Matrix Factorization. SVD factorizes a weight matrix W R as W = BA. where A grxdm and B Rdouxr, resulting reduced parameter count of r(dnn + dout). CoLA [ further extends this factorization to an autoencoder by injecting nonlinear activation between and B, replacing a linear layer Wx with the bottleneck structure B a (Ax).
Low-rank Tensor Factorization. We adopt tensor train decomposition [H6] as representative higher-order factorization method. It reshapes a weight matrix W E IR Ront into an order-n
121
tensor W E TR with dout din- II di and decomposes it with sequence of tensor <=0
cores {C ....,,C--11 where each C gerxddxri1, The weight is represented via chain of
tensor contractions:
W X3,1 C³ X3,1 ... X3,1 Cn-1
However, the extent of parameter reduction, as well as its benefits heavily depend on the chosen rank. Lower-rank settings often suffer from loss of expressiveness and degrade model perfor- mance [66. 93, 23, 59 42, II while higher ranks reintroduce computational overhead. In this work, propose simple, -ldg-andy module that complements general low-rank training methods in neural networks, aiming to recover lost performance without compromising their efficiency.
2.2 Residual Mechanism
ResNet [TS]] stands as one of the most influential milestones in deep learning by introducing skip connection that routes a layer's input directly to its output. This residual mechunism mitigates the vanishing gradients and enables the stable training of deep networks with hundreds of layers. This design principle has been broadly adopted in numerous architectures, including recurrent neural networks [[4],, transformers [9] 53), and diffusion-based models [2T]. In addition to its empirical success, the authors provided theoretical justification for the residual connection [T&]. Building on this foundation, subsequent research has proposed various improvements and theoretical analyses to further enhance the residual learning paradigm [58,, 20, 555, 65], consistently emphasizing the central role of residual pathways in improving both convergence and generalization in deep networks.
Inspired by ResNet, we propose Latent Crossing (LaX). LaX serves as model performance booster by enabling information flow across low-rank subspaces, restoring expressiveness often lost due to rank constraints. Across wide range of tasks, LaX delivers consistent performance gains while preserving the efficiency advantages of low-rank architectures.
3 The LaX Method
Our goal is augment existing low-rank models with a lightweight module that recovers the per- formance typically lost due to the low-rank constraints. In Section I.I. we present the background and motivation behind this work. Following this, Section B.2 introduces the design of the LaX mod- ule for different low-rank structures, Section BK introduces LaX Gate and outlines its key variants, and Section G offers practical guidelines to facilitate its integration into wide range of low-rank training frameworks.
3.1 Latent Crossing
As discussed in Section 2.11, given a weight matrix W E Rdo from arbitrary linear layer, low-rank methods approximate it as low-rank factors. We take SVD as a motivating example to illustrate this concept. Let x; E dd denote the input to the i-th low-rank layer. A down-projection matrix A; E xdd maps X; into lower-dimensional latent representation h, E R", which is subsequently transformed back the output space using an up-projection matrix B, E 1gdon Xr,
h = A, E. R",
B(Axx)) = B,h, E Rdout.
This factorization reduces the number of parameters by choosing a smaller rank r and compresses input into a narrow latent space, which can lead to information bottlenecks, often resulting drop of performance due to the constrained searching space. Empirically, increasing the rank typically improves performance but diminishes the efficiency benefits of the low-rank approach due to increased parameter count and computation.
3

---

Dense Layers
SVD/CoLA Layers
TT Layers
LORA Layers
oulputs
outputs
outputs
outputs
C
B;
dense layer weights W;
FREEZE
B,
LaX
pretrained weights W;
A
A
outputs
outputs
LaX
outputs
LaX
CL
B
FS
dense layer weights Wt
FREEZE
B
Lax
A
C
weights W
pretrained
A
Inputs
Inputs
Inputs
(a)
(8)
(c)
(d)
Figure 2: LaX is general module that be plugged into low-rank neural network models. (a) Dense layers:
information flow, effective but computationally expensive. (b) SVD/CoLA [B3]] layers: rank-r bottlenecks with two factors; LaX can be inserted into the latent space between layers. (c) Tensor train layers: bottleneck structure with four tensor cores, where data flow is governed by tensor contractions; LaX can be applied either between cores across layers. (d) LoRA adapters: LaX can be placed between different adapters.
Our motivation is that, can we improve the performance of low-rank layers without explicitly increasing the physical rank r? Our answer yes. Instead of directly applying the up-projection B, to h, LaX incorporates latent features from the previous layer into the up-projection process. Formally, propose LaX as follows:
h 1 h; = A;X; ER,
y: BB, + h:-1) E Rdduu.
Equivalently, if we stack inputs as 8;
E R2di,, then LaX can be formulated as Xi-1
x;
y, = w (ZaX)
Xis
w = [,A; B.A:-]] E R Row X2dm
Since h naturally produced by the preceding layer during the forward pass, this implicit reuse of intermediate representations facilitates direct information flow across consecutive low-rank pro- jections, requiring no additional parameters computation overhead.
3.2 Variants of LaX
LaX is general module that is widely applicable to low-rank structures. Eq (0) mainly describes its implementation on matrix factorization methods, where LaX is applied across two consecutive lay- ers. We also refer to this implementation as Inter-Layer LaX. In modern architectures such as the transformer, apply Inter-Layer LaX between the same type of layers across transformer blocks, ie., from attention (QKV projection) to attention, and from MLP to MLP. This design preserves structural and semantic alignment in residuals while avoiding cross-type interference.
For more fine rree-graind low-rank structure, such tensor factorization methods, LaX is not limited to laoss-layer only. Take the tensor-traii representation in Fig. E as an example: W (dropping layer index for simplicity) is approximated by 6 low-rank factors {C", C_ .c,,..,,C' Cs), therefore, a series of latent features will be produced when sequentially contracting each factor, such Cº C x, p' etc. Along this contraction sequence, earlier results can be used as residuals form multiple LaX pathways. We refer to this implementation Intra-Layer LaX.

---

Intra-Layer LoX
ri
T3
C
C
C2
C
C4
C5
To
T2
T4
Intra-Layer Lax
Figure 3; A 6-core Tensor Train layer with the symmetric setting. For Tensor Train layers with identical input and output shapes, can naturally arrange the tensor ranks a symmetric configuration, where 7b 74 and ri = 73 this example. This reduces the need shape transformation operations, making Intra-Layer LaX more efficient when applied.
3.3 LaX Gates
When the latent dimensions are aligned (e.g.. SVD with the same rank between layers, symmetric TT ranks within layer), direct residual path- ways can be formed without introducing extra parameters. For example, in QKV projection layers with symmetric TT setup (Fig E), we config- ure ensure residual compatibility. However, a direct addition becomes infeasible when latent features have mismatched dimensions. To handle this, we introduce LaX Gate, a module that aligns and modulates latent features before residual fusion (Fig. H). The gated residual formulation becomes:
A
B
G(-)
A
B:
y; W (LX) Xi
LLx) == B,A; B,G.A,-I]-
(4)
To accommodate different architectural needs and promote the versatility of LaX, unify the notion by introducing the following Gate variants:
Figure 4: LaX Gate.
Identity Gate: Passes latent features through a direct addition without introducing additional parameters, i.e. G(-)=1)
Linear Gate: Introduces a single trainable parameter to control how much information is passed forward, i.e G(.) = where IR is trainable parameter.
Tensor Gate: As illustrated in Fig. this variant first folds the latent feature vector into a tensor R E Roxixx,, then contracts it with two learnable gate tensor cores: C E g xro and C E Reixx/x1 j.e. R = CO X3,1 R X3,1 C X3,1 E RRxxier, to match targeting shape vo x ri- the two-core setting, each gating core includes a singleton dimension. This dimension can be generalized to larger sizes when extending the design to more than two gating cores, allowing for greater flexibility across different use cases.
Dense Gate: Passes latent feature using a G RR linear layer.
We remark that the overhead introduced by LaX in terms of parameter count and computation is often minimal, since the latent rank is relatively small (e.g., 64 or 128) in low-rank models. In ad- dition to addressing dimensional misalignment, we empirically find that LaX Gate can further boost performance with negligible parameter overhead (see details in Section I for ViT pre-training with Tensor Gate). Therefore we also experiment LaX Gate on scenarios where dimensions matched.
3.4 Feature Normalization
Additionally, following the common practice of normalizing features after residual connection, postpend Layer Normalization (LN) to each LaX pathway. With this normalization, the final form of LaX as follows:
y, LN(W! (LaX) &)).
WLLxx) = B, B.G.IA.-)].
(5)
3.5 Practical Guideline
Here, we provide practical guidelines for selecting the appropriate LaX variant based on empirical observations across different tasks:
5

---

Contractins
1
C
C
Output
Figure 5: Two-Core Tensor Gate. A residual tensor R E 1 is contracted with two gating tensor cores,
C and C² producing transformed residual tensor R RR R
ViT-B
ViT-L
Method
Variant
# Params (M)
Accuracy (%)
# Params (M)
Accuracy (%)
Original
86.56
76.74
304.33
77.10
Base Model
44.17
75,20
115.77
76,81
SVD
Lax (Ours)
44.24
77.20 (+2.00)
115.92
78.60 (+1.79)
Tensor Train
Base Model LaX (Ours)
41.18
71.11
101.97
75,21
41.44
75.43 (+4.32)
102.10
77.77 (+2.56)
Base Model
44.17
76,04
115.77
77.63
CoLAA
LaX (Ours)
44.24
77.84 (+1.80)
115.92
79.07 (+1.44)
Table I: Accuracy comparison of pre-training ImageNet- Ik datasets, LaX consistently improves pre-training performance various low-rank models and scales. When applied to CoLA [X)], CoLA+LaX achieves the highest accuracy on both ViT-B and VIT-L.. Tensor Train models observe largest gains, with improvements of +4.32%/42.566f on ViT-B/L.
ViTs Pre-training Task: Vision Transformer pre-training typically involves multiple epochs over medium-sized datasets. Under this setting, we observe (see Tab. M1 that the Tensor Gate consis- tently outperforms other gate variants. We therefore recommend using the Tensor Gate for ViT pre-training tasks.
LLMs Pre-training Task: contrast to ViTs, large language model pre-training typically in- volves processing significantly larger number of training tokens across vast semantic space (e.g., a vocabulary size of 32,000 in LLaMA-1/2), often without completing a full training epoch. In such case, empirically find that Identity Gate off-the-shelf provides consistent and signif- icant improvements to different low-rank architectures, with zero parameter overhead and only negligible compute overhead (see Section (.2))
Fine-Tuning Task: The optimization space of fine-tuning is already constrained and therefore very small, where introducing additional parameters is often unnecessary. In these scenarios, we recommend using the Identity Gate Linear Gate to preserve training efficiency while enabling information flow across low-rank subspaces.
4 Pre-training Experiments
We first evaluate the performance of LaX in some low-rank pre-training experiments of ViTs/LLMs.
4.1 Pre-training Vision Transformers
We pretrain ViT-Base/l.arge (224 resolution with 16 patch size) and its corresponding low-rank variants on ImageNet- 1K [89]. We consider SVD, Tensor Train and CoLA [ as the baseline low-rank models. All models trained from scratch for 300 epochs according to the setting of [Я]. For SVD and CoLA, we only place Inter-layer LaX with the Tensor Gate. For tensor train, use both Inter-Layer LaX and Intra- Layer LaX with the Tensor Gate. More details the model and training configurations provided in Appendix A.
4.25
3.75
B
3.50
3.25
3.00
2.75
As shown Tab. I. all low-rank baselines experienced accuracy drop compared to the full-rank training. However, LaX consistently improves performance across the three baseline methods (see the training curve in Fig. & additional curves in Appendix A with negligible parameter
2.50
100
200
300
Figure 6: Training Loss

---

rank=256
rank= 128
rank=64
Gate
Params (M)
Accuracy (%)
Params (M)
Accuracy (%)
# Params (M)
Accuracy (%)
Base Model
44.17
75.20
22.94
74,47
12.32
71.26
+Identity Gate
44.17
75.81 (+0.61)
22.94
74.65 (40.18)
12.32
71.64 (+0.38)
Lineear Gate
44.17
76.11 (+0,91)
22.94
75.31 (+0.84)
12.32
71.72 (+0.46)
Tensor Gate
44.24
77.20 (42.00)
22.97
76.35 (+1.88)
12.34
72.94 (+1.68)
Dense Gate
48.55
77.03 (+1.83)
24.04
75.43 (+0.96)
12.60
72.33 (+1.07)
Table 2:
Pre-training performance of different LaX Gates on SVD-based ViT-B under varying rank settings. LaX consistently improves performance across all configurations, with the Tensor Gate achieving the largest
gains while incurring minimal parameter overhead.
overhead ( 0.2%). On the ViT-B scale, it recovers the lost performance, boosting accuracy by +2.00% % 77. 77.20% for SVD. For Tensor Train, LaX lifts accuracy from 71.11% 75.43% (+4.32%) turning the weakest model into a strong contender. Even for CoLA, the best pest-performing baseline, LaX adds boosts the aceuracy by 1.80%, reaching 77. ..%8.. Similar trends are observed in ViT-L.
We further evaluate the impact of different LaX Gate variants under varying rank r configurations in ViT pre-training. As shown Tab. all gate variants consistently improve accuracy across different rank settings compared to their respective base models. Among them, Tensor Gate achieves the highest gains with minimal parameter overhead. At rank 256, Tensor Gate improves accuracy by +2.00% with only +0.07M additional parameters. As the rank decreases, the added parameter cost also diminishes: at rank 128/64, the Tensor Gate requires only +0.03M/+0.022 more parameters to achieve +1 +1.88%/+1 1.////11..6. 1.68% accuracy gains, respectively. Additional experiments can be found in Appendix B
Gate Type
FLOPs Overhead
Model
Computation Complexity
Res / Norm
((nr)
Original
Ond n²dd
Identity
((1)
SVD/Col
O(ndr + nddd)
Linear
((r)
Tensor Train
Ondrr + n² لالل
Tensor
(
Dense
(nr))
Table 3: Model Complexity (per block) under batch size 1.
Table 4: LaX Gate Overhead under batch size 1.
Complexity Analysis We further analyzed the computational complexities (measured by the num- ber of FLOPs in each transformer block) of the original models and the gating mechanisms of LaX. Tab. B shows the FLOPs of the models without LaX while Tab. E shows the additional FLOPs re- quired by LaX gating, where the sequence length, is the hidden dimension, and r the rank. As shown by the tables, the computation overhead introduced by Res, Norm, Identity, Linear, and Tensor Gate is least an order of magnitude smaller than the original models, and so negligible. Dense Gate introduces overhead that is quadratic in but it could still be acceptable if r << d.
4.2 Pre-training Language Models
We further evaluate LaX in language model pre-training tasks where previous work suggests that pure low-rank architectures often cause performance drop [37,, 66, [S). More recent work such as CoLA [ and LORO [_]]] have shown promising results by imposing low-rank activations or performing manifold optimization. Since LORO optimizes A and B in the rank-r manifold that
W BA lies on, the proposed formulation of LaX contradicts this assumption. Consequently, we compare LaX with SVD and CoLAD, and directly cite the results reported in [66,
15. 92 [3]
We adopt the same experimental setup from recent benchmarks [[6 15
pre-training LLaMA-like models from 60M to IB parameters on C4 [А without data repetition and using compute-optimal token-to-pane porarmmeeer ratiosfi All linear layers in the original L.AMMA architecture are replaced with low-rank layers. For CoLAA, we follow [453], and implement the SVD baseline by removing its low-rank activations and/or restoring the original activation. All methods use the same rank for fairness. Full training details are provided in Appendix A
4%
This provides loose upper bound complexity, but still tighter than the one reported in [S]] I Due to resource constraint, we focus on haseline architectures that performed better in our experiments
'hee peen-to-parameter (T2P) ratios are roughly compute optimal [[2]]
7

---

60M
130M
350M
IB
r/d
128 / .122
256 / 768
256 / 1024
512/ 2048
Tokens
1.IB
2.2B
6.4B
13.1B
PPI.
Param
Mem
PPI.
Param
Mem
PPL.
Param
Mem
PPI.
Param
Mem
Full-rank
34.06
58
0.43
24.36
134
1.00
18.80
368
2.74
15.56
1339
9.98
ReLoRA 8X
37.04
0.37
29.37
134
0.86
29.08
1.94
18.33
1339
6.79
GaLore [68
34.88
58
0.36
25.36
134
0.79
18.95
368
1.90
15.64
1339
6.60
SLTTain [
34,15
44
0.32
26.04
97
0.72
19.42
194
1.45
16.14
646
4.81
LORO [+
33,96
43
0.32
24.59
0.70
18.84
185
1.38
15.19
609
4.54
SVD
36.25
43
0.32
26.84
94
0.70
21.18
185
1,38
16,54
609
4.54
SVD + LaX
(-2.71)
33.54
44
0.33
(-2.21)
24.63
94
0.70
(-2.28)
18.90
185
1.38
(-1.03)
609
4.54
CoLA
34.04
43
0.32
24.48
0.70
19,40
185
1.38
15.52
609
4.54
33.21
24.21
18.51
14.78
CoLA LaX
0.33
0.74
609
4.54
(-0.83)
(-0.27)
(-0.89)
(-0.74)
base models against other low-rank methods on pre-training C4 dataset [[ from 60M to IB. We report the validation perplexity (PPL. (4))), number of parameters in millions (Param), and the estimated total memory usage in GB (Mem) excluding activations based on BF16 precision. Results other than LaX and vanilla SVD from
Table 5: Comparisons LaX
LaX
60M
130M
350M
Res
Gate
PPL
Rank
PPL
Rank
PPL
Rank
Full-Rank
34.06
512
24.36
768
18.80
1024
LORO
33.96
128
24.59
256
18.84
256
SVD
X
x
36.25
26.84
21.18
/
33.61 (-2.64)
128
24.63 (-2.21)
18.90 (-2.28)
256
Lower Rank
/
33.54 (-2.71)
24.66 (-2.18)
18.93 (-2.25)
CoLA
X
x
34.04
24.48
19.40
Lower Rank
V
x
33.82 (-0.22)
128
24.37 (-0.11)
256
18.81 (-0.59)
256
/
33.21 (-0.83)
24.21 (-0,27)
18.51 (-0.89)
SVD
X
33.45
26.20
256
19.68
384
31.62 (-1.8)
224
23.86 (-2.34)
1 18.39 (-1.29)
Higher Rank
/
31.82 (-1.63)
23.97 (-2.23)
384
1 18.21 (-1.47)
512
CoLA-
31.52
23.97
256
18.32
384 512
Higher Rank
x
31.42 (-0.10)
224
23.74 (-0.23)
384
17.53 (-0.79)
30.90 (-0.60)
23.42 (-0.55)
17.34 (-0.98)
Table 6:
Comparisons LaX on SVD/CoLA between different Gate variants (xdenotes Identity Gate, denotes Dense Gate) and rank choices across 60M to 350M seales. For scenarios where a vector ranks is provided, smaller one is for attention layers and the larger one is for MLP layers.
As shown in Tab. N LaX improves the validation perplexity of SVD and CoLA across all scales. In particular, vanilla SVD performs poorly compared to most baselines but can be boosted to per- form on par with surpassing LORO and CoLA. While CoLA perform similarly to LORO, its LaX bboosted version surpasses LORO on all scales. In addition, LaX just uses the standard Adam opti- mizer and does not need LORO's complex manifold gradient computations and deeply customized training strategies
Tab. R compares SVD/CoLA variants across different ranks and LaX configurations. From Tab. E we observe that with either Identity Gate (denoted by x) or Dense Gate (denoted by v), LaX con- tinues to increase performance. In particular, the results in Tab. R further demonstrate that LaX is consistently effective regardless of whether the base model has a higher rank. The trend continues to hold that LaX boosts more weaker base model than stronger base model.
The only mixed message in Tab. B is the effectiveness of LaX Gate. In CoLA from 60M to 350M, the Dense Gate consistently outperforms the Identity Gate. However, for the SVD method, a Dense Gate does not provide a consistent benefit. When results are mixed, the nule-of-thumn is to be conservative; therefore, we recommend using Identity Gate in language model mre-training, as it
'LORO requires periodical computations of manifold gradient which involves tuning the update frequency. each update step learning rate warm-up and refreshment of Adam statisties.
8

---

Model
Method 32)
Params (%)
MultiArih
GSMBK
AddSah
AQuA
Singleki
SNAMP
LLaMA-7B
LaRA
LoRA + LaX (Ours)
0.83
0.83
96.9 (-1.9)
95.0
37.7
36.1
(-1,))
84.8 (-0.5)
19.3
(-1.)
17.7
87.8 (+344)
53.6 (+188
51,8
63.4+1.8
61.6
LLaMA-13B
LoRA
0.67
95.2
47.5
86.0
1822
89.8
54.6
65.2
(Ours)
67.3
Table 7: Accuracy comparison of LoRA and LaX- LoRA on six math reasoning datasets.
Model
Methed ir 32)
Parma (5)
RoolQ
PIQA
SIQA
HollsSwag
WinoGrande
ARC-
ARC-
OBQA
Avg
LLaMA-78
LoRA
0.83
68.9
80.7
77.4
78.1
78.8
71.8
61.3
74.8
74.7
(Ours)
(+1.5)
LLMMA 130
LoRA
0.67
72.1
83,5
80.5
90.5
83.7
82.8
68.3
82.4
80.5
LoRA Tax (Ours)
0.67
85.4
(41.))
81.3
(103(+
84,1 (40.4) 84.4 (+1.6)
7L8 (43.5)
(4355)
83.1 (40.7)
(416 (111)
Table 8: Comparison of LoRA and LaX-LoRA Commonsense Reasoning Benchmarks.
already effectively boosts SVD and CoLA architectures. Consequently, conduct the pre-training experiments on the IB scale using the Identity Gate.
On larger-scale CoLA (e.g., 350M and IB), LaX tends to bring more benefit, contrary to the stable or decreasing trend observed in SVD. This may be caused by the architectural difference between SVD and CoLA, indicating that CoLA might benefit more from LaX when it scales up.
Fine-tuning Experiments
Finally we show the effectiveness of LaX in low-rank fine-tuning. We incorporate LaX into LoRA (Fig. (d)) and consider two widely used reasoning benchmarks 25]:: Arithnetic/Commonsenee Reasoning. The fine-tuning configuration in this section strictly follows [2] Additional configura- tion details are provided Appendix A.4.
5.1 Arithmetic Reasoning
We fine-tune LLaMA-7B /13B on the Math10K dataset and assess performance in six arithmetic reasoning subtasks. For this evaluation, we configure LaX with Linear Gate.
Tab. 0 shows that augmenting LoRA with LaX yields consistent accuracy improvements across all six arithmetic subtasks for both LLaMA-7B and LLaMA-13B. For the 7B model, the average score improves from 61.6% to 63.4% (+1.8%), while for the 13B model it increases from 65.2% to 67.3% (+2.1%), suggesting that LaX's rank expansion mechanism effectively provides additional representation capacity required by fine-tuning. Even on subtasks where LoRA already performs strongly (e.g. MultiArith, AddSub), LaX delivers consistent improvements.
5.2 Commonsense Reasoning
Following [25] [I]], we merge the training datasets from all eight commonsense reasoning tasks into a unified training set and evaluate the performance separately on each task. In this experiment, we configure LaX with the Identity Gate. As shown in Tab. & LaX consistently outperforms the LoRA baseline in all reasoning tasks. For LLaMA-7B, the average accuracy increases from 74.7% to 77.3% (+2.6%), with notable gain of +6.6% on HellaSwag. For LLaMA-13B, the overall score rises by +1.11119 %;; only BoolQ exhibits a marginal decline (-0.8%).
Conclusion
In this work, we have presented Latent Crossing (LaX), lightweight and versatile module de- signed to improve the training performance of low-rank compressed models. Although low-rank methods are effective in reducing computational overhead, they often suffer from a loss in model expressiveness and performance. LaX addresses this limitation by enabling information flow be- tween low-rank subspaces through residual connections equipped with simple gating mechanisms. As result, LaX serves as a general plug-in booster that enhances wide range of low-rank models, across both pretraining and fine-tuning scenarios, and for both language and vision tasks.

---

References
[1] L. Balzano, T. Ding, B. D. Hacffele, S. M. Kwon, Q. Qu, P. Wang, Z. Wang, and C. Yaras. An overview of low-rank structures the training and adaptation of large models. arXiv preprint arXiv:2503. 19859, 2025.
[2] Biamonte and V. Bergholm. Tensor networks nutshell. arXiv preprint 1rXivv1708.0000 2017.
[3] Brown, B. Mann, N. Ryder, M. Subbiah, Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances neurul information processing systems, 33:1877-1901, 2020.
[4] J. D. Carroll and J.-J. Chang. Analysis of individual differences in multidimensional scaling via an n- n-way generalization of eckart-young decomposition. Psychometrika, 35(3):283-319 1970.
[5] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Jour- nal of Machine Learning Research, 24(240):1-113, 2023.
[6] A. Cichocki. Era of big data processing: A new approach via tensor networks and tensor decompositions. arXiv preprint arXiv:1403. 2014.
[7] M. Dehghami, Djolonga, B. Mustafa, Padlewski. Heek, Gilmer, A. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[8] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information pro- cessing systems, 27, 2014.
[9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[10] C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psy- chometrika, 1(3):211-22 1936.
[11] R. Feng, K. Zheng, Y. Huang, D. Zhao, M. Jordan, and Z.-J. Zha. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35:33054 -306555--3365, 2022.
[12] T. Garipov, D. Podoprikhin, A. Novikov, and D. Vetrov. Ultimate tensorization: compressing convolutional and layers alike. CoRR, (lb//14,,,0324,, 2016.
[13] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
[14] A. Graves and A. Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37-45, 2012.
[15] A. Han, J. Li, W. Huang, M. Hong, A. Takeda, P. Jawanpuria, and B. Mishra. SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
[16] R. A. Harshman et al. Foundations of the parafac procedure: Models and conditions for an explanatory multi-modal factor analysis. UCLA working papers in phonetics, 16(1):84, 1970.
[17] Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient low rank adaptation of large models. In forty-first International Conference on Machine Learning, 2024.
10

---

[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV 14, pages 630 645. Springer, 2016.
[20] T. He, Z. Zhang, H. Zhang. Z. Zhang. J. Xie, and M. Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 558-567, 2019.
[21] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 2020.
[22] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks. J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprinl arXiv:2203. 15556, 2022.
[23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[24] Z. Hu, L. Wang, Y. Lan, W. Xu, E.P.. Lim, L. Bing, X. Xu, S. Poria, and R. Lee. LLM- adapters: An adapter family for parameter- efficient fine-tuning of large language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5254-5276, Singapore, Dec. 2023. Associa- tion for Computational Linguistics.
[25] Z. Hu, L. Wang, Y. Lan, W. Xu, E.-P. Lim, L. Bing, X. Xu, S. Poria, and R. K.-W. Lee. LLM- adapters: An adapter family for parameter-efficient fine-tuning of large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[26] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint 1405.:1405.38800 2014.
[27] S. R. Kamalakara, A. Locatelli, B. Venkitesh. J. Ba, Y. Gal. and A. N. Gomez. Exploring low rank training of deep neural networks. CoRR, abs/2209. 2022.
[28] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint MxXi:200110886/1 2020.
[29] H. A. Kiers. Towards a standardized notation and terminology in multiway analysis. Journal of Chemometrics: A Journal of the Chemometrics Society, 143)):05--122, 2000.
[30] H. Kim, M. U. K. Khan, and C.-M. Kyung. Efficient neural network compression. In Proceed- ings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12569- 12577, 2019.
[31] Y. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
[32] D. Kressner and C. Tobler. htuckera matlab toolbox for tensors in hierarchical tucker format. Mathicse, EPF Lausanne, page 11, 2012.
[33] T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, M. Paul, C. Pehlevan, C. Re. and A. Raghunathan. Scaling laws for precision. In The Thirteenth International Conference on Learning Representations, 2025.
[34] V. Lebedev, Y. Ganin, M. Rakhuba, I. V. Oseledets, and V. S. Lempitsky. Speeding-up convolu- tional neural networks using fine-tuned cp-decomposition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.

---

[35] J. Li and V. Papyan. Residual alignment: uncovering the mechanisms of residual networks Advances in Neural Information Processing Systems, 36:57660-57713 27771,202..
[36] S. Li, P. Zhang, G. Gan, X. Lv, B. Wang, J. Wei, and X. Jiang. Hypoformer: Hybrid decom- position transformer for edge-friendly neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7056-7068, 2022.
[37] V. Lialin, S. Muckatira, N. Shivagunde, and A. Rumshisky. ReloRA: High-rank training through low-rank updates. In The Twelfth International Conference on Learning Represen- tations, 2024.
[38] V. Lialin, N. Shivagunde, S. Muckatira, and A. Rumshisky, Relora: High-rank training through low-rank updates. arXiv preprint axiiv2337.056695 2023.
[39] L.. Liebenwein, A. Maalouf, D. Feldman, and D. Rus. Compressing neural networks: Towards determining the optimal layer-wise decomposition. Advances in Neural Information Process- ing Systems, 14:5332-5544,, 2021.
[40] Liu, Z.-F. Gao, W. X. Zhao, Z.-Y. Xie, Z.-Y. Lu, and J.-R. Wen. Enabling lightweight fine-tuning for pre-trained language model compression based on matrix product operators. In C. Zong. F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5388-5588 5398, Online, Aug. 2021. Association for Computational Linguistics.
[41] S.-Y. Liu, C.-Y. Wang. H. Yin, P. Molchanov, Y.-C. E. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[42] X. Liu, J. Su, and F. Huang. Tuformer: Data-driven design of transformers for improved gen- eralization or efficiency. In The Tenth International Conference on Learning Representations (ICLR 2022), 2022.
[43] Z. Liu, Zhang, Z. Wang, Z. Yang, Hovland, B. Nicolae, F. Cappello, and Z. Zhang. Cola: Compute-efficient pre-training of Ilms via low-rank activation. arXiv preprint arXiv:2502. 10940, 2025.
[44] Z. Mo, L.-K. Huang, and S. Pan. Parameter and memory efficient pretraining via low-rank riemannian optimization. In The Thirteenth International Conference on Learning Represen- tations.
[45] A. Novikov, D. Podoprikhin, A. Osokin, and Vetrov. Tensorizing neural networks. In Advances in Neural Information Processing Systems 28 (NIPS). 2015.
[46] I. V. Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295-2317) 2011.
[47] X. Ou, Z. Chen, C. Zhu, and Y. Liu. Low rank optimization for efficient deep learning: Making a balance between compact architecture and fast training. Journal of Systems Engineering and Electronics, 35(3):509-555 531, 2023.
[48] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 21(140):1-67, 2020.
[49] O. Russakovsky. J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recog- nition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252 2015.
[50] N. Schuch, M. M. Wolf, F. Verstraete, and J. I. Cirac. Computational complexity of projected entangled pair states. Physical review letters, 98(14): 2007.
[51] I.. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279-311, 1966.
12

---

[52] M. A. O. Vasilescu. Causal deep learning: Causal capsules and tensor transformers. CoRR, abs/2301.00314 2023.
[53] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[54] F. Verstraete, V. Murg, and J. I. Cirac. Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems. Advances in physics, 57(2):143-224, 2008
[55] T. Wang, Z. Dou, C. Bao, and Z. Shi. Diffusion mechanism in residual neural network: theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2):667- 680, 2023.
[56] W. Wang. Y. Sun, B. Eriksson, W. Wang, and Aggarwal. Wide compression: Tensor ring nets. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 9329-9338, 2018.
[57] B. Wu, D. Wang, G. Zhao, L. Deng, and G. Li. Hybrid tensor dccomposition in neural network compression. Neural Networks, 132:309-322 2020.
[58] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[59] Y. Yang. J. Zhou, N. Wong, and Z. Zhang. LORETTA: Low-rank economic rensor-train adapta- tion for ultra-low-parameter fine-tuning of large language models. In K. Duh, H. Gomez, and Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pages 3161-3176, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[60] Z. Yang, Z. Liu, S. Choudhary, X. Xie, C. Gao, S. Kunzmann, and Z. Zhang. Comera: Computing-ane and memory-efficien training via rank-adaptive tensor optimization. Advances in Neural Information Processing Systems, 37:77200-77225, 2024.
[61] X. Yu, T. Liu, X. Wang, and D. Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 7370-7379, 2017
[62] L. Zhang, L. Zhang, Shi, X. Chu, and Li. LoRA-FA: Memory--fficient low-rank adapta- tion for large language models fine-tuning, 2024.
[63] Q Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficien fine-tuning. In The Eleventh International Conference on Learning Representattons, 2023.
[64] Q Zhang, R. Zhang, J. Sun, and Y. Liu. How sparse tal limit perspective. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 91337-91372. Curran Associates, Inc., 2024.
prune deep network: A fundamen-
[65] X. Zhang. J. Zou, K. He, and J. Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38( 19433) 1955, 2016.
[66] J. Zhao, 7. Zhang, B. Chen, 7. Wang, A. Anandkumar, and Y. Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507. 2024.
[67] Q. Zhao, G. Zhou, S. Xie, L. Zhang, and A. Cichocki. Tensor ring decomposition. arXiv preprint arXiv: 1606. 05535, 2016.
13

---

A Hyperparameter
A.1
ViTs Pre-training Configurations
Models
LaX
Rank
Epochs
Base LR
LR decay
Weight decay
Dropout
Warmup
ViT-B
300
3e-3
cosine
0.3
0.1
10
SVD
256
V
TT
336
300
1e-3
cosine
0.3
0.1
10
CoLA
256
Table 9: Training hyperparameter for ViT-B and its low-rank variants on ImageNet-Ik lk.
Models
LaX
Rank
Epochs
Base LR
LR decay
Weight decay
Dropout
Warmup
ViT-L
300
3e-3
cosine
0.3
0.1
10
SVD
256
V
TT
256
300
1e-3
cosine
0.3
0.1
10
CoLA
256
Table 10: Training hyperparameter for ViT-L and its low-rank variants on ImageNet- lk.
Models
Inter-Layer LaX
Intra-Layer LaX
SVD
TT
Tensor Gate
Tensor Gate
CoLA
x
Table 11: LaX gating variants for different low-rank methods.
14

---

Models
Inter-Layer LaX
Intra-Layer LaX
SVD
TT
QKV+MLP
QKV+MLP
CoLA
Table 12: LaX placed layers for different low-rank methods.
Models
Cores
QKV d;
MIP1 d;
MLP2 d;
VIT-B
4
(32,24.24.32)
(32,24,48,64)
(48,64,24332)
VIT-L
4
(32.32,64.64) (32,32,32,3)
(64,64,32,32)
Table Tensor Train Dimension Configuration ViTs.
15

---

A.2
VIT Pre-training Loss Curves
SVD SVD+LaX
Loss
3-
50
100
150 Epoch
200
250
300
Figure Pre-training Loss Curve of SVD and SVD+LaX ImageNet- 1k
7.0
Tensor Train
6.5
Tensor Train+LaX
6.0
5.5
Loss
5.0
4.5
4.0
3.5
3.0
50
100
150 Epoch
200
250
300
Figure Pre-training Loss Curve Tensor Train Tensor Lra on ImageNet-Ik
16

---

COLA
CoLA+LaX
Loss
50
100
150 Epoch
200
250
300
Figure 9: Pre-training Loss Curve of CoLA and CoLA+LaX ImageNet-Ik lk
17

---

A.3
LLMs Pre-training Configurationss
Scales
Models
Lax
Steps
Base LR
LR decay
Weight decay
Warmup
Gradient clipping
2e-3
SVD
60M
10k
cosine
0.01
2k
0.5
6c-3
CoLA
4e-2
Tc-3
SVD
Te-2
130M
20k
4e-3
cosine
0.01
4k
0.5
CoLA
2e-2
le-3
SVD
le-2
350M
60k
cosine
0.01
12k
0.5
3e-3
CoLA
2e-2
8c-4
SVD
3e-3
IB
100k
cosine
0.01
20k
0.5
2e-3
CoLA
le-2
Table 14: perammerreeterss for pre-training SVD and CoLA and their LaX variants for LLaMA-like models from 60M to IB.
The primal hyper- pepprr-prrmeter for LLM pre-training experiments is the learning rate. For small models such 60M and 130M, we typically sweep the scale of 1e-3 for base models and 1e-2 for their LaX variants. The rulc-of-thumb that we empirically found is to choose the largest learning rate that does not cause divergence issues. particular, SVD base models are severely more sensitive to learning rate, and can only afford smaller settings compared to base CoLA. For both SVD and CoLA, LaX offers additional stability that facilitates an order of magnitude larger learning rates. The experience of training smaller models are then adopted for larger scales such as 350M and IB, which continue following the trend that the proper choice of learning rate decreases when model scale increases. For the same scale, we did not find evident that further tuning learning rates are beneficial. Consequently, we adopt same setting when only changing the rank for each scale.
18

---

A.4 LaX-LoRA Fine-tuning Configurations
Hyperparameters (LoRA)
LLaMA-7B
LLaMA-13B
Rank
32
64
64
Dropout
0.0
0.0
Optimizer
AdamW
AdamW
LR
3e-4
3e-4
Scheduler
Linear
Linear
Batch size
16
Accumulation steps
4
Cut off length
256
256
Warmup steps
100
100
Epochs
3
Where
V.K.
Q.K, V.Wp,Down
Table Commonsense Hyperparameter settings for LoRA
LLaMA-7B LLaMA-13B.
Hyperparameters (LaX-LoRA)
LLaMA-7B
LLaMA-13B
Rank
32
32
64
64
Dropout
0.0
0.0
Optimizer
Adam
AdamW
LR
3e-4
3e-4
Scheduler
Linear
Linear
Batch size
16
16
Accumulation steps
4
Cut off length
256
256
Warmup steps
100
100
Epochs
3
Where
Q.K. Down Q.K, V.Up. Identity
V.Up
Where LaX
Q.K, VKV.UUDDowwn
LaX Gate
Identity
Table Commonsense Hyperparameter settings LaX-LoRA on LLaMA-7B LLaMA-13B.
19

---

Hyperparameters (LoRA)
LLaMA-7B
LLaMA-13B
Rank
32
32
OF
64
Dropout
0.0
0.0
Optimizer
A
AdamW
LR
3e-4
3e-4
Scheduler
Linear
Linear
Batch size
16
16
Accumulation steps
4
Cut off length
256
256
Warmup steps
100
100
Epochs
3
Where
Q.K.
Q.K, D.W.
Table 17: Arithmetic Hyperparameter settings for LoRA on LLaMA-7B and LLaMA-13B.
Hyperparameters (LaX-LoRA)
LLaMA-7B
LLaMA-13B
Rank
32
32
64
64
Dropout
0.0
0.0
Optimizer
Adam
AdamW
LR
3c-4
3e-4
Scheduler
Linear
Linear
Batch size
16
16
Accumulation steps
4
Cut off length
256
256
Warmup steps
100
100
Epochs
3
Where
Q,K, Down Q,K, D.u.
Dow.
Where LaX
Q Q.K, Down
LaX Gate
Linear
Linear
Table 18: Arithmetic Hyperparameter settings LaX-LoRA on LLaMA-7B and LLaMA-13B.
20

---

B Additional Experiments
B.1 Where LaX Contributes Mostly
To identify where LaX is most effective, we analyzed the checkpoint from Tab. 2 (rank = 256), focusing the scalar values of the Linear Gates. Since larger gate values indicate stronger reliance on LaX, this allows us to quantify the relative contribution of LaX across components.
LaX Layer
Q
K
Proj
FC1
FC2
3.222
3.825
0.254
0.472
2.966
0,003
2
1.370
1.653
0.406
0.098
2.005
0.004
3
1,596
1.960
0.816
0.037
1.462
0.040
4
1.550
1.522
1.412
0.033
1.360
0.552
5
2.071
2.161
0.847
0.018
1.272
0.026
6
2.184
2.023
1.746
0.004
1.929
0.002
7
0.533
1,210
2.116
0.002
2.038
0.004
8
0.370
1.336
1.895
0.001
1.860
0.005
9
0.259
1.040
1.911
0.002
1.800
0.004
10
0.857
0.955
1.356
0.869
2.332
0.002
11
1.203
2.330
2.983
1.024
2.012
1.477
Table 19:
Scalar values of Linear Gates for each module within LaX layers. Higher values indicate stronger reliance on LaX.
Our analysis reveals several distinct patterns in how LaX contributes across Transformer layers. For the and K projections, LaX plays a stronger role the early layers, followed by a reduction in the middle layers, and then sharp increase in the final layer. In contrast, the V projection shows the opposite trend, i.e., LaX contributes very little in the earlier layers but gradually increases its influence in the later ones. In the attention output projection (Proj), only Layers 1, 10, and 11 exhibit meaningful gate values, indicating that LaX rarely needed in this component. For the FC1 layer in the MLP, LaX contributions are more evenly distributed across the depth, but still follow a pattern of being higher in the initial and final layers. Finally, in FC2, LaX is largely unused throughout the network, except in Layers 4 and 11, where its contribution becomes significant.
B.2 Tensor Train Networks with CoLA-style Activations
We explored how LaX interacts with CoLA-style activations in Tensor Train models by injecting nonlinearity after the down projection. The Tensor Train models and the training script used in this part are identical to Tab. M. Notably, combining LaX with CoLA leads to a synergistic effect, outperforming both standard TT models and SVD-based low-rank baselines:
Model
LaX
CoLA Activation
Test Ace (%)
Tensor Train
x
X
71.11
Tensor Train
/
75.43 (+4.32)
Tensor Train
X
72.60 (+1.49)
Tensor Train
V
77.73 (+6.62)
SVD
X
77.20
Dense VIT
X
76.74
Table 20: How LaX Interacts with CoLA Activation in Tensor Train Networks
B.3 Ablation Study on Intra-LaX and Inter-LaX
To better understand the source performance gains, we separately examined the impact of Inter- LaX and Intra-LaX model accuracy.
We observe that Inter-LaX serves as the primary source performance gains in Tensor Train models. Moreover, when Intra-LaX is applied selectively, MLP blocks benefit more than Attention blocks.
21

---

Model
Inter-LaX Applied To
Intra-LaX Applied To
Test Ace (%)
Tensor Train
71.11
Tensor Train
75.43 (+4.32)
Tensor Train
73.25 (+2.14)
Tensor Train
73.87 (+2.76)
Tensor Train
Attention Only
74.38 (+3.27)
Tensor Train
MLP Only
74.98 (+3.87)
Table 21: Effect of Inter-l.aX and Intra-L.x Model Performance
I LayerNorm
Training Loss
Test Acc
2.486
77.20
x
nan
0.10
Table 22: Effect Removing LayerNorm from LaX
B.4 Ablation Study Normalization
As expected, removing LayerNorm from LaX causes training to fail immediately. LayerNorm plays critical role in stabilizing activation distributions and mitigating gradient explosion/vamishing Without this normalization, activation statistics drift over time, leading to severe gradient instability and ultimately causing the training process collapse.
B.5 Ablation Study Layer Types
We further conducted pre-training experiments to investigate the impact of applying LaX to differ- ent layer types. Specifically, we evaluated this on the pre-training of the SVD-ViT-B model on ImageNet-1K.
Where LaX is Applied
Test Acc (%)
None
75.20
Q only
75.54 (+0.34)
only
75.43 (+0.23)
V only
75.91 (+0.71)
Attention Block only
76.38 (+1.18)
MI.P only
76.09 (+0.89)
Attention Block MLP
77.20 (+2.00)
Table 23: Ablation study of applying LaX to different layer types.
As shown in the Table, applying LaX to the V projection yields the largest performance gain among the individual attention components. When comparing broader structures, applying LaX the entire Attention Block provides slightly greater benefits than applying it only to the MLP. Furthermore, the improvements appear to be additive.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section 1 Empirical validation is in Section 2 and B.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: See Section LLMs pre-training tasks.
We discussed the limitation regarding gating module in
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We release our code link in this paper. All the datasets used are public, and we provide full hyperparameters in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided open access to our code, the anonymized URL is included. All the datasets used are public, and we provide full hyperparameters in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ccm cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ittps:)
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Pre-training large foundation models highly computationally expensive. For instance, single pre-training run of an SVD-based ViT-B model under our setup re- quires approximately 450 A100 GPU hours. Larger models used in this study incur higher costs. Due to computational constraints, we are unable to conduct multiple pre- training runs for all variants.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section B We provide the number of parameters for all experimented models, and also provide the memory in Tab.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We confirm that our research adheres to the NeurlPS Code of Ethics in every respect and preserves anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Our paper proposes new methods for efficient training of large models. The research has the possibility to significantly reduce the training costs of large models and also accelerate the inference.
26

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our research didnt release any data or models at such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Yes, they are properly credited, mentioned, and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, paperswi thcode contaataest has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: Our work introduces no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We confirm that the proposed method in our paper does not involve L.Mss.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy ottp:e/rDeeeriioc.ccc///o/eeeeep for what should or should not be described.
29


Reviews:
[{"review_id": "NkyYgOaffj", "reviewer": "Reviewer_bjrv", "summary": "The authors introduce Latent Crossing (LaX), a small addition to many low-rank modules/PEFT methods that improves performance across the board. LaX involves equipping the latent space of low-rank modules with residual connections, followed by layer normalization. Before applying the residual connection, however, one of four proposed gating layers is applied to the low-rank representation. The proposed layers are lightweight in terms of computation and memory complexity, especially for the PEFT setting, where these considerations are less important. To validate the effectiveness of their method, the authors provide extensive experiments across ViT pre-training, LM pre-training, and LLM fine-tuning. The results generally show that the author’s proposed method improves performance beyond.", "questions": "- In Table 2, the Identity gate has the same number of parameters as the Base model, but the authors state on line 164 that they postpend Layer Normalization to each residual connection. Don’t the layer norms have parameters? \n\n- In Table 4, why does CoLa + LaX outperform Full-rank?\n\n- Do you think the result from table 4 would hold if a different optimizer, e.g., Muon, were used, which might be able to take advantage of the full rank matrix?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- The paper is well written and easy to follow.\n- The tables are clear and provide all the necessary details\n- The method outperforms all baselines tested within the author’s study.\n- The authors provide training curves for their method in the appendix, which is appreciated. \n\n**Weaknesses**:\n- The authors show a surprising result: pre-training with CoLa + LaX outperforms Full-rank pre-training (see Table 4), but this is not explained in the text, leading me to wonder if the hyperparameters for Full-rank were well tuned?\n- In section A.3 of the appendix, the authors state that they sweep at the scale of 1e-3 for base models and 1e-2 for LaX variants. For reproducibility, it would be best to mention how many HP configurations were tried for each method and ideally the values shown.\n- When browsing related literature, I found a closely related work published last year that should be discussed [1]. It does not overlap with the author's method, but could complement it and is close enough to warrant discussion. \n\n\n\nWhen browsing related literature, I also found [2], a closely related concurrent work which might be of use to the authors, but that is too new to require evaluation.\n\n---\n**References**\n\n[1][ResLoRA: Identity Residual Mapping in Low-Rank Adaption; ACL 2024]\n\n[2][LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuning.]", "was_revised": true, "final_justification": "I have maintained my score. I recommend accepting the paper.\n\n**Issues resolved**\n- The reason for CoLa + LaX outperforming Full-rank pre-training.\n- More information regarding hyperparameter selection choices.\n- Discussion of an additional related work.", "rebuttal": "We really appreciate the reviewers’ thoughtful review and recognition of our contributions. Hope our point-wise responses would further clear things out:\n\n### **1. Clarification of why CoLA+LaX outperforming full-rank baselines**\n\n---\n\nThanks for your comments. We confirm that **CoLA + LaX outperforms the dense model in both Vision** (Table 1, we train both full-rank and low-rank models using configurations from [1]) and **Language model pre-training** (Table 5, we directly compare LaX with baselines reported in related work [Galore, SLTrain, CoLA, LORO]). According to the CoLA paper, CoLA improves architectural efficiency by replacing an arbitrary linear operator by a composition of linear-nonlinear-linear operator, which is theoretically capable of approximating any continuous function, powered by the universal approximation theorem. We speculate that such a deeper, nested structure might empower CoLA with a higher capacity while using fewer parameters, and LaX further boosts its performance, therefore surpassing full-rank models.\n\n### **2. Clarification of the parameters introduced by LaX’s layer norm**\n\n---\n\nThanks!  We would like to clarify that the LayerNorm modules introduced in LaX indeed introduce parameters to the base model. However, **the increase is negligible** — only $2 r$ parameters per layer, $\\frac{1}{d}$ of the base model. Given that $d$ is normally very large, the overall increase is trivial and does not visibly change the reported parameter counts due to rounding. We appreciate the opportunity to clarify this point and will consider explicitly noting it to avoid confusion.\n\n### **3. Do authors think CoLA+LaX can still outperform full-rank models if using other optimizers, such as Muon?**\n\n---\n\nThank you for your insightful comment! It is possible that a fine-tuned optimizer might further improve the accuracy of either CoLA+LaX, full-rank models or the both. In this paper, we demonstrate the performance boost of LaX while using mainsteam optimizer such as Adam. Definitely many other research opportunities can be inspired by considering the combinations of various optimizers and architectures.  \n\n### **4. Discussion regarding ResLoRA/LoR2C.**\n\n---\n\nWe thank the reviewer for highlighting these relevant and recent works. Both are targeting the **gradient vanishing/explosion challenges in PEFT**. We acknowledge their potential relevance and see **promising opportunities for incorporating their insights into future extensions** of our approach.\n\n- **ResLoRA** tackles gradient propagation issues in LoRA by introducing **shortcut connections at various positions within the adapter structure**, supported by a **mathematical analysis from an optimization perspective**. **LoR2C** mitigates gradient vanishing by introducing **residual connections across entire blocks with an extra LoRA block.**\n- In LaX, we currently apply LaX between the **same type of layers** across transformer blocks, guided by a prior on semantic alignment. However, their work suggests that exploring **LaX variants that span different connection types** is possible, particularly in the context of **Tensor Train models.** Owing to their richer and more structured latent spaces, Tensor Train models offer greater flexibility for experimenting with alternative LaX connection patterns. We are looking forward to exploring this in our future research.\n- Moreover, we are also deeply interested in the **theoretical understanding of LaX**. We believe that their optimization-based analyses could inspire new theoretical analyses to explain LaX.\n\nWe thank the reviewer once again for pointing out these valuable works, which may serve as complementary directions to LaX in our future research efforts!\n\n**References:**\n\n[1] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020."}, {"review_id": "jgHleBgWES", "reviewer": "Reviewer_QiUF", "summary": "This paper proposes Latent Crossing (LaX), a simple plug-and-play module that enhances low-rank training methods by enabling information flow between low-rank subspaces through residual connections. The authors show that LaX consistently improves performance across several low-rank methods (SVD, Tensor Train, CoLA) on both vision and language tasks, with minimal parameter overhead.", "questions": "- Could you run multiple experiments with different seeds to increase the number of training curves in Figures 6, 7, 8, 9 and to include confidence bounds in Tables 2, 6, 7, 8?\n- Have you investigated where in the network LaX provides the most benefit?\n- Can you provide any theoretical insight into why LaX improves optimization/generalization in low-rank settings?\n- How does LaX interact with other techniques for improving low-rank training? (e.g. Q-LoRA, LoRA-FA, ...)", "limitations": "- **Modest improvements in some cases**: While improvements are consistent, they can be modest (e.g. +0.27% for CoLA on 130M LLM). The cost-benefit trade-off isn't always clear. \n- **Dependency on baseline quality:** LaX appears to help weaker methods more (e.g. Tensor Train) but provides diminishing returns on already strong methods like CoLA.\n- **Gate selection**: the optimal gate type (Identity/Linear/Tensor/Dense) appears to be task and model dependent, requiring empirical search rather than principled selection.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- **Clear motivation and simple solution**: the authors propose a new and intuitive method to improve the training of low-rank models, taking inspiration from ResNet-style residual connections.\n- **Comprehensive experimental validation**: the validation spans multiple domains (vision and language), model scales (up to 1B parameters), and different low-rank methods. Results show consistent improvements, with notable gains for Tensor Train methods on ViTs (+4.32%).\n- **Practical applicability**: LaX works as a drop-in module for existing low-rank methods, making it easy to adopt in practice.\n- **Reproducibility**: the authors will release the code with the paper and have included the hyperparameters of the experiments in the Appendix.\n\n\n# Weaknesses\n- **Novelty**: while LaX proves to be effective, the novelty of the method is rather limited since the core contribution is conceptually simple: adding residual connections in the latent space of the low-rank modules.\n- **Limited analysis of when LaX helps most**: while the results show LaX improves the performance the most on weaker baselines (Tensor Train), there's insufficient analysis of what factors determine the magnitude of the improvement.\n- **Lack of theoretical analysis**: the paper provides no theoretical justification as to why LaX improves the training of low-rank models. \n- **Statistical significance**: the authors just train one model for each setting, and do not include confidence intervals. The small improvement over the baselines could be in part explained by the inherent randomness of the training process. \n- **Missing ablations**: the paper could benefit from more ablation studies, such as the effect of applying LaX to different layer types or the impact of normalization in LaX.\n\n\n\n## Citations\n- The LaX Gates are tightly related to the [*Highway Networks*](https://arxiv.org/abs/1505.00387) , but the authors fail to cite this work.\n## Experimental concerns\n- The ViT experiments use relatively small models. Do gains persist for bigger models like ViT-H/Vit-G?\n- Missing comparisons with other modern LoRA variants, and lack of exploration of orthogonal benefits with these approaches\n\n\n\n## Minor issues\n- line 113: incorrect grammar \"our motivation is **that**, can we improve the performance...\"\n- line 120: \"requiring no additional parameters or computation overhead\". Isn't LaX increasing the FLOPs in the forward pass by $(L - 1)h_{in}$, where $L$ is the number of layers? Also, $h_{i-1}$ needs to stay in memory for the computation of the next layer. This is, in fact, mentioned in lines 157, 178.\n- In Table 6, the choice of the cross/check for the identity gate and dense gate usage is unintuitive.", "was_revised": true, "final_justification": "The authors introduce LaX, a method that introduces residual connections between the LoRA spaces.\n\nAlthough the presented method is extremely simple (a strength in its easiness to deploy), the authors show that LaX is a consistent improvement over the baselines, especially when taking into account the minimal overhead added.\n\nThe authors have addressed the weaknesses we exposed, and we are pleased with the additions to the revision of the manuscript. We recommend accepting the paper.", "rebuttal": "We are very grateful with the reviewer’s insightful review. Hope our point-wise responses would further clear things out:\n\n### **1. Confidence bounds and more training curves.**\n\n---\n\nWe agree that including confidence bounds would provide additional insights. Due to the long pre-training time and limited rebuttal period, running multiple trials for all models (which also include related methods such as ReLoRA, GaLore, SLTrain, LORO, etc) to estimate variance is infeasible. However, to provide a valid proof of concept, we have ran 60M LLaMA CoLA+LaX with 5 random seed, and estimate the 95% confidence interval as [33.15, 33.36].\n\n### **2. Where LaX benefits mostly.**\n\n---\n\nThank you for the insightful question. To assess where LaX is most effective, we analyzed the checkpoint from Table 2 (rank=256), focusing on the **scalar values of the Linear Gates**—with higher values indicating greater contribution from LaX.\n\n| LaX Layer | Q | K | V | Proj | FC1 | FC2 |\n| --- | --- | --- | --- | --- | --- | --- |\n| 1 | 3.222 | 3.825 | 0.254 | 0.472 | 2.966 | 0.003 |\n| 2 | 1.370 | 1.653 | 0.406 | 0.098 | 2.005 | 0.004 |\n| 3 | 1.596 | 1.960 | 0.816 | 0.037 | 1.462 | 0.040 |\n| 4 | 1.550 | 1.522 | 1.412 | 0.033 | 1.360 | 0.552 |\n| 5 | 2.071 | 2.161 | 0.847 | 0.018 | 1.272 | 0.026 |\n| 6 | 2.184 | 2.023 | 1.746 | 0.004 | 1.929 | 0.002 |\n| 7 | 0.533 | 1.210 | 2.116 | 0.002 | 2.038 | 0.004 |\n| 8 | 0.370 | 1.336 | 1.895 | 0.001 | 1.860 | 0.005 |\n| 9 | 0.259 | 1.040 | 1.911 | 0.002 | 1.800 | 0.004 |\n| 10 | 0.857 | 0.955 | 1.356 | 0.869 | 2.332 | 0.002 |\n| 11 | 1.203 | 2.330 | 2.983 | 1.024 | 2.012 | 1.477 |\n\n Our analysis reveals several interesting observations:\n\n- For the **Q and K projections**, LaX contributes more in the first half of the layers than the latter half, with a sharp increase in gate values observed in the final layer.\n- The **V projection** exhibits a distinct pattern, where LaX contributes less in the earlier layers compared to the later ones.\n- In the **attention output projection layer**, only Layers 1, 10, and 11 show relatively higher gate values, while the remaining layers contribute minimally.\n- For **FC1 in the MLP**, the contribution of LaX is more evenly distributed across layers, but still follows the general trend of higher values in the early and final layers.\n- For **FC2 in the MLP**, only Layers 4 and 11 show meaningful gate values, with others being close to zero.\n\nWe thank the reviewer once again for their insightful observations, which may point toward a promising future direction—**a sparse variant of LaX** that selectively activates only the most impactful layers.\n\n### **3. How does LaX interact with other techniques for improving low-rank training?**\n\nWe fully agree that exploring the interaction between LaX and other techniques for low-rank training is a highly promising direction. In line with your suggestion regarding Q-LoRA, we’d like to share some preliminary results from an ongoing work investigating the integration of **LaX with quantization methods**.\n\n- We pre-train ViT-B and its CoLA + LaX variant (w/ Tensor Gate) at FP32 on Imagenet-1k, then we perform a series of simple post-training quantization (Optimum Quanto from Huggingface for linear quantization) on the resulting model,\n\n| Precision | Model |  # Params (M) | Test Acc |\n| --- | --- | --- | --- |\n| FP32  | ViT-B | 86.56  | 76.74 |\n| FP32  | CoLA | 44.17 | 76.04 |\n| FP32  | CoLA + LaX | 44.24   | 77.84 |\n| Int8  | CoLA + LaX | 44.24   | 77.70 |\n| Int4  | CoLA + LaX | 44.24   | 76.57 |\n- As shown, **LaX integrates effectively with quantization techniques**. Notably, Int4 CoLA + LaX achieves performance comparable to the full-precision dense ViT-B model, while achieving a **16× reduction in model size** (**2× in parameters** and **8× in precision**).\n\n### **4. Ablation for applying LaX to different layer types and for Normalization.**\n\n---\n\nWe conducted the suggested ablation experiments specifically on the **pre-training of the SVD-ViT-B model** on **ImageNet-1K**. \n\n- Ablation for applying LaX to different layer types:\n\n| Where LaX | Test Acc  |\n| --- | --- |\n| - | 75.20 |\n| Q | 75.54 (+0.34) |\n| K | 75.43 (+0.23) |\n| V | 75.91 (+0.71) |\n| Attn. Block | 76.38 (+1.18) |\n| MLP | 76.09 (+0.89) |\n| Attn. Block + MLP | 77.20 (+2.00) |\n\nAs shown, applying LaX to the V-proj yields the most notable gains among attention layers. Between MLP and Attention Blocks, the latter benefits slightly more. Notably, the benefit of LaX appears additive, with multiple applications leading to cumulative improvements.\n\n- Ablation for Normalization:\n\nRemoving LayerNorm (LN) from LaX causes training to **fail from the start, as expected**. LN stabilizes activation distributions and prevents gradient instability; without it, training collapses due to shifting statistics.\n\n| LN | Training Loss | Test Acc |\n| --- | --- | --- |\n| w/ | 2.486 | 77.20 |\n| w/o | nan | 0.1 |\n\n### **5. Dependency on baseline quality. Modest improvements in some cases. The cost-benefit trade-off isn't always clear.**\n\n---\n\nThank you for your valuable comments. As discussed in the paper:\n\n- **LaX is specifically designed to recover the performance lost in low-rank models relative to their full-rank counterparts.** Naturally, when starting from a strong low-rank baseline, such as CoLA, the improvement is less significant. Nonetheless, the integration of LaX consistently enables CoLA to surpass the full-rank baseline in both vision and language modeling tasks.\n- **LaX introduces minimal computational overhead**, even with the most expensive practical variant. As noted in the practical guideline section, **Tensor Gate** is the most costly one in practice. The **relative FLOPs overhead**, compared to the baseline low-rank model,\n\n| Gating Type | ViT-B | ViT-L |\n| --- | --- | --- |\n| **Tensor Gate** | 1/48 | 1/64 |\n\n### **6. The optimal gate type appears to be task and model dependent, requiring empirical search rather than principled selection.**\n\n---\n\nWe appreciate the reviewer’s concern and agree that different gate types yield different trade-offs. However, we would like to clarify that the core motivation of the gate design in LaX is to **align latent dimensions across or within low-rank layers,** which is not mandatory in many cases, especially with SVD and CoLA. Without gating, **LaX still yields consistent gains across various domains,** as shown in Table 2 and Table 6.\n    \n### **7. Novelty/theoretical analysis.**\n\n---\n\nThank you for the thoughtful comment. While LaX incorporates residual connections, its **core novelty lies in addressing a unique challenge in low-rank adaptation**. As stated in the paper, our primary motivation is to **enhance the capacity of low-rank layers without increasing their matrix rank**.\n\n- Importantly, the **effective rank $\\mathrm{sr}=\\frac{\\mathbf{||W||}^2_F}{||\\mathbf{W}||^2_2}$** of a layer is often lower than its matrixl rank. This observation opens up an opportunity: improve the effective rank in a lightweight and principled way — latent feature fusion.\n- We support this intuition with both **theoretical justification** under assumptions and **empirical evidence** during the actual training process, demonstrating that LaX increases the effective rank and leads to consistent performance gains. While the mechanism may resemble residual connections in form, LaX is functionally and conceptually distinct in its purpose. Specifically (notion is the same as in our paper),\n\n$$\n\\mathrm{sr}(\\mathbf{W}^{\\text{LaX}}) \\approx \\left( \\frac{\\sqrt{2d_{\\text{out}}} + \\sqrt{2d_{\\text{in}}}}{\\sqrt{d_{\\text{out}}} + \\sqrt{2d_{\\text{in}}}} \\right)^2 \\mathrm{sr}(\\mathbf{W}^{\\text{SVD}}) > \\mathrm{sr}(\\mathbf{W}^{\\text{SVD}})\n$$\n\n- In the simulation, we train a simple SVD/LaX layers with identical encoders and decoders on the CIFAR-10. Due to rebuttal constraints, we are only able to present how the **amplification ratio** $\\frac{\\mathrm{sr}(\\mathbf{W}^{\\text{LaX}})}{\\mathrm{sr}(\\mathbf{W}^{\\text{SVD}})}$**and training loss** **evolve during training** in tabular format. We can see that **SVD w/ LaX always has a higher effective rank than w/o LaX**.\n\n| Epoch | 1 | 20 | 40 | 60 | 80 | 100 |\n| --- | --- | --- | --- | --- | --- | --- |\n| Amplification Ratio  | 114.12% | 111.69% | 116.42% | 108.22% | 108.75% | 110.00% |\n| Loss w/o LaX | 2.05 | 1.42 | 1.31 | 1.21 | 1.13 | 1.11 |\n| Loss w/ LaX | 2.05 | 1.39 | 1.23 | 1.10 | 1.00 | 0.97 |\n\n### **8. Bigger models like ViT-H/G.**\n\n---\n\nWe agree that studying the **scaling behavior of LaX combined with low-rank models** is an important direction. Due to budget constraints, we were unable to pre-train many large models. But we have included **four pre-training results on 1B-scale LLMs** in Table 5, which serve as meaningful references for a scale positioned between ViT-H (\\~650M) and ViT-G (1.8B). At 1B scale, LaX still consistently brings benefits to all the low-rank models.\n\n### **9. More LoRA-like variants.**\n\n---\n\nFor low-rank fine-tuning, we adopt LoRA as the base method because it is the foundational approach in this domain and we aim to isolate the impact of LaX itself, just as in our pre-training experiments. However, in order to explore LaX’s orthogonal benefits when combining with other modern LoRA variants, we added addtional experiments on **DoRA** [1], and show results below. Notably, **DoRA + LaX consistently performs better or same than plain DoRA in 7 out of 8 tasks.**\n\n| Method | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DoRA | 69.9 | 82.1 | 78.8 | 83.2 | 80.6 | 80.6 | 65.1 | 78.4 | 77.3 |\n| DoRA + LaX | 69.9 (+0.0) | 82.1 (+0.0) | 77.9 (-0.9) | 87.6 (+4.4) | 81.1 (+0.5) | 81.7 (+1.1) | 66.2 (+1.1) | 78.4 (+0.0) | 78.1 (+0.8) |\n\nThe experiment is conducted on Llama-7B with an adapter rank=16, using the same hyperparameters as in [1].\n\n**References:**\n[1] Dora: Weight-decomposed low-rank adaptation."}, {"review_id": "YmfcDKl8iH", "reviewer": "Reviewer_1Hwt", "summary": "This paper introduces Latent Crossing (LaX), a lightweight, plug-and-play module that enhances low-rank training for foundation models like ViTs and LLMs. By injecting residual connections between latent features across or within low-rank layers, LaX improves cross-subspace information flow and effectively expands model capacity without increasing parameter count. It also includes flexible gating mechanisms to manage dimension mismatches. Experiments across pre-training and fine-tuning settings demonstrate consistent performance gains with negligible computational overhead.", "questions": "N/A", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Lightweight and parameter-free design with negligible computational overhead.\n2. Compatible with various low-rank methods and architectures (e.g., ViT, LLM).\n3. Demonstrates consistent performance gains across pre-training and fine-tuning tasks.\n\nWeakness\n1. LaX primarily integrates residual fusion into the LoRA setting, with minimal architectural changes and limited theoretical advancement in low-rank modeling.\n2. The method is only evaluated under fixed-rank settings; its compatibility with adaptive or dynamic-rank methods (e.g., AdaLoRA) remains unexplored.\n3. Performance gains on LLM fine-tuning tasks are relatively modest (see Tables 7–8), and the paper lacks direct comparison with other strong PEFT baselines in this setting.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. We would like to take this opportunity to clarify that **LaX is not designed solely for Adapters**; rather, it is a **general mechanism for enhancing low-rank models**, applicable to both **low-rank foundation model pre-training from scratch** and **adapter-based fine-tuning** scenarios. We hope that our point-by-point responses below will address any remaining concerns and further clarify the scope and impact of our contributions.\n\n### **1. Clarification of LaX's broader impact and theoretical justification**\n\n---\n\nThank you for the helpful comments. We provide clarifications on the two points raised:\n\n- **We address  a major challenge in low-rank pre-training.** Pre-training low-rank foundation models is fundamentally different from LoRA-style fine-tuning. Rather than inserting adapters, these models **replace dense layers entirely with low-rank layers** and are trained **from scratch**. We would like to emphasize that **enabling effective low-rank model pre-training is a key contribution of LaX**. While such low-rank models typically suffer from performance degradation due to limited capacity, LaX consistently helps **recover**, and **even surpass**, the performance of full-rank models (Table 1, 2, 5, 6), using **2–3× fewer parameters** overall and with negligible computational overhead.\n- **Theoretical analysis**: As stated in the paper, our primary motivation is to improve the capacity of low-rank layers *without explicitly increasing their actual matrix rank*. However, we do offer a theoretical justification, complemented by practical evidence, showing that **LaX increases the *effective rank[1][2][3][4]*** after being applied to the SVD layer. Specifically:\n\n$$\n\\mathrm{sr}(\\mathbf{W}^{\\text{LaX}}) \\approx \\left( \\frac{\\sqrt{2d_{\\text{out}}} + \\sqrt{2d_{\\text{in}}}}{\\sqrt{d_{\\text{out}}} + \\sqrt{2d_{\\text{in}}}} \\right)^2 \\mathrm{sr}(\\mathbf{W}^{\\text{SVD}}) > \\mathrm{sr}(\\mathbf{W}^{\\text{SVD}})\n$$\n\nThis expression is derived from concentration results of the Frobenius and spectral norms. In the simulation, we train a simple SVD/LaX layers with identical encoders and decoders on the CIFAR-10 dataset. Due to rebuttal constraints, we are only able to present how the **amplification ratio** $\\frac{\\mathrm{sr}(\\mathbf{W}^{\\text{LaX}})}{\\mathrm{sr}(\\mathbf{W}^{\\text{SVD}})}=\\left( \\frac{\\sqrt{2d_{\\text{out}}} + \\sqrt{2d_{\\text{in}}}}{\\sqrt{d_{\\text{out}}} + \\sqrt{2d_{\\text{in}}}} \\right)^2$**evolves during training** in tabular format. We can see that **SVD w/ LaX always has a higher effective rank than w/o LaX**.\n\n| Epoch | 1 | 20 | 40 | 60 | 80 | 100 |\n| --- | --- | --- | --- | --- | --- | --- |\n| Amplification Ratio | 114.12% | 111.69% | 116.42% | 108.22% | 108.75% | 110.00% |\n| Loss w/o LaX | 2.05 | 1.42 | 1.31 | 1.21 | 1.13 | 1.11 |\n| Loss w/ LaX | 2.05 | 1.39 | 1.23 | 1.10 | 1.00 | 0.97 |\n\n\n### **2. Comparison with other strong baseline on LLM fine-tuning**\n\n---\n\nThank you for pointing this out. We agree that integrating LaX with more advanced methods is a meaningful direction.  \n\n- We would like to clarify that **we intentionally chose standard mainsteam methods in both pre-training and fine-tuning** in order to **isolate and clearly demonstrate the effect of LaX** without interference from additional techniques.\n- Regarding other PEFT baselines/adaptive rank methods, LaX is technically compatible with many existing approaches. For example, in AdaLoRA, the LaX Gate can align low-rank spaces with varying shapes. Notably, while existing adaptive low-rank pre-training methods typically fall short of full-rank performance, LaX enables vanilla low-rank models to surpass the full-rank baseline. Due to the limited time window,  we focused on one strong PEFT baseline, **DoRA** [5], and show the comparative results below. **DoRA + LaX consistently performs better or same than plain DoRA in 7 out of 8 tasks.**\n\n| Method | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DoRA | 69.9 | 82.1 | 78.8 | 83.2 | 80.6 | 80.6 | 65.1 | 78.4 | 77.3 |\n| DoRA + LaX | 69.9 (+0.0) | 82.1 (+0.0) | 77.9 (-0.9) | 87.6 (+4.4) | 81.1 (+0.5) | 81.7 (+1.1) | 66.2 (+1.1) | 78.4 (+0.0) | 78.1 (+0.8) |\n\nThe experiment is conducted on Llama-7B with an adapter rank=16, using the same hyperparameters as in [5].\n\n**References:**\n\n[1] Vershynin R. High-dimensional probability: An introduction with applications in data science[M]. Cambridge university press, 2018.\n\n[2] Tropp J A. An introduction to matrix concentration inequalities[J]. Foundations and Trends® in Machine Learning, 2015, 8(1-2): 1-230.\n\n[3] Rudelson M, Vershynin R. Sampling from large matrices: An approach through geometric functional analysis[J]. Journal of the ACM (JACM), 2007, 54(4): 21-es.\n\n[4] Ipsen I C F, Saibaba A K. Stable rank and intrinsic dimension of real and complex matrices[J]. arXiv preprint arXiv:2407.21594, 2024.\n\n[5] Liu S Y, Wang C Y, Yin H, et al. Dora: Weight-decomposed low-rank adaptation[C]//Forty-first International Conference on Machine Learning. 2024."}, {"review_id": "7hJmQQ7eML", "reviewer": "Reviewer_CLBm", "summary": "This paper aims to address the common bottleneck in low-rank compressed deep neural models, where lower-rank models lead to lower performance, but higher-rank models increase the computational burden. To overcome this, the authors propose the LaX method as a plug-in module to enable information flow between low-rank subspaces, inspired by the Residual Mechanism. Moreover, the paper introduces the LaX Gate to align latent spaces with different dimensions. In the experiment, the paper evaluates the effectiveness of LaX through a set of low-rank pre-training and fine-tuning experiments.", "questions": "How LaX works on tensor decomposition is unclear, and it lacks a formal description, like its matrix counterpart. Specifically, regarding the Intra-Layer LaX:\n\n• In Figure 2(C), the input is a 2-order tensor, which can first contract with either $C^0$ or $C^1$. What is the contraction order? Assuming the input is a matrix of size $m \\times n$ and contracts with $C^0$ first, then after the contraction, will the rank $r_0$ be boosted to $nr_0$?\n\n• In Row 131, the authors mention that a series of latent features will be produced in order: $C^{0}x$, $C^{0}C^{1}x$. According to this description, it seems like the input is just a vector. However, in many cases, the input should also be tensorized, so it should contract with $C^{0}$, $C^{1}$, and $C^{2}$ simultaneously. Therefore, I do not understand how the Intra-Layer LaX is performed, especially when the input is a high-order tensor and contracts with $C^{0}$, $C^{1}$, and $C^{2}$.\n\n• Most of the experimental results are used to support how LaX works on the matrix decomposition-based compression model, with only Table 1 demonstrating how LaX works on the tensor decomposition-based model. The work would benefit from more experiments demonstrating how LaX works on tensor decomposition-based methods.\n\n• As an extension to question 3, tensor decomposition has a more complex structure than matrix decomposition. If the paper could include some additional experiments to better understand how LaX works on tensor decomposition—specifically the pros and cons, and when it is most useful or fails—it would greatly enhance the impact of the paper.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**:\n\n• Low-rank methods for compressing deep neural networks have proven to be an effective way to improve the efficiency of DNNs. However, the performance gap between the full model and low-rank models has always been an important problem to solve. This paper proposes a simple yet effective method that is widely suitable for many different types of low-rank models. The motivation drawn from the Residual Mechanism is interesting, and the implementation is original.\n\n### **Weaknesses**:\n\n• Some parts of how the model works are not described clearly enough, and the paper lacks further experiments to deepen our understanding of the proposed LaX module. Please refer to the questions for more details.", "was_revised": true, "final_justification": "This paper proposes a novel residual-driven mechanism to improve the performance of low-rank compression models. In the rebuttal phase, the authors provided additional explanations that helped clarify the paper’s formulation and addressed most of my concerns. Therefore, I have increased my confidence score. However, I find the experimental validation still somewhat limited.", "rebuttal": "Thank you for your meaningful comments! We sincerely appreciate your thoughtful feedback and hope that our responses below will help address your questions:\n\n### **1. Clarification regarding tensor operations**\n\n---\n\nThank you for raising this important point! We will include a more detailed explanation in the revised version. Let us walk through a simple example to clarify here: consider an input $\\mathbf{x} \\in \\mathbb{R}^{B \\times d}$, where $B$ is the batch size. We reshape $\\mathbf{x}$ into $\\mathbb{R}^{B \\times d_0 \\times d_1}$ to match the first two TT-cores: $\\mathbf{C}^0 \\in \\mathbb{R}^{1 \\times d_0 \\times r_0}$, $\\mathbf{C}^1 \\in \\mathbb{R}^{r_0 \\times d_1 \\times r_1}$, within a 4-core Tensor Train: $\\{\\mathbf{C}^0, \\mathbf{C}^1, \\mathbf{C}^2, \\mathbf{C}^3\\}$ setting.\n\n- The TT output $\\mathbf{y}_{\\text{TT}} = \\mathbf{xC^0C^1C^2C^3}$ consists of two stages: (1) **down-projection** via $\\mathbf{C}^0, \\mathbf{C}^1$ and (2) **up-projection** via $\\mathbf{C}^2, \\mathbf{C}^3$. Contraction order within each stage is flexible, but for simplicity, we follow the natural order (0→1→2→3),i.e., we first contract $\\mathbf{x}$ with $\\mathbf{C}^0$, eliminating the $d_0$ mode and producing $\\mathbf{xC^0} \\in \\mathbb{R}^{B \\times d_1 \\times r_0}$, then we contract $\\mathbf{C}^1$ with $\\mathbf{xC^0}$, eliminating $d_1$ and $r_0$ and producing $\\mathbf{xC^0C^1} \\in \\mathbb{R}^{B \\times r_1}$.\n- Each core $\\mathbf{C}^i$ is contracted with intermediate output $\\mathbf{xC^0\\cdots C^{i-1}}$, so there is no case where multiple cores are contracted with the same $\\mathbf{x}$ simultaneously.\n- To illustrate **intra-layer LaX**, we continue from $\\mathbf{xC^0C^1} \\in \\mathbb{R}^{B \\times r_1}$ and contract it with $\\mathbf{C}^2$, yielding $\\mathbf{xC^0C^1C^2} \\in \\mathbb{R}^{B \\times d_2 \\times r_2}$. If the shapes align (i.e., $d_1 r_0 = d_2 r_2$), we can fuse $\\mathbf{xC^0}\\in \\mathbb{R}^{B \\times d_1 \\times r_0}$  with $\\mathbf{xC^0C^1C^2}$ directly to build a residual path. If not, we use a gating mechanism to align and merge the features.\n\n### **2. Additional experiments and discussions on LaX with tensor decomposition**\n\n---\n\nThank you for your insightful feedback. We have conducted additional experiments specifically targeting **Tensor Train** models, which we summarize below and plan to include in the revised version.\n\n**2.1 LaX with CoLA Activations in Tensor Train Models**\n\nWe explored how LaX interacts with **CoLA-style activations** in TT models by injecting a nonlinearity after the down projection result $\\mathbf{xC^0C^1}$. This extension is conceptually natural. Notably, combining **LaX with CoLA** leads to a **synergistic effect**, outperforming both standard TT models and SVD-based low-rank baselines:\n\n| Model | LaX | CoLA Activation | Test Acc (%) |\n| --- | --- | --- | --- |\n| Tensor Train | ✗ | ✗ | 71.11 |\n| Tensor Train | ✓ | ✗ | 75.43 (+4.32) |\n| Tensor Train | ✗ | ✓ | 72.60 (+1.49) |\n| Tensor Train | ✓ | ✓ | **77.73 (+6.62)** |\n| SVD | ✓ | ✗ | 77.20 |\n| Dense ViT | ✗ | ✗ | 76.74 |\n\nThis result may suggest that LaX can **amplify the benefits of CoLA-style activations** in tensor format models, significantly boosting the weakest base model to be even stronger than the full-rank model.\n\n**2.2 When Intra-LaX Helps: Pros and Cons**\n\nUnderstanding where **Intra-LaX** contributes most is crucial, especially for **TT-LaX models with more cores**. To investigate this, we conducted a series of **ablation experiments**, where **✓** indicates that **LaX is applied to all layer types**:\n\n| Inter-LaX Applied To | Intra-LaX Applied To | Test Acc (%) |\n| --- | --- | --- |\n| ✗ | ✗ | 71.11 |\n| ✓ | ✓ | 75.43 (+4.32) |\n| ✗ | ✓ | 73.25 (+2.14) |\n| ✓ | ✗ | 73.87 (+2.76) |\n| ✓ | Attention Only | 74.38 (+3.27) |\n| ✓ | MLP Only | **74.98 (+3.87)** |\n\nThese results reveal two key findings:\n\n1. **Inter-LaX plays a relatively primary role** in performance gains for TT models.\n2. When applying Intra-LaX selectively, **MLP blocks benefit more than Attention blocks.**\n\nA more granular study, e.g., dissecting Q, K, V projections, is still needed to pinpoint where LaX is most critical. We plan to further investigate this in future work."}]

Author rebuttal:

Meta-review: This submission proposes a new method to improve the training of low-rank models, taking inspiration from ResNet-style residual connections. Reviewers found the method intuitive and the experiments demonstrate the method's efficacy. The weaknesses of the paper are that the method is not very novel (i.e., residual connections are a common idea), it lacks theoretical motivation, and the method is only compared against LoRA as a baseline. 

Ultimately, I think a simple idea that works is worth publishing, and the only negative reviewer was not engaged at all and did not identify any weaknesses the other reviewers missed. I would strongly encourage the authors to include other baseline PEFT methods that have been shown to improve on LoRA. I would also encourage running the method on multiple random seeds and reporting the variance.
Decision: Accept (poster)