Title: LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Efficient Pre-training

Efficient Fine-tuning

Adapters

Low-rank Training

Vision Transformers

Large Language Models
Abstract: Training foundation models such as ViTs and LLMs requires tremendous computing cost. Low-rank matrix or tensor factorization offers a parameter-efficient alternative, but often downgrades performance due to the restricted parameter space. In this work, we introduce ${\textbf{Latent Crossing (LaX)}}$ -- a simple yet effective plug-and-play module that enhances the capacity of low-rank models by enabling information flow across low-rank subspaces. We extensively validate the benefits of LaX on pre-training tasks with ViT-Base/Large and LLaMA-like models ranging from 60M to 1B parameters. LaX boosts low-rank model performance to match or exceed the full-rank baselines while using 2-3$\times$ fewer parameters. When equipped with low-rank adapters (i.e., LoRA) for fine-tuning LLaMA-7/13B, LaX consistently improves performance on arithmetic and common sense reasoning tasks with negligible cost.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the common bottleneck in low-rank compressed deep neural models, where lower-rank models lead to lower performance, but higher-rank models increase the computational burden. To overcome this, the authors propose the LaX method as a plug-in module to enable information flow between low-rank subspaces, inspired by the Residual Mechanism. Moreover, the paper introduces the LaX Gate to align latent spaces with different dimensions. In the experiment, the paper evaluates the effectiveness of LaX through a set of low-rank pre-training and fine-tuning experiments.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**:\n\n• Low-rank methods for compressing deep neural networks have proven to be an effective way to improve the efficiency of DNNs. However, the performance gap between the full model and low-rank models has always been an important problem to solve. This paper proposes a simple yet effective method that is widely suitable for many different types of low-rank models. The motivation drawn from the Residual Mechanism is interesting, and the implementation is original.\n\n### **Weaknesses**:\n\n• Some parts of how the model works are not described clearly enough, and the paper lacks further experiments to deepen our understanding of the proposed LaX module. Please refer to the questions for more details.\n\nQUESTIONS:\nHow LaX works on tensor decomposition is unclear, and it lacks a formal description, like its matrix counterpart. Specifically, regarding the Intra-Layer LaX:\n\n• In Figure 2(C), the input is a 2-order tensor, which can first contract with either $C^0$ or $C^1$. What is the contraction order? Assuming the input is a matrix of size $m \\times n$ and contracts with $C^0$ first, then after the contraction, will the rank $r_0$ be boosted to $nr_0$?\n\n• In Row 131, the authors mention that a series of latent features will be produced in order: $C^{0}x$, $C^{0}C^{1}x$. According to this description, it seems like the input is just a vector. However, in many cases, the input should also be tensorized, so it should contract with $C^{0}$, $C^{1}$, and $C^{2}$ simultaneously. Therefore, I do not understand how the Intra-Layer LaX is performed, especially when the input is a high-order tensor and contracts with $C^{0}$, $C^{1}$, and $C^{2}$.\n\n• Most of the experimental results are used to support how LaX works on the matrix decomposition-based compression model, with only Table 1 demonstrating how LaX works on the tensor decomposition-based model. The work would benefit from more experiments demonstrating how LaX works on tensor decomposition-based methods.\n\n• As an extension to question 3, tensor decomposition has a more complex structure than matrix decomposition. If the paper could include some additional experiments to better understand how LaX works on tensor decomposition—specifically the pros and cons, and when it is most useful or fails—it would greatly enhance the impact of the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper introduces Latent Crossing (LaX), a lightweight, plug-and-play module that enhances low-rank training for foundation models like ViTs and LLMs. By injecting residual connections between latent features across or within low-rank layers, LaX improves cross-subspace information flow and effectively expands model capacity without increasing parameter count. It also includes flexible gating mechanisms to manage dimension mismatches. Experiments across pre-training and fine-tuning settings demonstrate consistent performance gains with negligible computational overhead.

STRENGTHS AND WEAKNESSES:
Strength:
1. Lightweight and parameter-free design with negligible computational overhead.
2. Compatible with various low-rank methods and architectures (e.g., ViT, LLM).
3. Demonstrates consistent performance gains across pre-training and fine-tuning tasks.

Weakness
1. LaX primarily integrates residual fusion into the LoRA setting, with minimal architectural changes and limited theoretical advancement in low-rank modeling.
2. The method is only evaluated under fixed-rank settings; its compatibility with adaptive or dynamic-rank methods (e.g., AdaLoRA) remains unexplored.
3. Performance gains on LLM fine-tuning tasks are relatively modest (see Tables 7–8), and the paper lacks direct comparison with other strong PEFT baselines in this setting.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Latent Crossing (LaX), a simple plug-and-play module that enhances low-rank training methods by enabling information flow between low-rank subspaces through residual connections. The authors show that LaX consistently improves performance across several low-rank methods (SVD, Tensor Train, CoLA) on both vision and language tasks, with minimal parameter overhead.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- **Clear motivation and simple solution**: the authors propose a new and intuitive method to improve the training of low-rank models, taking inspiration from ResNet-style residual connections.\n- **Comprehensive experimental validation**: the validation spans multiple domains (vision and language), model scales (up to 1B parameters), and different low-rank methods. Results show consistent improvements, with notable gains for Tensor Train methods on ViTs (+4.32%).\n- **Practical applicability**: LaX works as a drop-in module for existing low-rank methods, making it easy to adopt in practice.\n- **Reproducibility**: the authors will release the code with the paper and have included the hyperparameters of the experiments in the Appendix.\n\n\n# Weaknesses\n- **Novelty**: while LaX proves to be effective, the novelty of the method is rather limited since the core contribution is conceptually simple: adding residual connections in the latent space of the low-rank modules.\n- **Limited analysis of when LaX helps most**: while the results show LaX improves the performance the most on weaker baselines (Tensor Train), there's insufficient analysis of what factors determine the magnitude of the improvement.\n- **Lack of theoretical analysis**: the paper provides no theoretical justification as to why LaX improves the training of low-rank models. \n- **Statistical significance**: the authors just train one model for each setting, and do not include confidence intervals. The small improvement over the baselines could be in part explained by the inherent randomness of the training process. \n- **Missing ablations**: the paper could benefit from more ablation studies, such as the effect of applying LaX to different layer types or the impact of normalization in LaX.\n\n\n\n## Citations\n- The LaX Gates are tightly related to the [*Highway Networks*](https://arxiv.org/abs/1505.00387) , but the authors fail to cite this work.\n## Experimental concerns\n- The ViT experiments use relatively small models. Do gains persist for bigger models like ViT-H/Vit-G?\n- Missing comparisons with other modern LoRA variants, and lack of exploration of orthogonal benefits with these approaches\n\n\n\n## Minor issues\n- line 113: incorrect grammar \"our motivation is **that**, can we improve the performance...\"\n- line 120: \"requiring no additional parameters or computation overhead\". Isn't LaX increasing the FLOPs in the forward pass by $(L - 1)h_{in}$, where $L$ is the number of layers? Also, $h_{i-1}$ needs to stay in memory for the computation of the next layer. This is, in fact, mentioned in lines 157, 178.\n- In Table 6, the choice of the cross/check for the identity gate and dense gate usage is unintuitive.\n\nQUESTIONS:\n- Could you run multiple experiments with different seeds to increase the number of training curves in Figures 6, 7, 8, 9 and to include confidence bounds in Tables 2, 6, 7, 8?\n- Have you investigated where in the network LaX provides the most benefit?\n- Can you provide any theoretical insight into why LaX improves optimization/generalization in low-rank settings?\n- How does LaX interact with other techniques for improving low-rank training? (e.g. Q-LoRA, LoRA-FA, ...)\n\nLIMITATIONS:\n- **Modest improvements in some cases**: While improvements are consistent, they can be modest (e.g. +0.27% for CoLA on 130M LLM). The cost-benefit trade-off isn't always clear. \n- **Dependency on baseline quality:** LaX appears to help weaker methods more (e.g. Tensor Train) but provides diminishing returns on already strong methods like CoLA.\n- **Gate selection**: the optimal gate type (Identity/Linear/Tensor/Dense) appears to be task and model dependent, requiring empirical search rather than principled selection.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Latent Crossing (LaX), a small addition to many low-rank modules/PEFT methods that improves performance across the board. LaX involves equipping the latent space of low-rank modules with residual connections, followed by layer normalization. Before applying the residual connection, however, one of four proposed gating layers is applied to the low-rank representation. The proposed layers are lightweight in terms of computation and memory complexity, especially for the PEFT setting, where these considerations are less important. To validate the effectiveness of their method, the authors provide extensive experiments across ViT pre-training, LM pre-training, and LLM fine-tuning. The results generally show that the author’s proposed method improves performance beyond.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The paper is well written and easy to follow.\n- The tables are clear and provide all the necessary details\n- The method outperforms all baselines tested within the author’s study.\n- The authors provide training curves for their method in the appendix, which is appreciated. \n\n**Weaknesses**:\n- The authors show a surprising result: pre-training with CoLa + LaX outperforms Full-rank pre-training (see Table 4), but this is not explained in the text, leading me to wonder if the hyperparameters for Full-rank were well tuned?\n- In section A.3 of the appendix, the authors state that they sweep at the scale of 1e-3 for base models and 1e-2 for LaX variants. For reproducibility, it would be best to mention how many HP configurations were tried for each method and ideally the values shown.\n- When browsing related literature, I found a closely related work published last year that should be discussed [1]. It does not overlap with the author's method, but could complement it and is close enough to warrant discussion. \n\n\n\nWhen browsing related literature, I also found [2], a closely related concurrent work which might be of use to the authors, but that is too new to require evaluation.\n\n---\n**References**\n\n[1][ResLoRA: Identity Residual Mapping in Low-Rank Adaption; ACL 2024]\n\n[2][LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuning.]\n\nQUESTIONS:\n- In Table 2, the Identity gate has the same number of parameters as the Base model, but the authors state on line 164 that they postpend Layer Normalization to each residual connection. Don’t the layer norms have parameters? \n\n- In Table 4, why does CoLa + LaX outperform Full-rank?\n\n- Do you think the result from table 4 would hold if a different optimizer, e.g., Muon, were used, which might be able to take advantage of the full rank matrix?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-09"}