Title: Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LoRA

Token-wise Projection
Abstract: Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). 
LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank.
In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection.
This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens.
To address this limitation, we propose **Token-wise Projected Low-Rank Adaptation (TopLoRA)**, which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner.
Formally, the weights of TopLoRA can be expressed as $B\Sigma_X A$, where $A$ and $B$ are low-rank matrices (as in standard LoRA), and $\Sigma_X$ is a diagonal matrix generated from each input token $X$.
Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections).
Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. 
The code is available at https://github.com/Leopold1423/toplora-neurips25.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes a token-wise projected LoRA method, which dynamically adjusts the LoRA weights for different tokens. It outperforms the standard LoRA method across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe authors demonstrate superior results in multiple datasets.\n\nWeaknesses:\n\n1. The authors devote a substantial portion of the paper to describing the motivation, but lack experimental support.\n2. The authors mention in the abstract that \"all input tokens share the same weights.\" This appears to contradict the standard LoRA design, where each module has its own set of independent weights without weight sharing across tokens. A more detailed explanation or clarification would be appreciated.\n3. While the authors state in line 116 that TopLoRA improves the expressiveness of LoRA under low-rank settings, the experiments reveal that, for the same rank, TopLoRA introduces about 50% more parameters than standard LoRA. This raises the question of whether the observed performance improvements are due to the proposed design or merely the increased number of parameters. The current experiments lack sufficient evidence to disentangle these two factors.\n4. Since the proposed method learns a distinct $\\theta$ for each token, $\\Delta W$ becomes token-dependent. This raises a concern about whether the approach sacrifices the mergeability advantage of standard LoRA and incurs additional computational cost during inference. It would be helpful if the authors could clarify this trade-off and discuss its impact on inference efficiency.\n5. The authors have not compared their method with other rank enhancement techniques such as HiRA and KronA, or with rank adaptation methods like AdaLoRA. Including such comparisons would strengthen the experimental evaluation and better position the proposed approach within the existing literature.\n6. This paper lacks a related work section.\n7. There is an error in line 40: a unitary matrix must be square, but $Q_B$ and $Q_A$ do not satisfy this condition. This needs to be corrected or further clarified.\n\nQUESTIONS:\nQ1: While the paper spends a lot of effort discussing its motivations, it lacks experimental support for its claimed advantages. Can the authors provide additional explanations or empirical evidence to better justify the motivations behind their proposed approach?\n\nQ2: In the abstract, the authors state that standard LoRA \"all input tokens share the same weight\", which seems to contradict the traditional LoRA framework. Can the authors explain further and how it differs from the TopLoRA design?\n\nQ3: The paper TopLoRA claims to improve expressiveness under low-rank constraints, but TopLoRA introduces about 50% more parameters than LoRA of the same rank. How do the authors distinguish between the performance improvement brought by the architectural improvements and the increased number of parameters?\n\nQ4: Since $\\Delta W$ becomes token-dependent, does this method give up the mergeability and inference efficiency advantages that LoRA usually has? Can the authors quantify the additional inference cost and comment on whether this trade-off is reasonable?\n\nQ5: Why are relevant baselines such as HiRA, KronA, and AdaLoRA omitted from the comparison? Inclusion of these approaches provides a clearer presentation of the relative strengths and weaknesses of TopLoRA.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reformulates the low-rank adapter to show the limited accessibility of the current LoRA residuals to input-output subspaces. Coupled with the static behavior of conventional LoRA implementation when it comes to different tokens in varying contexts, they propose to modify this formulation to allow for token-dependent adjustments on the residual during both training and inference. This is done by adding an additional set of weights that transform the activations from the previous layer using normalization and exponential operators to a new vector. This vector is then added to a diagonal matrix within the new formulation of the adapter.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The writing is clear and fluid.\n- The experiments are comprehensive and tests are done on a variety of models.\n- The proposed reformulation is straightforward.\n\nWeaknesses:\n- The statement \"LoRA typically exhibits a performance gap when compared to FFT\" in L29 means that you'd need to compare to methods addressing this gap. You've already compared to DoRA but other methods that address the same problem are not compared against, e.g. LoRA+ [a].\n- Similar works like MoELoRA have already been mentioned by the authors but dismissed due to differences in goals. I understand the difference in goals and that MoELoRA should not be compared against as a SoTA for this work but it would have been complementary to see combinations of the two works as an additional experiment.\n- A parameter count on Figure 2 would complement the visualization.\n\n[a] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. \"LoRA+: Efficient Low Rank Adaptation of Large Models.\" Proceedings of the Forty-first International Conference on Machine Learning, 2024. https://openreview.net/forum?id=NEv8YqBROO.\n\nQUESTIONS:\n- Regarding the statement “different tokens exhibit distinct semantics and distributions, and the shared input-output projection” in L51, given that both model weights and LoRA weights can learn to map different tokens differently, the only way for me to make sense of this statement is to relate it to the rank again. That way, I can interpret this in the sense that when the rank is low, there are fewer subspaces available to express every concept represented by the tokens in the fine-tuning dataset and additional token-dependent mechanisms are required. Is that what you mean by L51? The reason for the question is that you start the paragraph with “beyond the LoRA rank…” and that makes me think that you’re stating this is something completely disentangled with respect to the rank.\n\nLIMITATIONS:\n- An additional limitation would also be the dependence of the additional weights on the rank of the adapter. This also makes me wonder if it is possible to share the overhead between layers by attributing a single $\\Sigma_X$ operation for the whole model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a PEFT method called Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token. Specifically, it learns a mapping that projects the input token to a scaling matrix that is further multiplied by the LoRA matrix. The authors conducted extensive experiments on various models and datasets, demonstrating that TopLoRA consistently outperforms LoRA and its variants. At the same rank, TopLoRA achieves a 2-3% accuracy improvement over LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n(1) This paper is well motivated by considering token-wise differences, and the proposed method TopLoRA is clear and makes sense to me.\n\n(2) The proposed method is comprehensively evaluated on multiple tasks (General Language Understanding, Mathematical Reasoning, Commonsense Reasoning) on multiple LLMs, and has performance improvement compared to baseline methods. \n\n(3) Some theoretical analysis is shown to make the motivation more convincing.  \n\n**Weaknesses**\n\n(1) The authors claim that their method achieves 2-3% accuracy improvement over LoRA. Firstly, if this comparison is between TopLoRA and LoRA with the same rank, then it will be unfair to LoRA as TopLoRA leads to ~50% trainable parameter increase. Under a fair setting, the improvement of TopLoRA will shrink. In addition, if we compare TopLoRA to stronger LoRA variants, the improvement is smaller or even marginal. \n\n(2) In L113, the authors claim that “Existing methods that apply a uniform input-output projection across all tokens fail to capture token-specific information.” But in my opinion, the original Q,K,V mapping is capturing token-wise information, so the attached updated LoRA layers are capable of capturing token-wise information as well. It seems to me that TopLoRA is actually providing an additional channel that strengthens the token-wise information. In such cases, the methodological contribution of TopLoRA is limited to me.\n\nQUESTIONS:\nAre there any evidence that support the authors' claim \"Existing methods fail to capture token-specific information\"?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve the performance of Low-Rank Adaptation (LoRA) for fine-tuning large language models. The key insight is that the limitations of standard LoRA stem not only from its rank (which determines input/output space dimensionality) but also from its shared input-output projection across all tokens. TopLoRA addresses this by dynamically adjusting the LoRA weights ($B\\Sigma_X A$) for each input token $X$ through learned diagonal matrices $\\Sigma_X$. It enables token-wise projections while maintaining the original rank. TopLoRA is evaluated on a range of tasks and models, clearly demonstrating its advantages.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper is well written and easy to understand. It provides a clear framework to understand LoRA through QR/LQ decomposition, showing how to view LoRA weights from the perspective of input space, output space, and input-output projection.\n- TopLoRA does not follow the common method of increasing rank, but focuses on enhancing projection diversity. Its token-wise diagonal matrix $\\Sigma_X$ is simple and powerful, and can improve expressiveness without increasing rank. This is very inspiring for the optimization of LoRA.\n- The experiments cover three model sizes (7 B, 8 B, 14 B), multiple architectures (RoBERTa, Gemma, LLaMA-3, Qwen 2.5), multiple tasks (GLUE, mathematical reasoning, common sense reasoning), multiple ranks and strong baselines (DoRA, MELoRA, HydraLoRA). Compared with LoRA, the accuracy continues to increase by 2-4% at the same rank, clearly proving the effectiveness of TopLoRA.\n\nWeakness:\n\nAlthough MoELoRA is mentioned (in Section 3.3), only its variant HydraLoRA is compared. A detailed comparison with MoELoRA is needed as the two are very similar.\n\nQUESTIONS:\nThe paper mentions applying Exp and RMSNorm to diagonal matrices, but their role needs further explanation. Also, does the order in which they are applied matter?\n\nHow does TopLoRA perform on vision tasks?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}