Title: Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
Venue: NeurIPS 2025
Accepted: yes
Keywords: noise-robustness

asymmetric LoRA

mixture of experts

parameter-efficient fine-tuning
Abstract: Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces LoPE, a novel fine-tuning framework that strengthens the robustness of LLMs to noisy data. LoPE employs a dedicated \"poisoning expert\" within an asymmetric LoRA architecture. Through a two-stage fine-tuning process with hybrid noise injection and expert masking during inference, LoPE effectively minimizes the impact of noise without requiring data cleaning or architectural modifications, achieving strong performance.\n\nSTRENGTHS AND WEAKNESSES:\nPros: \nLoPE leverages the HyNoIse and DyCompEnSate components to isolate noise handling to specific experts and mask it during inference without data cleanup. Meanwhile, it is evaluated on different downstream tasks, highlighting its efficiency and robustness\n\nCons: \n1. Sharing and fine-tuning the A-matrix in both Stage I and Stage II raises the question of whether harmful information learned during the first stage (from noisy data) might be absorbed into the A-matrix, potentially leading to performance degradation.\n2. HyNoIse is primarily a character-level noise and may not reflect real-world, high-complexity datasets. Is it possible for the authors to provide analysis of the performance in a real task?\n3. Missing hyperparameter analysis of β.\n4. The first paragraph of Section 1 lacks a relevant citation;\n5. For the without noise performance in Table 1, why does LoPE improve compared to HydraLoRA, if I understand correctly, LoPE lies mainly in the two stages of fine-tuning the noise data, but why would this improve the Orig dataset as well?\n\nQUESTIONS:\n1. The role of the shared matrix A remains somewhat unclear, particularly regarding the rationale for fine-tuning it in both Stage I and Stage II. It would be helpful if the authors could clarify the motivation behind this design choice and provide ablation studies to highlight the impact of updating A across both stages.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method called LoPE for making large language models more robust to noisy data during fine-tuning. LoPE is designed based on HydraLoRA which is an existing framework that uses asymmetric LoRA adapters. The key idea is to assign a poisoning expert to learn from noisy data. The training includes two steps: 1) the poisoning expert is trained using mixed types of noise, while the other adapters are kept unchanged; 2) the normal adapters are trained on clean data while keeping the poisoning expert frozen. When making predictions, the poisoning expert is turned off so that only the clean experts affect the results. The authors show experimental results on several tasks like MMLU and QA to support their claims.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The experiments employ state-of-the-art LLMs as backbones. \n2. Ablation studies are conducted to understand the effectiveness of each module. \n\nWeaknesses:\n1. The problem setting is somewhat unclear. Based on the descriptions in the Introduction and Experiment sections, my understanding is that the fine-tuning data (training data) are assumed to be noisy—for example, by manually injecting 5% noise to simulate real-world scenarios. However, LoPE appears to use both noisy data (in stage 1) and clean data (in stage 2) during training. Since the noisy data are synthetically generated from the clean data using specific strategies in HyNoIse, the actual input to LoPE is still the clean dataset. This makes the problem setup and the role of noise in the training pipeline somewhat confusing.\n2. The overall framework is largely built on HydraLoRA. While the extension to noisy data training is appreciated, the core technical contribution is a bit incremental. \n3. The results reported for HydraLoRA on MMLU (46.10 at rank r=4) are not representative of the best-known baseline performance. The HydraLoRA paper reports 47.22 at r=8, which makes the performance comparison in Table 1 of this submission less convincing.\n4. The motivation and effectiveness of the two-stage fine-tuning strategy are not well explained. In particular, it is unclear why the \"poisoning expert\" B_D​, which is trained to handle noise, is ignored during inference. Is it because you assume the test data are relatively clean? \n5. The paper criticizes prior methods for being limited to specific noise types, yet the HyNoIse strategy also relies on predefined types of discrete and continuous noise. As such, it does not effectively address the core issue it raises. More evidence is needed to support the claim that LoPE generalizes across diverse noise patterns.\n6. All compared baselines are general LoRA variants, none of which are explicitly designed or adapted for noisy data. This weakens the empirical claims. \n7. The presentation of the paper requires significant improvement. It contains several unclear expressions and awkward phrasing. Some phrases like “generate output with relatively clean knowledge” are vague. A thorough language and writing revision is strongly recommended to improve readability and ensure the technical content is clearly communicated.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LoPE (asymmetric LoRA poisoning experts), a pretty clever approach to handling noisy data during fine-tuning. Instead of trying to clean noise, they deliberately inject it. They use an asymmetric LoRA structure with a dedicated \"poisoning expert\" trained to handle noise. During inference, they simply mask this expert out. Their two-stage process is neat: first train the poisoning expert on noisy data, then freeze it while training other components. They ran experiments across several benchmarks and showed some improvements over other PEFT methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The \"noise through noise\" approach represents a creative departure from conventional denoising methods.\n\n2. Their use of asymmetric LoRA architecture to separate functionality between experts is technically sound and well-executed.\n\n3. The method shows promising results across multiple tasks, with notable performance on mathematical reasoning and complex reasoning tasks.\n\n4. The DyCompEnSate approach demonstrates careful consideration of the knowledge gaps that could arise from expert masking.\n\n## Weaknesses\n\n1. This paper mostly compares against other PEFT methods like LoRA and HydraLoRA, but I'm missing comparisons with actual denoising methods (LeCoRE, MICL, etc.). I'd like to see them pit their method against mainstream denoising techniques - that would make their case much stronger.\n\n2. This paper primarily tests with 5% noise injection, but then cite research saying real-world noise can be 8%-38.5%. \n   - Figure 4 shows different noise ratios but doesn't dig into why performance drops at higher levels.\n   - Would love to see how this holds up at 10%, 20%, 30% noise - does it collapse or gracefully degrade?\n\n3. It would be better that include visualization or analysis of its internal representations to describe what exactly is this poisoning expert learning.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}