Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
Authors: Zhaokun Wang
Jinyu Guo
Jingwen Pu
Lingfeng Chen
Hongli Pu
Jie Ou
Libo Qin
Wenhong Tian
Venue: NeurIPS 2025 poster ()
Keywords: noise-robustness
asymmetric LoRA
mixture of experts
parameter-efficient fine-tuning
Primary area: deep_learning

Abstract: Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning.

Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
Zhaokun Wang', Jinyu Guo'; : Jingwen Pu', Lingfeng Chen', Hongli Put, Jie Ou', Libo Qin', Wenhong Tian 'School of Information and Software Engineering, University of Electronic Science and Technology of China
School of Computer Science and Engineering. Central South University (guojinyu, sian_wenhong sreenoong@uestc... edu.
Abstract
Current parameter-efficient fine-tuning methods for adapting pre-trained language models downstream tasks susceptible interference from noisy data. Con- ventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to eггоr accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in asymmetric LoRA configuration. Through two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine- tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowl- edge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning.
Introduction
Recently, pre-trained language models (PrLMs) have demonstrated remarkable success across various natural language processing tasks (Gao et al. 2025 Bianchi et al. 2024 Guo et al. [025 Yang et al.) 2025). To further enhance model performance on downstream tasks, researchers typically employ domain-specific corpora for targeted fine-tuning of pre-trained models (Li et al. 2024a). However, this mainstream solution critically depends on the quality of training data in both pre-training and fine-tuning stages, though with distinct emphases. Different from the pre-training stage, where models acquire general linguistic knowledge through exposure to large-scale corpora, the fine-tuning stage requires models to adapt specific downstream tasks using comparatively smaller datasets, thereby imposing stricter requirements on data quality (Mai et al. 2024 Shi and Lipani| 2023).
In downstream NLP applications (Ou et al. 2025), noise poses multiple critical challenges: 1) labeling errors, syntactic irregularities, and extraneous content disrupt the model's ability capture and learn effective features, leading to unstable training and hindering the establishment of precise, robust knowledge systems (Mao et al. 2023 Xu al. 2024); 2) noise weakens model generalization when encountering unseen data, impairing flexible knowledge transfer (Ahn et al., 2024); 3) noisy data may introduce biases, causing models to disproportionately emphasize or neglect specific categories,
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A
Roder
Naise Peisoning
B
B
Minkc
The unique configuration the asymmetric LoRA architecture our approach, where the grey area on the right represents the mask for the poisoning expert, thereby eliminating the knowledge affected by the noise learned by the poisoning expert.
Figure I:
compromising fairness and interpretability during fine-tuning stage (Zhao al. 2024). These issues collectively manifest as drastic performance degradation in downstream tasks.
Recent research noise can be broadly divided into two types of work. The first type focuses on reconstructing the data before training by cleaning, filtering, or relabeling to construct purified datasets YYe al. 2022 Yuan al., 2024). The second type involves developing dedicated denoising architectures during training (Wu et al. 2023 Zhu et al. 2024). Through the analysis of existing denoising approaches, identify two primary limitations current research paradigms: (1) Numer- ous methods heavily rely on manual intervention or prior assumptions during data pre- prepprcesssigg requiring noise detection and cleaning before model training: (2) Methods that focus on improving the model architecture during training avoid explicit data cleaning, but still cannot avoid discriminating noise information. noise types or data distributions. In addition, data processing pipelines not only incur additional computational and annotation expenses but also lead to error propagation.
light of this, the effectiveness of these methods is often limited to specific
Fortunately, compared to noise injection identification and processing, noise injection presents a cost-effective and easily automatable alternative that enhances data distribution authenticity through stochastic generation. However, noise addition to training data to improve model robustness seems like pipe dream, as both clean and noisy samples influence all model parameters, preventing optimal utilization of noise patterns, Therefore, to exploit noise data for beneficial robust effects, a possible approach is integrate noise injection into a dedicated noise-adaptive module. Subsequently, the module that has learned the noise patterns can be eliminated through complementary set operations, thereby effectively shielding the main network from noise perturbations. We name this strategy "Guided Poisoning".
To achieve this goal, it is necessary to model and separate the noise with dedicated modules to split it. The recent emergence of Mixture-of-Experts (MoE) systems (Shazeer et al. 2017) provides an ideal implementation framework, their architectural design naturally supports functional specialization through expert combinations. In this context, Low-Rank Adaptation (LoRA) (Hu et al. 2022) within asymmetric MoE architectures offers unique advantages. As shown in Figure ! the modified MoE-LoRA architectures replace multiple A matrices with shared A matrix, where each matrix B forms an independent expert. Prior studies (Tian al. 2024) demonstrate that in such asymmetric LoRA architectures, the shared A matrix typically captures universal knowledge, while individual B matrices adapt to discrepancy knowledge.
Building on these insights, this paper proposes a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE). Specifically, take one of the matrices B to serve the poisoning expert within the MoE to route and process noise data as shown Bp on the right side of Figure During the inference stage, we achieve noise robustness by masking the pathway of the poisoning expert. The overall process consists of two stages: fine-tuning (I-I, 1-II) and inference (II).
I-D) In the first stage of fine-tuning, combines augmentation at both the input and embedding levels to enhance the original dataset. While freezing other parameters, augmented data. This strengthens the poisoning expert's noise processing capability and leverages the properties matrix A learn universal knowledge.
propose hybrid noise injection (HyNolse) approach that
train the matrix A and poisoning expert Bp of LoRA with the
I-II) In the second stage of fine-tuning, we focus on the router's ability to guide both clean and noisy data and the normal experts' capacity for learning from clean data. Specifically, we fine-tune

---

the entire LoRA architecture using original data without noise injection, freezing the poisoning expert Bp 's parameters but allowing its participation forward propagation. Moreover, further strengthen the integrity of all expert outputs, we propose the Dynamically Compensated Expert Synergy (DyCompEnSate) mechanism to mitigate the impact of removing the poisoning expert during inference.
II) During inference, we only utilize the outputs from normal experts trained on the original data. By masking the poisoning expert, we complete the inference task with nearly noise-free knowledge.
Extensive experiments benchmark datasets across mainstream NLP tasks demonstrate that our method achieves strong performance without relying on processed clean data, addressing limitations in existing denoising methods. A subsequent series of experiments further validates the fitness between our asymmetric LoRA architecture and noise-robust enhancement. Notably, since LoPE is based on low-rank parameter-efficient fine-tuning framework, it maintains noise robustness while also achieving high efficiency. The main contributions of this paper are summarized as follows:
1. We propose LoPE, novel noise-robust adaptation method that utilizes noise injection to handle noise. This approach integrates the noise handling module as an independent expert using the asymmetric LoRA architecture. Combined with multi-stage fine-tuning and masking operations, we enhance the model's 's robustness to noise only through the generated noisy data.
2. We design a flexible hybrid noise injection strategy, introducing discrete noise at the input level and continuous noise at the embedding level. We also devise a dynamically compensated expert synergy mechanism to compensate for the gap caused by directly masking the poisoning expert.
3. Extensive experiments multiple mainstream benchmark datasets show that method achieves competitive performance, which departs from conventional noise-handling paradigms and addresses critical limitations in existing denoising methods.
2 Related Work
2.1 Research on Data Noise
As data quality continues to play an increasingly crucial role in the performance of deep learning models, handling data noise has become a pressing and critical challenge across various domains. Recently, numerous studies have been conducted to address this issue, aiming to enhance the accuracy and reliability of deep learning models. The primary approach involves constructing denoising models for specific tasks data structures to learn from the data. (Xu et al., 2024) designed MICL, a contrastive learning module to filter out irrelevant interactions in recommendation systems. (Biester et al. 2024) introduced LLMClean, leveraging generated contextual modeling and rule-based correction for tabular data denoising. (Mao et al. 2023) proposed LeCoRE, which mainly solves the problem of complex search intent understanding conversational search. These existing methods usually have limitations when dealing with data from different domains, and the trained models are difficult to transfer to other domains. Another common approach involves cleaning, filtering, or relabeling raw data before training to eliminate noisy samples and construct a cleaner dataset. (Feng al. 2024) introduced an intelligent receiver integrating a pre-denoising network and an LSTM module for denoising preprocessing. (li et al. (024) developed self-denoising method to refine LLM predictions on noisy inputs. Despite the advancement of existing denoising approaches, they have certain limitations, such as essential noise detection, limitation to data distributions, and error propagation cascades.
2.2 Development of LoRA
LLMs have attracted considerable attention due their powerful natural language processing ca- pabilities (Kenton and Toutanova, However, the massive parameters of these models make fara-maer tuning prohibitively expensive. Low-Rank Adaptation (LoRA) (Hu al.) (022) decomposes weight updates into trainable low-rank matrices (A, B), maintaining performance while drastically reducing parameters and computation, establishing it as a key parameter-effin -arameter-efficient fine-tuning (PEFT) (Li and Liang] 202T) technique. Recent innovations integrate LoRA with Mixture- of-Experts (MoE) architectures. MoE-LLRR (Dou et al. 2024) synthesizes complementary strengths: MoE's specialized expert networks with gating mechanisms synergize with LoRA's low-rank adap-
3

---

tations, enabling cross-domain adaptability. This hybrid approach mitigates multitask interference while preserving parameter efficiency through task-specific adaptor coordination. (Li et al. 2024b) integrated LoRA experts into the feedforward network (FFN) layers of a frozen pre-trained model, improving task adaptability while addressing imbalance issues MoE models. To address performance degradation of LoRA in complex corpora while maintaining parameter efficiency, (Tian) et al., 2024) developed HydraLoRA, an asymmetric structure that partitions LoRA modules into specialized experts with MoE-based routing, improving efficiency and adaptability in heterogeneous corpora. The asymmetric LoRA structure's s unique design enables explicit noise modeling and sepa- rates processingn modules. This provides a foundation framework for achieving noise robustness through the generated noise.
3 Methodology
In this section, we present a comprehensive description of our proposed noise-robust adaptation method via asymmetrie LoRA poisoning experts, which encompasses two stages: fine-tuning and inference, as illustrated in Figure 2 We begin by introducing the asymmetric LoRA architecture and the construction of poisoning experts, followed by sequential explanation of our framework's workflow. Details of each module are given respectively in the remainder of this section.
lasa
Ispk
Htylesbb
Tabbeding
Traheeking
Candiaanall Iaalss
Reste
Recke
Rocker
FEX
B
B2
B
FEN
B
B
B
BEX
B
B
B2
Finestuing I
Stage
Figure 2: The LoPE pipeline consists of two stages: fine-tuning and inference stages. Fine-tuning Stage I: HyNolse is to enhance the poisoning expert's noise understanding. Fine-tuning Stage II: Freezing the poisoning expert while fine-tuning the remaining experts and the shared matrix. Inference stage: Masking the poisoning expert with noise-affect knowledge, allowing the remaining experts to generate outputs that are relatively robust to noise.
3.1 Asymmetric LoRA architecture
Here we first introduce the asymmetric LoRA architecture (Tian et al. 2024) we employed. This architecture utilizes a centrally shared matrix A and multiple distinct matrices B,, shown fol- lows: W Wo + -EE, W.B.A. Compared symmetric architectures, asymmetric configurations demonstrate superior advantages in mitigating parameter redundancy and enhancing computational efficiency. By sharing a central matrix A, the architecture significantly reduces learnable parameters while maintaining representational capacity, yielding more lightweight architecture. The imple- mentation of multiple matrices B, enables flexible assignment of specialized adaptive modules for distinct objectives.
Because of these relatively independent experts, our approach, which involves isolating dedicated module within the model to separate and process noise, becomes technically feasible. In LoPF, the matrix B is treated as the expert adapter. Furthermore, randomly selected matrix designated as a dedicated poisoning expert, denoted as BD. This expert shares the same structure as the other experts but undergoes a targeted modification of its capabilities through the process described in the following sections, with the aim of identifying and processing noise. Its forward propagation is as follows: W Wo + (wDBp EN' w; W.).

---

3.2 Fine-tuning Stage
3.2.1 Hybrid Noise Injection (HyNoIse)
The original dataset is defined as S (i),, where I; denotes model inputs and P; their corresponding target labels. To increase the noise ratio in the data, we implement hybrid data augmentation strategy that integrates both discrete and continuous noise perturbations, with multi- level noise coverage spanning character, token, label, and structural anomalies. To uniformly cover disturbances from all noise types, we adopt equal-injection strategy. First, discrete noise injection is performed using a function NoiseFunctionn)) commonly referenced noise strategies that randomly introduce character-level noise (including word order shuffling, noise character insertion, and character deletion) to generate modified inputs, a/-- These transformations enable to simulation of realistic disturbances such as character-level corruption and random truncations. The augmented dataset is then produced S' (c).p())==
Concretely, we adopt three representative and
Additionally, we inject continuous noise at the embedding level, influencing both the label and token levels. Inspired by (Jain al. 2023), let E E R represent the embedding for input sequences, where b, I, and d correspond to batch size, sequence length, and embedding dimension, respectively. We introduce attention mask M R indicating valid token positions and continuous noise N ~ (() bxxix from a uniform distribution. The noise-injected embeddings are computed as E' E aN M, where @ denotes element-wise multiplication, and stands for noise ratio. Multiplying the noise with the attention mask element-wise ensures that the noise is applied only to the valid positions of the sequence. The proposed HyNolse method combines these complementary noise injection strategies to expose the model to diverse noise patterns during fine-tuning.
3.2.2 Fine-tuning Stage Specialized Poisoning Expert
In Stage we exclusively fine-tune the poisoning expert BD and shared matrix A using the HyNolse method from Section 3.1 while keeping the other expert adapters B; frozen. This stage approach enables BD focused learning of noise-handling without cross-expert interference. The forward computation during this stage is formalized as:
N-1
y = Wox + (WD + Σ (r,(
where wp denotes the adaptive weighting coefficient for the poisoning expert, which is part of wi. and f() stands for freezing parameter operation.
3.2.3 Fine-tuning Stage II: Dynamically Compensated Expert Synergy
In Stage II, we achieve collaboration and complementarity among all expert adapters. To simulta- neously enhance the routing mechanism's discriminative capacity for noise patterns and optimize normal experts for clean data processing, while preserving the poisoning expert's specialization. During this phase, we maintain parameter freezing of the Bp from Stage to keep its capabilities, while updating the remaining expert matrices B, and shared matrix A. The forward computation evolves to:
conduct joint fine-tuning of all adapter components
y = Wox + (WDI(BD) +
w,B.)Az.
We introduce router network (gating function) to dynamically allocate expert adapters' contributions. processes the token representation x through a fully connected layer and softmax to compute expert weights w; softmax W gate"" 2)), where W gate E ÆrxN denotes the router's parameters. To address the interdependency disruption caused by expert masking in the inference stage, we propose the Dynamically Compensated Expert Synergy (DyCompEnSate) mechanism. Our analysis reveals that joint expert training establishes learned inter-expert correlations, where the direct mask of Bp during inference has the potential to disrupt these dependencies. DyCompEnSate dynamically preserves weight compensation. Formally, we construct an
gatte
these relationships through dependency-awant expert dependency matrix 0 E NNN' where the element O,, quantifies the similarity of output between experts and j. The poisoning expert dependencies 0.D computed via output similarity
5

---

analysis: A.D
sim(y:, = ,,2 # D. where sim(-.-) denotes cosine similarity
metric. These dependencies compensate expert contributions through dynamic weight amplification:
K
= Wox + ((WD)(BD)
Σ (1+ (:D)w;
where K indicates the number of active experts participating in the computation. To maintain output stability, we introduce 3 to adjust the weight parameter normalization.
3.3 Inference Stage
During the inference stage, we exclusively employ experts that have acquired purified knowledge during the fine-tuning stage, which naturally leads masking the poisoning expert while preserving the normal experts. Since the matrices B; play the role of linear transformations, in the initial stage, we caleulate the weighted average of the remaining experts aggregate their specialized knowledge. Based on this, we perform the parameter-efficien fine-tuning transformation on the input data using the aggregated expert knowledge. In the inference stage, all parameters are frozen, and the contribution of expert adapters is dynamically adjusted based on the input data, thereby achieving the merging of adapters. Specifically, the poisoning expert BD is masked, such that the system exclusively outputs inference results from the remaining experts trained on clean data to achieve knowledge purification. The result obtained after inference:
K
y == Woz + Σ (1 + (.D)).((BBAAr.
(4)
1=1
4 Experiments
4.1 Datasets, Evaluation Metrics and Compared Baseline Models
Datasets and Evaluation Metrics. The dataset used in our experiment from Alpaca-52K (Taori et al., 2023). which consists of 52K instruction-following data. We used 10% of the dataset for fine-tuning. The dataset spans a wide range of task types, including calculations, question answering, and creation, among others. For evaluation, we first utilized the Massive Multitask Language Understanding (MMLU) (Hendrycks et al. (020) benchmark, and additionally selected three domains in MMLU encompassing total of 30 distinct tasks, for in-depth analysis: Natural Sciences and Engineering Technology (NSET), Social Behavior and Humanities (SBH), and History. We also use the GSM8K (Cobbe et al. (02T)) math problem set, Question Answering datasets (Physical Interaction QA (Bisk et al. 2020), Social Interaction QA (Sap et al., 2019), and ARC-easy (Clark et al. )) We used different datasets for fine-tur fine-tuning and evaluating, which effectively mitigates the issue of domain shift. Accuracy (%) used as the evaluation metric across all datasets. This setup systematically evaluates the model's noise-robust capabilities across general scenarios and specialized domains. A detailed description of these datasets can be found in Appendix A
Baselines. LoPE is general fine-tuning framework that works across different noise environments and data types, and does not conflict with existing denoising methods. Therefore, we mainly compare it with similar fine-tuning methods. Specifically, we compared LoPE with five baseline PEFT methods
on the same datasets: (1) P-Tuning (Liu et al., 2024). (2) Prefix Tuning (Li and Liang 2021 (3)
AdaLoRA (Zhang et al.),, (4) LoRA (Hu et al.) (022), and (5) HydraLoRA (Tian al. 2024 Detailed descriptions of these methods are provided in Appendix B.I
4.2 Implementation Details
We define two modes for the fine-tuning datasets: Orig, corresponding to the publicly available version discussed in Section 4.1, and N Nois, the version with discrete noise injected. To simulate the real-world noise distributions and create challenging and impartial noise learning environment across all baseline models, we employed a uniform discrete noise injection strategy same as Section 3.2.1 to generate the Nois dataset, introducing a controlled proportion of noise into the original data. Empirical studies (Song et al. 2022) suggest real-world datasets contain 8%-38.5% 5% noise. Given that

---

Table 1: The comparative performance (%) of different PEFT methods across various datasets. The methods marked with a f are fine-tuned with the N Nois datasets, while all subsequent methods are fine-tuned based on Orig datasets. r denotes the rank, #A represents the number of shared matrices, #B represents the number of B matrices involved in the inference stage.
Methods
MMLU
PIQA
SIQA
GSMSK
ARC-c
NSET
SRH
History
4&Para
#A
dB
46.10
76.28
52.92
16.15
62.61
35,99
54.78
55.95
0.062
LoPE(r=4)
0.062
P-Tuningt
37.23
71.65
39.97
8.87
45.21
26.04
37.14
45.09
0.193
Prefix Tuningt
37.91
71.79
40.42
9.25
43.48
27.13
38.77
44.21
0.077
AdaLoRA(==2F
39.11
72.29
41.07
10.38
47.84
29.16
39.83
49.69
0.025
1
LoRA(r-2)f
69.47
26.17
0.015
LoRA(e43F
71.45
43.17
27.96
HydaLoRAG=221
42,47
74.65
46.38
10,74
56.08
31.26
42.37
52.65
0.031
1
HydralaRMr-41f
43.08
74.92
47.29
11.83
56.26
34,74
52.35
55.80
0.062
1
Hydrat.aRA(-88f
43.98
75.30
48.
57.47
35,23
$3.07
56.35
0.124
LoPB(r=2)
43.05gn22
76.03_ua0
16.70au.17
11 11.2840.37
58 28.98
33.36g2.47
48, 18.60
54.72..029
LoPB(=1)f
48.76g0.a:
75.49.0.1
18.33
12.72
58.66
31.06
48.98
56.46
0.047
1
LoPE(=4y9
44.12:0.1
76, 26.28
49.0030.41
13.72a0.34
60. 10..0
35. $5.45
52.88
56.84:0044
0.062
1
LoPE(r=0)
76.83 24.82
39.90
14.31,0.as
62.36.0.47
20.19
53.71.0.2
58.02.0034
0.124
the original dataset already contains some inherent noise, we used a conservative injection rate of 5% ensure consistency across models and maintain typical noise levels. The HyNoIse ratio used in our main experiment is also set to 5%, and the number of poisoning experts is set to 1. Detailed descriptions of experiment configurations are provided in Appendix B.2
4.3
Main Results
In this section, we evaluate LoPE's effectiveness through extensive experiments and compare with several mainstream PEFT methods on LLaMA2-7b (Touvron et al. (023). Table | presents
the performance of LoPE alongside the baseline methods. MMLU represents the average accuracy across all tasks. NSET, SBH, and History represent the average accuracy of tasks within each respective set. The specific accuracy for each sub-task can be found in Appendix D.2] We first compared LoPE with HydraLoRA on the Orig datasets mentioned in Section 4.2. In the original environment, LoPE exhibited only limited improvement across these datasets. Given that the Orig datasets are nearly noise-free with minimal inherent noise, LoPE's noise adaptation capability was not significantly demonstrated. Subsequently, we conducted experiments on the Nois datasets. The results indicate that LoPE exhibits robustness and adaptability across most tasks. Notably, LoPE excelled across three QA tasks, achieving 4.89% improvement on the ARC-e with a rank of Consistent with our theoretical analysis, LoPE also showed an improvement on the GSM8K task. Moreover, LoPE outperformed baseline methods in the MMLU benchmark (average accuracy of 57 sub-tasks), achieving a 1. 1.34% average improvement over HydraLoRA with rank of 4. Specifically, as the sub-task of MMLU, NSET, similar to GSM8K, often involves precise computations, LoPE maintained coherent reasoning under noise, whereas traditional methods struggled. Furthermore, LoPE also outperforms other methods in complex tasks such as SBH and History, which involve long texts and logical reasoning. In conclusion, LoPE overcomes the limitations of conventional denoising methods, achieving enhanced robustness through noise injection. Notably, our method maintains the same time complexity of Onn²) as other fine-tuning methods, ensuring high computational efficiency. In [ we also explore LoPE's sensitivity to parameter configurations by examining the impact of varying the rank and the number of experts, as well as comparing its parameter scale with other methods. Higher ranks generally lead to better performance; However, increasing the rank also incurs higher parameter and computational costs, thus requiring balance between scalability and effectiveness. In the following experiments, the rank is set to 4 by default, with three normal experts and one poisoning expert.
4.4 Ablation Study
4.4.1 Ablations on HyNolse
In this section, we explore the effectiveness of the proposed HyNolse method. We fine-tune the model on Nois data and set different basic noise ratios (3.5%, 5%, 8%). At the same time, the
7

---

ratio of our HyNolse method is 5%, and the results are shown in T The results indicate that both discrete noise and continuous noise can improve the method. Our method maintains relatively stable performance when handling different levels of noise. To further verify the effectiveness of our proposed HyNoise method, we conduct ablation studies by selectively removing each type of noise (discrete noise or continuous noise) during injection. As evidenced in Table the robustness of the model somewhat reduced under single-noise conditions. However, performance with either discrete or continuous noise still surpasses the None type noise injection, thereby highlighting the effectiveness of both noise types. Notably, fine-tuning with discrete-only noise outperforms fine- tuning with continuous-oni noise. This can be attributed to the fact that discrete noise directly manipulates the original natural language text, facilitating more effective alignment in the semantic space during the fine-tuning stage II.
Table 2: Average accuracy (%) of LoPE on PIQA and SIQA datasets under different noise conditions. None indicates that no noise is injected during the fine-tuning stage I.
Noise Type
None
Continuous
Discrete
Hybrid
3.5% Level 60.90
61.23
62.07
63.31
5% Level 59.89
60.71
61.95
62.66
8% Level 57.48
58.68
60.14
61.86
To further investigate whether LoPE's strong performance depends on the consistency between the noise types in the Nois dataset and the discrete noise in HyNoIse, we conducted additional experiments introducing inconsistent noise injection between the dataset construction and the discrete noise in HyNolse. The results in Table 3 show that when different operations are applied for LoPE training and dataset construction, all LoPE variants achieve considerable improvements over the baseline HydraLoRA. This demonstrates that LoPE's effectiveness generalizes across different noise categories and does not strongly depend noise type consistency.
Moreover, the observed performance gains on the Orig dataset (Table further indicate that LoPE is effective and generalizable to naturally occurring noisy data. It is worth noting that HyNolse also injects continuous noise, which contributes to enhancing the model's robustness under multiple types of noise.
Table 3:
Performance comparison of LoPE under inconsistent noise injection and Hydral.oRA. Each row represents distinct noise operation. Abbreviations used: NCI (Noise Character Insertion), WOS (Word Order Shuffling), CD (Character Deletion), and WR (Word Replacement), representing four types of noise operations.
Model
Dataset
HyNolse
MMLU
PIQA
SIQA
GSM8K
LoPE
Orig NCI
Continuous + WOS
44.58
75.15
50.11
14.92
HydraLoRA
Orig NCI
None
44.24
75.03
49.75
12.78
LoPE
Orig WOS
Continuous + CD
45.09
75.84
50.55
14.64
HydraL.RRA
Orig WOS
None
44.37
75.21
49.65
12.34
LoPE
Orig WR
Continuous NCI
45.13
75.35
50.17
14.73
HydraLoRA
Orig WR
None
44.31
75.07
49.73
12.39
We further applied HyNolse directly to the original HydraLoRA framework (Table and observed notable performance drop. This mainly due to the absence of the division of labor between normal experts and poisoning experts, and the multi-stage differentiated training, causing noisy and clean data to equally affect the model from the start. Therefore, LoPE's key contribution is its architecture and malti--tage process, enabling explicit noise filtering and robustness.
4.4.2 Impact of Backhone L.LMs
To evaluate the impact of backbone architectures on LoPE's performance, we conducted model substitution experiments, with results summarized in Table 5 LoPE consistently demonstrates stable performance improvements across ILaMMA-7b (a decoder-only model), T5-large (a classic encoder-decoder model), as well as Qwen2-7b and Qwenl.5-14b This result indicates that LoPE's
8

---

Table 4: Performance comparison of LoPE and HydraLoRA (with/without HyNolse) across bench- marks. The methods marked with are fine-tuned with the Nois datasets.
Methods
MMLU
PIQA
SIQA
GSM8K
HydraLoRAA (with HyNolse)
40.57
73.40
45.03
8.47
HydraL.oRA#
43.08
74.92
47.29
11.83
LoPE+
44.42
76.28
49.03
13.72
performance gains primarily stem from its asymmetric LoRA architecture and two-stage training strategy, which strengthen its generalization across diverse tasks and application scenarios.
Table 5: Performance (%) of different backbone L.MMs of method on SIQA dataset after fine- tuning of Nois datasets (5%).
Approaches T5-large
LLaMA2-7b
Qwen2-7b
Qwen 1.5-14b
HydraLoRA 33.64 36.37
47.29 49.03
66.49 68.99
78.23 80.02
LoPE
4.4.3 Ablation Study DyCompEnSate
We verified the effectiveness of the proposed DyCompEnSate method. First, directly masked the output of BD without introducing the DyCompEnSate mechanism, as shown in Figure when the column number = O, which indicates that no experts involved in the DyCompEnSate method, accuracy dropped at the lowest value.
62.66%
61.4%
61%
Accuracy
61%
59%
58.93%
58%
57%
56%
55%
Number of Experts Invelved in Dy CampEEnttee Method
Figure 3: The relationship between the number of experts involved in the DyCompEnSate method and the average accuracy (%) of datasets PIQA and SIQA.
This phenomenon validates our hypothesis: when all experts are jointly trained, there exist significant inter-dependencies among them, and directly eliminating the output of Bp indirectly affects other experts' outputs. With the introduction of DyCompEnSate and the increase in the number of normal experts involved, it dynamically weights the amplification of the contribution other experts to mitigate the information loss caused by removing Bp in smooth and adaptive manner.
These results demonstrate the existence of inter-expert dependencies and the effectiveness of our compensation strategy in preserving performance when masking a compromised expert.
Analysis
5.1 Does Higher HyNoise Ratio Enhance Performance?
In Section 4.2, while our baseline experiments employed 5% HyNolse ratio, we investigated whether an increased noise ratio during Stage I enhances noise robustness. We evaluated four HyNolse ratios (3.5%, 5%, 10%, 15%), with comparative results visualized in Figure 4 Contrary to potential expectations, our analysis reveals a non-monotonie relationship between HyNolse intensity and model robustness. Excessive noise injection (i.e. 2 10%) degrades the router's discriminative capability for noise patterns, as evidenced by reduced expert specialization. We consider that this was due to overwhelming data distortion, which prevents effective differentiation between clean and

---

noisy inputs, ultimately compromising expert weight allocation. We also evaluated the performance under different basic noise ratios; additional experimental results are provided in Appendix [D.]
%
62%
61%
60%
00.10%
89%
58%
58.32%
57%
3,5%
5%
15%
HyNolse Ratio
Figure 4: Average accuracy to different HyNolse ratios in PIQA and SIQA datasets.
5.2
Can the Poisoning Expert Truly Accomplish Its Task?
The core of our proposed method the pioneering introduction of the poisoning expert. Section 4.3, investigated the impact of the number of normal experts in LoRÁ the experimental results. Here, we discuss the influence of the number of poisoning experts on the results. According to the design, poisoning experts play an auxiliary role. Therefore, their number should be lower than normal experts. We conducted experiments with three different configurations of the number of experts, and the parameter freezing settings remain consistent with Section 3. The experimental results in Table 6 demonstrate that increasing the number of poisoning experts has little impact, as it does not affect the objective of stage I of fine-tuning. These poisoning experts share the functionality previously assigned to single poisoning expert. During Stage II of fine-tuning, after freezing their parameters, the normal experts can learn clean knowledge as usual. When the total number of experts remains constant, changing the number of poisoned experts will have a significant impact on the results, as the normal experts responsible for inference are replaced.
Table 6: Average accuracy of LoPE in PIQA and SIQA datasets with different experts, PE is the number of poisoning experts, NE is the number of normal experts.
Method NE-3,PE=1
NE-3.PE-2
NE-2,PE-2
Mask(NE=3,PE=I)
Not Mask(NE=3,,PE-I)
LoPE
62.66
62.31
60.31
62.66
60.75
Also, we propose that the involvement of the poisoning expert in the inference process may result in the contamination of knowledge to a catastrophic degree. In light of this, we set up a comparative experiment: Masking vs. Not Masking. The experimental results show that after the first-stage fine-tuning, the poisoning expert intends to specifically adapt and handle the noisy data. If the poisoning expert's output is not masked during inference, allowing it to participate, the model may be affected by the misinformation it learns from the noise, which may affect the reliability of the inference results. This aligns with original intention for designing the poisoning expert. By masking the poisoning expert that learns from noise-affected data during inference, this approach allows other experts to perform inference using relatively pure knowledge.
6 Conclusion
In this paper, we introduce LoPE, a noise-robust adaptation method via asymmetric LoRA poisoning experts, which is specifically designed address performance degradation caused by noisy datasets. The two-stage paradigm and selective masking mechanism enable the model to enhance its robustness only through generated noise perturbations, as evidenced by our experimental results. This approach effectively overcomes the inherent limitations of conventional noise-handling methods. In future research, we believe the combination of the two-stage fine-tuning paradigm and selective masking mechanism opens a door to a promising area of intractable problem resolution through complement set approaches, and we will explore it for more than noise processing the future.
10

---

Acknowledgments and Disclosure of Funding
This research is supported by the Sichuan International Science and Technology Innovation Coopera- tion Project (also known the Hong Kong, Macao, and Taiwan Science and Technology Innovation Cooperation Project, ID: 2024YFHZ0317), the Chengdu Science and Technology Bureau Project (ID: 0024-YF09-000411
the National Natural Science Foundation of China (Project ID: W2433163), Huawei Funding (Project ID: H04W241592), and the Postdoctoral Fellowship Program (Grade C) of the China Postdoctoral Science Foundation (Grant No. GZC20251053).
References
Qwen2 technical report. 2024.
Sumyeong Ahn, Sihyeon Kim, Jongwoo Ko, and Se- Young Yun. Fine-tuning pre-trained models for robustness under noisy labels. In Proceedings of the Thirty-Thire International Joint Conference on Artificial Intelligence, IJCAI-24, pages 3643-3651. International Joint Conferences on Artificial Intelligence Organization, 2024. Main Track.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang. Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chenggiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijic Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang. Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 2023.
F Bianchi, P Chia, M Yuksekgonul, J Tagliabue, D Jurafsky, J Zou, et al. How well can Ilms negotiate? negotiationarena platform and analysis. PROCEEDINGS OF MACHINE LEARNING RESEARCH, 235:3935-3998 2024.
Fabian Biester, Mohamed Abdelaal, and Daniel Del Gaudio. LImclean: Context-aware tabular data cleaning via Ilm-generated ofds. In European Conference on Advances in Databases and Information Systems, pages 68-78. Springer. 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432-7439, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803 00445, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. Loramoe: Alleviating world knowledge forgetting in large language models via mte-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1932-1945, 2024.
Yunqi Feng, Yunxian Chen, Huimei Han, Weidang Lu, Nan Zhao, Arumugam Nallanathan, and Xiaoniu Yang. Deep learning based physical-layer receiver with pre-denoise. IEEE Transactions Vehicular Technology, 2024.
Jianqi Gao, Jian Cao, Ranran Bu, Nengjun Zhu, Wei Guan, and Hang Yu. Promoting knowledge base question answering by directing llms to generate task-relevant logical forms. In Proceedings of the AAAI Conference Artificial Intelligence, pages 23914-23922 2025.

---

Jinyu Guo, Xunlei Chen, Qiyang Xia, Zhaokun Wang. Jie Ou, Libo Qin, Shunyu Yao, and Wenhong Tian. HASH-RAG: Bridging deep hashing with retriever for efficient, fine retrieval and augmented generation. In Findings of the Association for Computational Linguisties: ACI. 2025, pages 26847-26858, 2025.
Dan Hendrycks, Collin Burs, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0332 2020.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint arXiv:2310.05914, 2023.
Jiabao Ji, Bairu Hou, Zhen Zhang, Guanhua Zhang, Wenqi Fan, Qing Li, Yang Zhang, Gaowen Liu, Sijia Liu, and Shiyu Chang. Advancing the robustness of large language models through self-denoised smoothing. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 246-257. 2024.
Jacob Devlin Ming-Wei. Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, Minneapolis, Minnesota, 2019.
Chengzhengxu Li, Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Chen Liu, Yu Lan, and Chao Shen. Concentrate attention: Towards eenerliageee prompt optimization for language models. Advances in Neural Information Processing Systems, 37:3391-3420, 2024a.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang. and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprini arXiv:2404.155 15159, 2024b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, 2021.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 5:208-215, 2024.
Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng Hao Tu, Hong- You Chen, Vardaan Pahuja, Tanya Berger- Wolf, Song Gao, Charles Stewart, Yu Su, et al. tine-tuning is fine, if calibrated. Advances in Neural Information Processing Systems, 37: 17:136004-136111 19, 2024.
Kelong Mao, Hongjin Qian, Fengran Mo, Zhicheng Dou, Bang Liu, Xiaohua Cheng, and Zhao Cao. Learning denoised and interpretable session representation for conversational search. In Proceedings of the ACM Web Conference 2023, pages 3193 3192-3202, 2023.
Jie Ou, Jinyu Guo, Shuaihong Jiang, Zhaokun Wang, Libo Qin, Shunyu Yao, and Wenhong Tian. Ac- celerating adaptive retrieval augmented generation via instruction-driven representation reduction of retrieval overlaps. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26983-27000, Vienna, Austria, 2025. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 1-67, 2020.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural 2019.
Language Processing (EMNLP-IJCNLP)), pages
12

---

Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of- -f-eepe layer. In International Conference on Learning Representations, 2017.
Zhengxiang Shi and Aldo Lipani. Don't stop pretraining? make prompt-based fine-tuning powerful learner. Advances in Neural Information Processing Systems, 36:5827-5848 5849, 2023.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions neural networks and learning systems, 34(1 8):8 111-88153,, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An mnstruction-following llama model, 2023.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems, pages 9565-9584. Curran Associates, Inc., 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wxXiv2307.092886 2023.
Shuhui Wu, Yongliang Shen, Zeqi Tan, Wenqi Ren, Jietian Guo, Shiliang Pu, and Weiming Lu. Mproto: Multi-prototype network with denoised optimal transport for distantly supervised named entity recognition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2361-2374 2023.
Fuyong Xu, Zhenfang Zhu, Yixin Fu, Wang, and Peiyu Liu. Collaborative denoised graph contrastive learning for multi-modal recommendation. Information Sciences, 679: 121017,, 2024.
Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. In International Conference Representation Learning, pages 33251-33277, 2025.
Fanghua Ye, Yue Feng, and Emine Yilmaz. Assist: Towards label noise-robust dialogue state tracking. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2719-2731, 2022.
Bo Yuan, Yulin Chen, Yin Zhang, and Wei Jiang. Hide and seek in noise labels: Noise-robust collaborative active learning with Ilms-powered assistance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10977- 11011, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. The Eleventh International Conference Learning Representations.
Jinxu Zhao, Guanting Dong, Yucyan Qiu, Tingfeng Hui, Xiaoshuai Song, Daichi Guo, and Weiran Xu. Noise-bert: A unified perturbation-rohut framework with noise alignment pre-training for noisy slot filling task. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6150-6154. IEEE, 2024.
Kun Zhu, Xiaocheng Feng. Xiyuan Du, Yuxuan Gu, Weijiang Yu, Haotian Wang. Qianglong Chen, Zheng Chu, Jingchang Chen, and Bing Qin. An information bottleneck perspective for effective noise filtering on retrieval- augeened generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages
2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract Section and Section of this article accurately reflect the contribution and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of this work in Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The paper provides the full set of assumptions and complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer:[ Yes]
Justification:
The paper provides open access the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ bublic/guides/CodeSuue www.ummiiiiooooooiioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results, See the NeurIPS code and data submission guidelines (ht httppubblicgguulee//oeeee (wwooeeeeeisssooniiiiiiiiiii ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided relevant details in Section 4 and Section B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided relevant details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided relevant details in Section B.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative
societal impacts of the work performed Section 0 and Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not address this issue.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, papersswi swperswitte thcode. thcoddee com/ danasttsss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
This paper does not address this issue.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
This paper does not address this issue.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: This paper does not address this issue
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does not address this issue.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy (ttes://beerrrssssccoooorrrrrres for what should or should not be described.
20

---

Appendix
A Dataset
A.1
Fine-tuning Dataset
Alpaca 52k is a dataset consisting of 52,000 cestructiio-following data, generated by Stanford Univer- sity using the self-instruct method by the text-davinci-003. This dataset aims enhance the model's ability to understand and follow instructions. The dataset covers wide range of domains and task types, including text generation, reasoning and explanation, transformation and calculation, classifi- cation and induction, question answering and suggestions, description and explanation, functionality and logic, well text processing and editing. By fine-tuning on these tasks, the model is better equipped to understand and execute various instructions, thereby improving its effectiveness and reliability in real-world applications.
A.2 Evaluation Dataset
Additionally, comprehensively evaluate the model's performance across different domains, employed multiple evaluation benchmarks. To assess general-domaij capabilities, we utilized the Massive Multitask Language Understanding (MMLU) benchmark. MMLU spans tasks across multiple fields, enabling a thorough evaluation of the model's language understanding abilities. For our evaluation, we focused on three major categories: Basic Natural Sciences and Engineering Technology (NSET), Social Behavior and Humanities (SBH), and History. The specific tasks are listed in the table below..
Table 7: Task categories and corresponding tasks
Task Category
Tasks
NSET
College Biology. College Chemistry, College Computer Science, College Math- ematies, College Physics, Electrical Engineering, Abstract Algebra, Astronomy, Machine Learning, Computer Security, High School Chemistry, Flementary Mathematics, High School Physics, High School Computer Science, High School Statistics, High School Biology. High School Mathematics, Conceptual Physics.
SBH
Moral Disputes, Moral Scenarios, Professional Psychology, World Religions, High School Psychology, Philosophy, High School Government and Politics, Sociology-
History
Prehistory, High School US History, High School World History, High School European History.
NSET consists of 18 tasks related to science, engineering, technology, and mathematics. This category encompasses foundational disciplines within the natural sciences and engineering fields, including biology, chemistry, computer science, mathematics, physics, electrical engineering, astronomy, and more. These tasks cover biological knowledge, ranging from cell structure and molecular biology to ecology. In chemistry, tasks span from analytical chemistry to organic chemistry, covering fundamental laws (e.g., Newtonian mechanics and principles of chemical reactions). In computer science, research focuses on algorithms, graph theory, recursive methods, and the logic of technical implementation (c.g., algorithm design and circuit systems). Mathematical tasks encompass various branches, such as calculus, combinatorics, and ordinary differential equations. These tasks are highly sensitive to data and require significant specialization and logical reasoning, making them ideal for evaluating the model's precision in computation and reasoning.
SBH includes 8 tasks from fields such psychology, philosophy, sociology, and political science, focusing on human behavior patterns, moral decision-making, and ethical dilemmas within a societal context. The tasks involve analyzing moral controversies (e.g., freedom of speech, the death penalty, addiction), examining moral situations (e.g., violence, theft), exploring psychological principles behind individual behavior (c.g., personality development, emotional changes), and discussing ethical
21

---

dilemmas from philosophical perspective (e.g., skepticism, utilitarianism). Additionally, the tasks address the impact of social structures on individual behavior, including socialization and inequality in sociology, and government functions and socio-political systems in political science.
The History category consists of 4 tasks related to the development of human history, including Prehistory, American history, world history, and European history. The content spans topics from the early evolution of humans and the origins of civilization major historical events and transformations in modern and contemporary society. These tasks involve complex logical structures, reflecting the model's 's depth of reasoning and its ability to transfer knowledge within the social sciences domain.
For evaluating mathematical reasoning ability, we adopted the GSM8K dataset, which includes complex mathematical problems and allows for quantitative assessment of the model's accuracy and efficiency in solving these problems.
The A12's Reasoning Challenge (ARC) dataset is multiple-choice response dataset containing science exam questions for grades three through nine. The dataset is divided into two sections: an easy section (Arc-E) and hard section (Are-C) for questions that test the model's ability in scientific reasoning. We used the Arc-E section.
Physical Interaction QA (PIQA) is a dataset designed to test model's ability to predict how objects interact in the physical world. The questions are typically based on everyday physical scenarios.
Social Interaction QA (SIQA) is question question-answerinn answering benchmark designed to assess a model's 's ability to reason about social commonsense intelligence. SIQA focuses inferring human actions and their social implications. The behaviors in SIQA span a wide range of social contexts, with answer choices including both human-crafted answers and adversarially filtered machine-generated options. Social IQa contains over 37,000 QA pairs, used to evaluate a model's ability to reason about the social implications of everyday events and situations.
Through this multidimensional evaluation, we assessed the proposed method's adaptability to both general and domain- specific tasks comprehensively, offering thorough examination of the model's capabilities across diverse multi-task settings and providing solid empirical support for future research.
B Baseline, Configurations and Backbone LLMs
B.1
Baseline
P-Tuning (Liu et al., 2024): P-Tuning employs a prompt encoder automatically optimize continuous prompt embeddings, avoiding the limitations of manually designing discrete prompts. By flexibly inserting anchor tokens into the input sequence, P-tuning can effectively improve the alignment accuracy of the semantic space, achieving better task adaptation performance.
2. Prefix Tuning (Li and Liang 202T): This method injects trainable prefix parameter matrices at the input of each layer of the Transformer. Through a hicrarchical representation rectification mechanism, the prefix parameter matrices can guide the model output while keeping the backbone network parameters frozen. This approach allows for efficient adaptation to new tasks while retaining the pre-trained knowledge.
3. AdLoRA (hhag et al.): Adaptive LoRA is based on a dynamic allocation algorithm that assesses parameter importance. It applies high-rank decomposition configurations to critical weight matrices and employs low-resource allocation strategies for non-core modules. This breaks the traditional paradigm of uniform parameter allocation, adaptively adjusting parameters for different parts to reduce computational and storage overhead while maintaining peraoiinnnerfoaaaannan performance.
4. LoRA (Hu et al., 2022)> LoRA (Low-Rank Adaptation) is an efficient fine-tuning technique for adapting pre-trained language models to downstream tasks. It aims to reduce the computational and storage overhead while maintaining the performance of the adapted model. The core idea of LoRA is to introduce low-rank update matrices to the pre-trained model. Instead of fine-tuning all the parameters of the model, LoRA keeps the original pre-trained weights frozen and injects trainable low-rank matrices into each layer of the model.
5. 5.HydraLoRA (Tian et al. 2024): HydraLoR.. LoRA is parameter-efficient fine-tuning method based on the LoRA mechanism, which aims to improve the performance of large-scale language models (LLMs) in multi-task learning and cross-domain adaptation. This method involves the decomposition
22

---

of the LoRA module into multiple independent expert modules, thereby enhancing the original symmetrical LoRA structure into single and multi-B LoRA structure. This modification enables the model to respond to varying task requirements with greater flexibility.
B.2 Experiment Configurations
The experiment configurations are as follows: learning rate is 0.0002, random seed is 614, and epoch is 1. The hardware and software configurations used in our experiments are as follows.
CPU: Intel(R) Xeon(R) Platinum 8468V, 800MHZ, 48cores; GPU: NVIDIA TESLA H800 80 GB; Operating system: Ubuntu 20.04; Deep learning framework: Pytorch 1.13.1.
B.3 Backbone LLMs
1. T5-large (Raffel al. 2020): T5-large is pre trained language model based on the Transformer architecture, which has wide applications in the field of natural language processing. T5 stands for "Text-to- Text Transfer Transformer", which converts all NLP tasks into text-to- format for processing. T5-large has over 7.7 million parameters, and compared to smaller T5 model variants, is able capture more complex patterns and semantic relationships in text. It performs well in tasks such as text generation, text classification, question answering systems, machine translation, etc. It can generate smooth and natural text, accurately understand context, and demonstrate good adaptability and universality in multiple language environments.
2. LLaMA2-7b: LL is large-scale language model launched by Meta, with approximately billion parameters. It has excellent language comprehension and generation capabilities, and can handle various natural language tasks such text generation, question answering, translation, sum- marization, etc. It performs well in multiple languages and can widely applied in various scenarios, providing users with convenient text processing services. Its open-source and free commercial license characteristics also promote its application and research in many fields.
3. Qwen2-7b 7b (qwe, 2024): Qwen2-7b is an open-source model launched by the Alibaba Cloud Tongyi Qianwen team, which supports multimodal input of text, images, and videos. It has 27 language processing capabilities and 32K long context understanding, and performs better than models of the same scale areas such as code generation and mathematical reasoning. Adopting efficient group query attention (GQA) technology, adapted to various scenarios from mobile to server, and open sourced under Apache 2.0 protocol, it is commercially available for free.
4. Qwen1.5- 14b (Bai et al. 2023): Qwenl.5-14b is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. It is based on the Transformer architecture with SwiGL.U activation, attention QKV V bias, group query attention, mixture of sliding window attention and full attention, etc.
C Limitations
Our study has several limitations. The reliance on LoRA's framework somewhat limits its extensi- bility. Additionally, during the first stage of fine-tuning, the introduction of discrete noise does not specifically consider the distribution of noisy data, which, to certain extent, increases the uncertainty. This presents a potential direction for future exploration, where more noise modeling techniques could be developed.
D Additional Experiments
D.1 Robustness Analysis under Varying Basic Noise Ratios
To investigate the robustness of LoPE when fine-tuning on datasets with higher inherent noise, we systematically evaluated basic noise ratios of 3.5%, 5%, 8%, 10%, 20%, and 30%. As shown in Table although overall performance gradually declines as the noise level increases, does not fully collapse, demonstrating that LoPE effectively delays performance degradation under high- noise conditions. This behavior can be attributed several factors. First, the overall quality of fine-tuning datasets impacts final inference performance. When the noise reaches certain level, it
23

---

Table 8: Performance under different noise ratios (transposed version).
Dataset / Noise Ratio
3.5%
5%
8%
10%
20%
30%
30% (HydraLoRA)
MMLU
44.59
44.42
44.15
43.81
42.35
40.78
37.96
PIQA+SIQA
63.31
62.66
61.86
61.27
60.14 14
59.50
56.83
GSM8K
15.31
13.72
11.83
9.25
8.04
7.60
7.12
blurs the boundary between clean data and noise, and fine-tuning becomes harmful, highlighting the importance of fine-tuning data quality. As result, for tasks requiring high precision, such as mathematics datasets, the rate of accuracy decline actually slows at higher noise levels. This may be because mathematical pattem-related content in the fine-tuning dataset has already been corrupted, resulting in less impact on the model itself, and the model may no longer recognize that it is learning mathematics-related knowledge.
D.2 Specific accuracy for sub-task
The specific accuracy for each task can be found in Table 9 and Table 14
Table 10
Table
11
Table 12
Table 13
Table 9: Performance across each specific task in NSET for LoPE and HydraLoRA (Orig).
Task
Model
NSET
LoPE
HydraLoRA
College Biology
44.44
43.75
College
Chemistry
32.00
30.00
College Computer Science
37.00
37.00
College Mathematics
38.00
34.00
College Physics
16.67
18.63
Electrical Engincering
42.76
45.52
Astronomy
40.79
42.76
Abstract Algebra
32.00
30.00
Machine Learning
38.39
36.61
Computer Security
58.00
58.00
High School Chemistry
35.96
32.51
Elementary Mathematics
29.89
27.51
High School Physics
28.48
29.14
High School Computer Science
37.00
37.00
High School Statistics
30.56
25.00
High School Biology
47.42
49.35
High School Mathematics
29.26
28.89
Conceptual Physics
44.26
42.13
Table 10: Performance across each specific task in SBH for LoPE and HydraLoRA(Origg)
Task
Model
SBH
LoPE
HydraLoRA
Moral Scenarios
27.71
24.69
Professional Psychology
43.14
43.63
World Religions
66.67
66.08
Philosophy
59.49
58.84
Moral Disputes
49.13
51.16
High School Government and Politics
66.84
68.39
Sociology
61.19
64.18
High School Psychology
61.47
61.28

---

Table 11: Performance across each specific task in History for LoPE and HydraLoRA(OOrig)
Task
Model
History
LoPE
HydraLoRA
High European History
60.61
63.64
Prehistory
47.22
47.22
High School World History
62.45
59.49
High School US History
56.37
53,43
Table 12: Performance across each specific task in NSET for LoPE and HydraLoRRA(Nois))
Task
Model
NSET
LoPE
HydraLoRA
College Biology
45.14
43.75
College Chemistry
31.00
29.00
College Computer Science
38.00
33.00
College Mathematics
33.00
40.00
College Physics
17.65
15.69
Electrical Engineering
38.62
33.79
Astronomy
42.11
38.16
Abstract Algebra
32.00
34.00
Machine Learning
38.39
37.50
Computer Security
53.00
52.00
High School Chemistry
33.50
33.00
Elementary Mathematics
26.72
27.51
High School Physics
31.13
27.81
High School Computer Science
41.00
38.00
High School Statistics
21.30
23.61
High School Biology
46.13
47.10
High School Mathematics
28.52
29.63
Conceptual Physics
40.85
41.70
Table 13: Performance across each specific task in SBH for LoPE and HydraLoRA((Nois))
Task
Model
SBH
LoPE
HydraLoRA
Moral Scenarios
26.03
25.92
Professional Psychology
44.28
44.12
World Religions
63.74
64.33
Philosophy
56.27
56.91
Moral Disputes
48.84
48.55
High School Government and Politics
65.28
64.25
Sociology
58.21
56.22
High School Psychology
60.37
58.53
Table 14: Performance across each specific task in History for LoPE and HydraL.RRA(N
Task
Model
History
LoPE
HydraLoRA
High European History
63.61
61.21
Prehistory
49.38
47.53
High School World History
59.92
59.07
High School US History
54.41
55.39
25

---

ESNE Visunlization of A Mateix Duft. LoRAs
1NN Visualiaction of B. Matzix Dift. LoRAs
50
-50
-50
-50
50
-50
50
Figure 5: Comparison of the average parameter changes of matrices A and B during first-stage fine-tuning, with and without HyNolse. Specifically, A and correspond to the matrix and poisoning expert Bn trained without HyNolse, while A' and B' represent the respective matrix A and poisoning expert Bp trained with HyNolse injected.
D.3 Matrix Parameter Visualization
We hypothesize that matrix A captures fundamental language-environment knowledge, which is both basic and universal, and minimally affected by noise (a dataset with added noise can be considered analogous to dataset from and without HyNolse data, observing the changes in matrix parameters assess whether harmful noise is absorbed into matrix A. Figure 5 shows that the average parameter change of matrix A, with or without HyNolse, is much smaller than that of matrix B under the same operations. This finding suggests that matrix A primarily encodes universal contextual features present across all data, while the influence of noise on matrix A is substantially smaller than on matrix
domain). To investigate this, we conducted Stage training with
L-NE Vasualention of A-Mazii Ddl. LaRAs
&SNE Vanuciuation of B-Matrix Diff LoRAs
A
Br
Figure 6: A denotes the parameter visualization of matrix A during the first-stage fine-tuning, while A' represents the updated matrix A in the second stage. B indicates randomly selected normal expert in the second-stage fine-tuning, and Bn corresponds to the poisoning expert from the first stage. This visualization highlights the parameter evolution clean and poisoned experts across different training phases.
We further visualize the parameter distributions of matrices A and B across the two-stage fine-tuning visualization space changes very little
process. shown Figure [ the position of matrix throughout training. This observation that matrix A primarily encodes general, universal knowledge. In contrast, we observe significant positional shifts between randomly selected normal expert matrix B and the poisoning expert matrix Bp across the two stages. We attribute this phenomenon to matrix
26

---

capturing task-specific knowledge, while Bp absorbs noise- contaminated patterns, fundamentally differing from the clean representations encoded by other experts.
D.4 Evaluation against Task-Specific Denoising Baseline
We have also included a dedicated comparison experiment against a task-specific denoising model further assess LoPE's generality in downstream applications. Specifically, we investigated the representative denoising method LeCoRE, which is designed for conversational search tasks. Follow- ing its evaluation protocol, we conducted experiments under comparable settings. The results are summarized in Table [5]
Table 15: Performance comparison between LoPE and the task-specific denoising model LeCoRE.
Method
MRR
NDCG@3
R@ 10
R@ 100
LcCoRE LoPE
51.1
48.5
73.9
89.7
54.7
50.1
75.6
91.3
27


Reviews:
[{"review_id": "yEOq3FZlDl", "reviewer": "Reviewer_tqFZ", "summary": "This paper introduces LoPE (asymmetric LoRA poisoning experts), a pretty clever approach to handling noisy data during fine-tuning. Instead of trying to clean noise, they deliberately inject it. They use an asymmetric LoRA structure with a dedicated \"poisoning expert\" trained to handle noise. During inference, they simply mask this expert out. Their two-stage process is neat: first train the poisoning expert on noisy data, then freeze it while training other components. They ran experiments across several benchmarks and showed some improvements over other PEFT methods.", "questions": "See above.", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The \"noise through noise\" approach represents a creative departure from conventional denoising methods.\n\n2. Their use of asymmetric LoRA architecture to separate functionality between experts is technically sound and well-executed.\n\n3. The method shows promising results across multiple tasks, with notable performance on mathematical reasoning and complex reasoning tasks.\n\n4. The DyCompEnSate approach demonstrates careful consideration of the knowledge gaps that could arise from expert masking.\n\n## Weaknesses\n\n1. This paper mostly compares against other PEFT methods like LoRA and HydraLoRA, but I'm missing comparisons with actual denoising methods (LeCoRE, MICL, etc.). I'd like to see them pit their method against mainstream denoising techniques - that would make their case much stronger.\n\n2. This paper primarily tests with 5% noise injection, but then cite research saying real-world noise can be 8%-38.5%. \n   - Figure 4 shows different noise ratios but doesn't dig into why performance drops at higher levels.\n   - Would love to see how this holds up at 10%, 20%, 30% noise - does it collapse or gracefully degrade?\n\n3. It would be better that include visualization or analysis of its internal representations to describe what exactly is this poisoning expert learning.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns.", "rebuttal": "Thanks for your time and feedback! We admire your deep understanding of this work and your extremely valuable comments. The core of our work that you considered is absolutely correct, and all other reviewers agree with you. Below, we summarize and explain your concerns:\n\n***1. This paper mostly compares against other PEFT methods like LoRA and HydraLoRA, but I'm missing comparisons with actual denoising methods (LeCoRE, MICL, etc.). I'd like to see them pit their method against mainstream denoising techniques - that would make their case much stronger.***\n\n**Q1 Response:** Thank you for your valuable comment! In general, aside from the performance improvement, our main advantage is that we completely eliminate the need for data cleaning. In contrast, by adding noise at a low cost, we achieve the goal of \"Noise Robustness Through Noise.\" This not only overcomes issues like uneven data distribution caused by data cleaning but also provides broad inspiration for diverse scenario tasks that are challenging to model or solve directly. Since it is a general fine-tuning architecture widely applicable to various noise environments and data types, we consider that it does not conflict with other denoising methods and can be widely combined with them. Therefore, we primarily compared it with other PEFT methods of the same category. \n\nDespite the above discussion, we fully agree with your point and have conducted an investigation of the two representative denoising methods you recommended: LeCoRE and MICL. For LeCoRE, a method specifically designed for conversational search tasks, we followed their evaluation protocol and conducted experiments under comparable settings. The experimental results are presented below:\n\n\n\n| Method | MRR  | NDCG@3 | R@10 | R@100 |\n| ------ | ---- | ------ | ---- | ----- |\n| LeCoRE | 51.1 | 48.5   | 73.9 | 89.7  |\n| LoPE   | 54.7 | 50.1   | 75.6 | 91.3  |\n\nThe experimental results demonstrate that our method achieves superior performance across all evaluation metrics, indicating the generalizability of LoPE to domain-specific tasks. As MICL is a multi-modal model that presents challenges for direct comparison and the author does not have open-source code, we have not included its results at this stage. We commit to conducting comprehensive and scientifically rigorous evaluations of various denoising models across different scenarios, including LeCoRE and MICL, and will present the corresponding results and analyses in the camera-ready version. Thanks again!\n\n\n\n***2. This paper primarily tests with 5% noise injection, but then cite research saying real-world noise can be 8%-38.5%.***\n\n**Q2 Response:** Thanks for your comment! As mentioned in Chapter 4.2 of our original text, given that the original dataset already contains some inherent noise, we used an additional rate of 5% to create a more challenging environment, where they work together with the inherent noise in the dataset and collectively affect the model.\n\n\n**2.1 Figure 4 shows different noise ratios but doesn't dig into why performance drops at higher levels.**\n\n**Q2.1 Response:** Due to space limitations, we provided only a brief discussion of this phenomenon. Here, we present a more comprehensive analysis of the observed performance drop under high levels of HyNoIse.\n\nOur analysis identifies several key contributing factors:\n\n**(1) Negative Effects of Excessive Noise:** When the HyNoIse ratio exceeds 10%, we observe a significant drop in performance. This is because excessive noise impairs the router’s ability to **distinguish between clean and noisy data**, leading to confusion in expert weight allocation and ultimately weakening the model's discriminative capacity to a certain extent.\n\n**(2) Distortion of Data Distribution:** As the HyNoIse ratio increases, the statistical properties of noisy samples can overwhelm and potentially disrupt the underlying distribution patterns of the original data.\n\n**(3) Randomization of Weight Allocation:** As the HyNoIse level rises, the embedding vectors of noisy samples increasingly overlap with those of clean data. This expanded overlap prevents the router from effectively separating the “clean” and “noisy” paths, resulting in an almost uniform distribution of expert weights. Consequently, the originally designated \"poisoning expert\" (BD) is forced to process clean data, while the normal experts (e.g., B1/B2) are exposed to noisy samples, undermining their role as independent repositories of \"clean knowledge.\"\n\nTo aid understanding, we offer an analogy (imperfect, but conceptually helpful):\n\n**Low HyNoIse:** Similar to giving the model a “**vaccine**” — a small dose of virus (noise) that stimulates immunity (robustness).\n\n**High HyNoIse:** Comparable to injecting a large amount of **virus** directly, overwhelming the immune system (model) and compromising normal functionality.\n\n\n\n**2.2 Would love to see how this holds up at 10%, 20%, 30% noise - does it collapse or gracefully degrade?**\n\n**Q2.2 Response:** Thank you for your interest! We have conducted extensive supplementary experiments in this regard and discovered some interesting phenomena. We systematically evaluated noise ratios of **3.5%, 5%, 8%, 10%, 20%, and 30%** in the noisy data. Partial results are shown below.\n\n| Noise Ratio in Nois                 | MMLU  | PIQA+SIQA                              | gsm8k |\n| ----------------------------------- | ----- | -------------------------------------- | ----- |\n| 3.5%                                | 44.59 | 63.31(*Table 2 in the original paper*) | 15.31 |\n| 5%(*Table 1 in the original paper*) | 44.42 | 62.66                                  | 13.72 |\n| 8%                                  | 44.15 | 61.86(*Table 2 in the original paper*) | 11.83 |\n| 10%                                 | 43.81 | 61.27                                  | 9.25  |\n| 20%                                 | 42.35 | 60.14                                  | 8.04  |\n| 30%                                 | 40.78 | 59.50                                  | 7.60  |\n| 30%(*HydraLoRA*)                  | 37.96 | 56.83                                  | 7.12  |\n\nFrom the results, it is evident that while overall performance does decline with increasing noise levels, **it does not completely collapse**.\nWe analyze the following reasons:\n\n**(1)** Our method delays performance collapse under higher noise conditions.\n\n**(2)** The overall quality of fine-tuning datasets impacts final inference performance. When the noise reaches a certain level, it blurs the boundary between clean data and noise, and fine-tuning becomes harmful, highlighting the importance of fine-tuning data quality.\n\n**(3)** When data noise rises to a certain degree, the accuracy decline rate for mathematics datasets requiring high precision actually slows down. This may be because mathematical pattern-related content in the fine-tuning dataset has already been corrupted, resulting in less impact on the model itself (the model may no longer recognize that it is learning mathematics-related knowledge).\n\nIn the camera-ready version, we will present the complete set of these supplementary experiments to provide a comprehensive view of our work and enrich the research community's understanding of LLM data noise.  We will include comprehensive results across various noise ratio settings in the camera-ready version, along with an in-depth analysis of the observed performance variations and the underlying differences between real-world noise and synthetically injected training noise in the corresponding section.\n\n\n\n***3. It would be better that include visualization or analysis of its internal representations to describe what exactly is this poisoning expert learning.***\n\n**Q3 Response:** Thank you for your valuable comment! Following your advice, we have conducted a comprehensive visualization of the A and B parameters in LoPE, which follows the LoRA matrix visualization method released in the HydraLoRA GitHub repository.  Although we cannot provide visualizations here due to the format limitation of the rebuttal, we list the key data and conclusions here. Overall, our findings show that **during the entire two-stage fine-tuning process, the position of matrix A in the visualization space changes very little**, which is fully consistent with our previous conclusions in the manuscript, and also aligns with the HydraLoRA paper’s conclusion that matrix A represents general knowledge. \nIn contrast, we observed significant positional changes between the standard expert B matrix and the poisoning B matrix across the training stage. We attribute this phenomenon to the B matrix capturing knowledge contaminated by noise, which is fundamentally **different from the clean knowledge encoded by other experts**.  This aligns with our hypothesis that the poisoning expert effectively diverts tokens affected by noise. We will add the complete visualization process and content mentioned above in the camera-ready version and provide corresponding theoretical analysis and explanations. Thank you again for your insightful suggestion!\n\n\n\nThank you again for taking the time to review our response. We sincerely hope that our response can solve your concerns and better evaluate the comprehensive value of our work. Thanks again!"}, {"review_id": "T80ViAoVgG", "reviewer": "Reviewer_vbzw", "summary": "This paper presents a method called LoPE for making large language models more robust to noisy data during fine-tuning. LoPE is designed based on HydraLoRA which is an existing framework that uses asymmetric LoRA adapters. The key idea is to assign a poisoning expert to learn from noisy data. The training includes two steps: 1) the poisoning expert is trained using mixed types of noise, while the other adapters are kept unchanged; 2) the normal adapters are trained on clean data while keeping the poisoning expert frozen. When making predictions, the poisoning expert is turned off so that only the clean experts affect the results. The authors show experimental results on several tasks like MMLU and QA to support their claims.", "questions": "See weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The experiments employ state-of-the-art LLMs as backbones. \n2. Ablation studies are conducted to understand the effectiveness of each module. \n\nWeaknesses:\n1. The problem setting is somewhat unclear. Based on the descriptions in the Introduction and Experiment sections, my understanding is that the fine-tuning data (training data) are assumed to be noisy—for example, by manually injecting 5% noise to simulate real-world scenarios. However, LoPE appears to use both noisy data (in stage 1) and clean data (in stage 2) during training. Since the noisy data are synthetically generated from the clean data using specific strategies in HyNoIse, the actual input to LoPE is still the clean dataset. This makes the problem setup and the role of noise in the training pipeline somewhat confusing.\n2. The overall framework is largely built on HydraLoRA. While the extension to noisy data training is appreciated, the core technical contribution is a bit incremental. \n3. The results reported for HydraLoRA on MMLU (46.10 at rank r=4) are not representative of the best-known baseline performance. The HydraLoRA paper reports 47.22 at r=8, which makes the performance comparison in Table 1 of this submission less convincing.\n4. The motivation and effectiveness of the two-stage fine-tuning strategy are not well explained. In particular, it is unclear why the \"poisoning expert\" B_D​, which is trained to handle noise, is ignored during inference. Is it because you assume the test data are relatively clean? \n5. The paper criticizes prior methods for being limited to specific noise types, yet the HyNoIse strategy also relies on predefined types of discrete and continuous noise. As such, it does not effectively address the core issue it raises. More evidence is needed to support the claim that LoPE generalizes across diverse noise patterns.\n6. All compared baselines are general LoRA variants, none of which are explicitly designed or adapted for noisy data. This weakens the empirical claims. \n7. The presentation of the paper requires significant improvement. It contains several unclear expressions and awkward phrasing. Some phrases like “generate output with relatively clean knowledge” are vague. A thorough language and writing revision is strongly recommended to improve readability and ensure the technical content is clearly communicated.", "was_revised": true, "final_justification": "Authors's responses have clarified my major concerns about the method design. I find the proposed method interesting and insightful, and thus lean toward acceptance.", "rebuttal": "Thanks for your time and feedback, we deeply sense your expectation for us to make this work better. In this paper, we boldly explored the 'Noise Robustness Only Through Noise' approach and innovatively proposed LoPE. By combining the advantages of multi-stage differentiated training under the PEFT framework, we completely eliminate the requirement of data cleaning. Furthermore, the proposed LoPE can be widely applied to the parameter-efficient fine-tuning of large language models, enhancing its potential to improve model performance with limited computational resources.\n\nAdditionally, we consider that the proposed poisoning expert elimination strategy is not only applicable to noise-related tasks but also offers new insights for diverse scenario tasks that are challenging to model or solve directly.\n\nIn the main aspects, including motivation, paper quality, experimental results, and reproducibility, our work was fully approved by all other reviewers.\n\nThrough your comments, we noticed that there are some misunderstandings about our work. In the following, we will summarize and explain your concerns point by point in order to clarify them.\n\n**R1:**\n\nThanks for your comment. In our experiments, Orig and Nois represent two independent experimental workflows, not stages within a single pipeline. Each follows the same full two-stage fine-tuning and evaluation process. Specifically, Orig is the original dataset (with inherent noise), while Nois augments it with 5% additional discrete noise to create a more challenging and comparable setting. The clean data (in stage 2) you mentioned actually corresponds to the datasets used in each setting.\nIn contrast, the Stage 1 data generated via HyNoIse is additional noise specifically crafted to train the poisoning expert, which is distinct from the Nois setting created for a more challenging and comparable evaluation.\n\nWhether fine-tuning on Orig or Nois, we follow the same mechanism: \n\nStage I: We use the fine-tuning dataset augmented via HyNoIse to exclusively train only the poisoning expert, while freezing other experts, ensuring that noise adaptation is only highly concentrated in it. \n\nStage II: We freeze the poisoning expert to preserve its noise adaptation and train the normal experts and router on the dataset without further HyNoIse (i.e., the Orig or Nois dataset itself). Consequently, the learned routing distribution increasingly assigns noisy samples to the poisoning expert and clean samples to others, achieving our intended noise isolation. \n\nCrucially, we use the HyNoIse only in Stage I to train the poisoning expert; it's never used in Stage II. In the inference stage, we mask the poisoning expert, so the final output is generated only from the normal experts.\n\nRegarding \"Since the noisy data that synthetically generated from the clean data, and many studies on noise injection or data augmentation rely on clean data to synthesize noise, we consider that this does not diminish their effectiveness. “Data Augmentation Approaches in Natural Language Processing: A Survey” explicitly states that “adding noise to the original data while preserving label validity can improve model robustness.” \n\nThanks again for your comment! Please feel free to reach out for any further discussion.\n\n**R2:**\n\nThanks for your comment. With PEFT paradigms becoming the mainstream framework for large language models, we recognize that HydraLoRA serves as the architectural foundation for many subsequent innovations, and we believe that building upon this architecture through novel designs should be encouraged, similar to how many works build upon the LoRA framework. Several of HydraLoRA-based works have been widely acknowledged by the community (e.g., TeamLoRA, ACL 2025). As mentioned below, our core contribution lies in proposing a novel, complementarity-inspired paradigm. We believe that it is not only applicable to noise-related tasks but also provides new insights for more scenarios that are difficult to model or solve directly.\n\n**R3:**\n\nThanks for your comment. As demonstrated in the experiments of the HydraLoRA paper, higher ranks generally lead to better performance. However, increasing the rank also incurs higher parameter and computational costs, making it necessary to balance scalability and effectiveness. We believe that a similar trend holds for our LoPE framework. \n\nWe have added results for LoPE with ranks 8 and 16, and compared them with HydraLoRA at the same ranks. The observed performance trends remain consistent with our previous conclusions, and we also expect a similar trend to hold at higher ranks. † denotes methods fine-tuned on Nois.\n\n| Methods           | MMLU  | PIQA  | SIQA  | GSM8K |\n| ----------------- | ----- | ----- | ----- | ----- |\n| HydraLoRA(r=8) †  | 43.98 | 75.30 | 48.12 | 12.44 |\n| LoPE(r=8) †       | 44.82 | 76.83 | 49.90 | 14.31 |\n| HydraLoRA(r=16) † | 45.20 | 76.92 | 49.63 | 12.95 |\n| LoPE(r=16) †      | 45.76 | 77.22 | 50.87 | 15.01 |\n\nNotably, HydraLoRA’s MMLU result (47.22%) was obtained using Databricks-Dolly-15k. However, we can't find a usable version in their repository. All of our experiments were fine-tuned on the Alpaca-52K dataset (see Section 4.1). We believe that differences in fine-tuning datasets naturally lead to variations in inference performance.\n\nWe will include both the results for ranks 4 and 8 as the main results in the camera-ready version, and provide a complete ablation study with corresponding analysis for different ranks in a later section. We thank you again for your valuable suggestion.\n\n**R4:**\n\nThank you for your comment. The poisoning expert’s core role is to leverage the complementary set concept to facilitate routing, rather than to contribute to output during inference. Please review the process details in R1. We deliberately mask the poisoning expert during inference because it acquires noise-related bias during Stage I, so its outputs inherently carry noise-related bias. If not masked, the model will be affected by the incorrect knowledge learned from noise, which in turn reduces the model's performance. This phenomenon is validated in Section 5.2 of the manuscript (Table 4 and lines 330).\n\nWe also believe that LLMs during training typically rely on large-scale datasets of varying quality, inevitably containing noise, whereas the testing phase usually employs high-quality data for specific tasks. This fact is well supported by “Learning from noisy labels with deep neural networks: A survey.” We will include this reference in the camera-ready version. Thanks again!\n\n**R5:**\n\nWe clarify our statement in line 42 as follows in our paper. The term “noise type” refers to the scenario-specific noise patterns and characteristics of each dataset in the context of its corresponding task, rather than to a classification of noise as discrete or continuous.\n\nIn contrast, LoPE is not tailored to any specific downstream task or domain, and can thus be widely applied to the parameter-efficient fine-tuning of large language models. In our main results, LoPE has already been validated on over 60 diverse NLP tasks, demonstrating its broad applicability. Its design does not rely on any fixed or pre-defined noise type; instead, it incorporates noise at the character, token, label, and structural levels—covering common real-world perturbations (Section 3.2.1).\n\nAdditionally, we have included a dedicated comparison experiment against a task-specific denoising model to further evaluate LoPE’s generality in downstream applications. We have conducted an investigation of the representative denoising method LeCoRE (specifically designed for conversational search tasks). We followed their evaluation protocol and conducted experiments under comparable settings. The experimental results are presented below:\n\n| Method | MRR  | NDCG@3 | R@10 | R@100 |\n| ------ | ---- | ------ | ---- | ----- |\n| LeCoRE | 51.1 | 48.5   | 73.9 | 89.7  |\n| LoPE   | 54.7 | 50.1   | 75.6 | 91.3  |\n\n**R6:**\n\nThank you for your valuable comment! Lope is a general fine-tuning architecture widely applicable to various noise environments and data types. We consider that it does not conflict with other denoising methods and can be widely combined with them. Therefore, we primarily compared it with other PEFT methods of the same category.\n\nDespite the above discussion, we fully agree with your point and have conducted an investigation of the two representative denoising methods. The experimental results are presented in R5. We will include more comparative experimental results and corresponding analyses in the camera-ready version.\n\n**R7:**\n\nThank you for your comment. Regarding the phrase “generate output with relatively clean knowledge” in the caption of Figure 2, although we hope that the model can produce clean outputs, in practice, it is difficult for the model to consistently generate perfectly clean and ideal results, which is why we avoided using absolute terms. Nonetheless, we sincerely appreciate your careful review and will refine this expression in the camera-ready version. We will also conduct a thorough review of the entire paper to ensure clarity throughout. Thank you again!\n\nFor all typos, grammatical errors, and unclear notations, we will carefully revise them one by one. Despite these writing issues, we sincerely argue for the value of this work. As mentioned earlier, we have completely eliminated the need for manual noise cleaning by adopting the opposite approach of noise injection. Moreover, we believe the proposed poisoning expert mechanism offers broad inspiration for diverse scenario tasks in the future. Based on the feedback from all the other reviewers, our paper is well-worked and of great value to the academic community. Therefore, we warmly welcome continued discussion and exchange, and we sincerely hope you can take the comprehensive value of this work into consideration. Thanks again!"}, {"review_id": "okbxMTqiJk", "reviewer": "Reviewer_CYJM", "summary": "This work introduces LoPE, a novel fine-tuning framework that strengthens the robustness of LLMs to noisy data. LoPE employs a dedicated \"poisoning expert\" within an asymmetric LoRA architecture. Through a two-stage fine-tuning process with hybrid noise injection and expert masking during inference, LoPE effectively minimizes the impact of noise without requiring data cleaning or architectural modifications, achieving strong performance.", "questions": "1. The role of the shared matrix A remains somewhat unclear, particularly regarding the rationale for fine-tuning it in both Stage I and Stage II. It would be helpful if the authors could clarify the motivation behind this design choice and provide ablation studies to highlight the impact of updating A across both stages.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros: \nLoPE leverages the HyNoIse and DyCompEnSate components to isolate noise handling to specific experts and mask it during inference without data cleanup. Meanwhile, it is evaluated on different downstream tasks, highlighting its efficiency and robustness\n\nCons: \n1. Sharing and fine-tuning the A-matrix in both Stage I and Stage II raises the question of whether harmful information learned during the first stage (from noisy data) might be absorbed into the A-matrix, potentially leading to performance degradation.\n2. HyNoIse is primarily a character-level noise and may not reflect real-world, high-complexity datasets. Is it possible for the authors to provide analysis of the performance in a real task?\n3. Missing hyperparameter analysis of β.\n4. The first paragraph of Section 1 lacks a relevant citation;\n5. For the without noise performance in Table 1, why does LoPE improve compared to HydraLoRA, if I understand correctly, LoPE lies mainly in the two stages of fine-tuning the noise data, but why would this improve the Orig dataset as well?", "was_revised": true, "final_justification": "Thanks for your detailed reply. At this stage I think my concerns are addressed except minor points.", "rebuttal": "Thank you very much for your recognition of the innovation and overall quality of our work. We are very impressed by your meticulous and insightful comments, through which we deeply sense your expectation for us to make this work better. The core of our work that you considered is absolutely correct, and almost all other reviewers agree with you. Below, we summarize and explain your concerns point by point:\n\n***1. Sharing and fine-tuning the A-matrix in both Stage I and Stage II raises the question of whether harmful information learned during the first stage (from noisy data) might be absorbed into the A-matrix, potentially leading to performance degradation.***\n\n**Response:** Thank you for your valuable comment! Here is our theoretical analysis regarding this question: Based on the further exploration of the knowledge of matrix A in the HydraLoRA paper, we believe that matrix A learns relatively language environment knowledge, which is very basic and universal, and very little affected by noise(a dataset with added noise can be analogized as a new domain dataset). In addition, we have conducted two experiments in the past few days to address your concerns. Firstly, we conducted a comprehensive visualization of the A and B parameters in LoPE to present the actual changes in each part of the process. Due to format limitations of the rebuttal, we will first provide the key visualization results, and we will include all the figures and a complete analysis in the camera-ready version. Our findings show that during the entire two-stage fine-tuning process, **the position of matrix A in the visualization space changes very little**, which is fully consistent with our previous conclusions in the manuscript, and also aligns with the HydraLoRA paper’s conclusion that matrix A represents general knowledge. And we further conducted experiments by freezing the matrix A obtained after the first stage. The table presents our new findings and reveals an interesting phenomenon: we observed a significant drop in accuracy. Through careful investigation, we identified the root cause and are happy to share and discuss it with you as follows:\n\n| Method                 | MMLU  | PIQA  | SIQA  | GSM8K |\n| ---------------------- | ----- | ----- | ----- | ----- |\n| LoPE(Freeze A in Stage I)         | 43.93 | 74.81 | 46.67 | 11.20 |\n| LoPE(Original Table 1) | 44.42 | 76.28 | 49.03 | 13.72 |\n\n\nOur analysis shows that **freezing matrix A immediately after its initialization in the first stage will hinder convergence and reduce performance**. This finding is consistent with observations from the ablation study in \"LoRA: Low-Rank Adaptation of Large Language Models\", which documented a similar effect. However, our freezing approach does not result in severe negative impacts, as we still have the second stage of training that unfreezes and normally trains matrix A, though freezing A in Stage I does significantly affect the speed of the first stage.\n\nBased on these results, we draw two key conclusions:\n\nFirst, as noted above, matrix A exhibits stability—its fundamental structure remains largely unchanged throughout training. Furthermore, as demonstrated in HydraLoRA, the general knowledge encoded in matrix A is resistant to noise-induced perturbations.\n\nSecond, and perhaps more importantly, our two-stage training paradigm actually enhances the robustness of matrix A. The dynamic learning process does more than just preserve its initial state; it reinforces its representation of noise while maintaining its core knowledge structure over both stages. This indicates that allowing matrix A to adapt during training, rather than freezing it prematurely, is crucial for achieving optimal noise robustness.\n﻿\n\n\n***2.HyNoIse is primarily a character-level noise and may not reflect real-world, high-complexity datasets. Is it possible for the authors to provide analysis of the performance in a real task?***\n\n**Response:** Thank you for your valuable comment! As described in Section 3.2.1, HyNoIse represents a comprehensive noise injection method that extends beyond character-level perturbations. It encompasses **character-level, token-level, label-level, and structural-level** typical noise patterns, which comprehensively cover the common anomalies we have surveyed in real-world scenarios, including **continuous noise in the embedding level**, designed to simulate diverse real-world noise conditions.  I understand your concern about the performance of real-world tasks as we do. Therefore, our evaluation employs a carefully curated suite of benchmark datasets: MMLU, PIQA, SIQA, and GSM8K, collectively spanning mathematics, commonsense reasoning, chemistry, history, law, and numerous other domains.  Here are a few examples to illustrate its properties:\n﻿\n\n- MMLU (Massive Multitask Language Understanding) spans tasks across multiple domains, enabling comprehensive evaluation of models' language understanding capabilities across diverse knowledge areas.\n- PIQA (Physical Interaction QA) is designed to evaluate models' ability to predict physical object interactions and commonsense physical understanding.\n- SIQA (Social Interaction QA) serves as a benchmark for evaluating social commonsense reasoning and interpersonal intelligence.\n- GSM8K provides grade school math problems requiring multi-step mathematical reasoning.\n\nAnd we also added experiments on the tasks mentioned in the related work as follows:\n\n| Method | MRR  | NDCG@3 | R@10 | R@100 |\n| ------ | ---- | ------ | ---- | ----- |\n| LeCoRE | 51.1 | 48.5   | 73.9 | 89.7  |\n| LoPE   | 54.7 | 50.1   | 75.6 | 91.3  |\n\nThe experimental results demonstrate that our method achieves superior performance across all evaluation metrics, indicating the generalizability of LoPE to real tasks. \n\nWe believe that these widely adopted benchmarks collectively provide a comprehensive assessment of our model's practical capabilities across different cognitive tasks and knowledge domains. In addition, we will add more performance and corresponding analysis of LoPE in specific downstream natural language processing tasks in the Camera-Ready version. Thanks again!\n\n***3.Missing hyperparameter analysis of β.***\n\n**Response:** Thank you for your valuable comment! We are deeply sorry that there is an ambiguous notation regarding β in Equation (3)... To clarify, β serves as a normalization factor rather than a hyperparameter. Since the sum of the subsequent coefficients exceeds 1, we introduce β to ensure that\n$$\n\\beta(\\omega_D+\\sum_{i = 1}^{K}(1+\\theta_{iD})\\omega_i)=1,\n$$\n\nthereby maintaining consistent scaling across all coefficients (the same principle applies to Equation (4)). This normalization ensures that all components adhere to the same scale. We will revise this notation and provide a clearer explanation in the revised manuscript.\n\n***4. The first paragraph of Section 1 lacks a relevant citation***\n\n**Response:** Thank you for your valuable comment!  We have added the relevant citations to the introduction section as suggested. We greatly appreciate your guidance in this regard.  We will carefully investigate and add other related work in the Camera-Ready version. If you believe there are other important works relevant to our research contributions that should be cited, we would be very grateful for your valuable suggestions. Thanks again!\n\n***5. For the without noise performance in Table 1, why does LoPE improve compared to HydraLoRA, if I understand correctly, LoPE lies mainly in the two stages of fine-tuning the noise data, but why would this improve the Orig dataset as well?***\n\n**Response:** Thank you for your valuable comment! Our primary motivation centers on handling data containing substantial noise. After completing the noisy data experiments, we recognized that even \"original\" datasets **inherently contain noise**, albeit potentially at lower levels. As documented in \"*Learning from noisy labels with deep neural networks: A survey*\" (cited in our manuscript), no real-world dataset achieves perfect cleanliness.\n\nWhile our framework treats original data as \"zero-noise\" theoretically for comparison with our artificially noise-injected datasets, this represents a relative rather than an absolute characterization. The reality of large-scale datasets is that complete clean remains unattainable—**an inherent limitation of current data collection and annotation practices**. Consequently, LoPE demonstrates performance improvements even on unprocessed original datasets because it effectively handles the **latent noise present in ostensibly \"clean\" data**. This observation underscores the practical utility of our approach beyond artificial noise scenarios.\n\nThank you again for taking the time to review our response. We sincerely hope that our response can solve your concerns and better evaluate the comprehensive value of our work. Thanks again!"}]

Author rebuttal:

Meta-review: In this paper the authors introduce LoPE, a novel method for robust fine-tuning on noisy data using a dedicated "poisoning expert" within an asymmetric LoRA architecture. The idea is that during training, since the poisoning expert adds noise, other experts are encouraged to generate nose resistant representations which during inferencing become slightly stronger since the poison expert is masked. This is an interesting approach in my opinion. Reviewers praised this "noise through noise" approach as creative, technically sound, and promising. There were concerns about the lack of comparison with dedicated denoising methods, limited testing at higher noise ratios, an unclear problem setup, and about the method's success being dependent the same synthetic noise for both training and evaluation.

Authors, in their response, while addressing the concerns, have made clarifications, presented new experiments showing LoPE outperforms a standard denoising method and degrades gracefully at noise levels up to 30%, and show that inconsistent noise types to prove the method's robustness. 

Overall this paper will be of interest to the community.
Decision: Accept (poster)