Whose Instructions Count? Resolving Preference Bias in Instruction Fine-Tuning
Authors: Jiayu Zhang
Changbang Li
Yinan Peng
Weihao Luo
Peilai Yu
Xuan Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Instruction Fine-Tuning
Preference Bias
Self-Supervised Learning
Primary area: deep_learning

Abstract: Instruction fine-tuning (IFT) has emerged as a ubiquitous strategy for specializing large language models (LLMs), yet it implicitly assumes a single, coherent "ground-truth" preference behind all human-written instructions. In practice, annotators differ in the styles, emphases, and granularities they prefer, introducing preference bias that can erode both robustness and generalization. We propose Dynamic Cross-Layer Preference Correction (\textsc{DCPC}), it couples (i) a preference-sensitive similarity estimator that detects mismatched instructional cues, (ii) cross-layer prefix alignment to reconcile semantic representations across transformer layers, and (iii) a lightweight Preference Correction Module (PCM) that dynamically adjusts hidden states to honor the inferred dominant preference.   On five Super/GLUE tasks and the Alpaca set—plus six preference-shifted variants—DCPC boosts accuracy/F1-EM by 4.0–6.7 points and gpt-score by +0.7, while cutting inter-seed variance up to 35% on LlaMA-2 13B and Mistral-7B, setting a new state of the art for robust instruction tuning.

Whose Instructions Count? Resolving Preference Bias in Instruction Fine-Tuning
Jiayu Zhang! Changbang Li Yinan Peng', Weihao Luo", Peilai Yuς, Xuan Zhang!.7 1.7t
Airon Technology CO., LTD
'Peking University
'University of Pennsylvania
'Hengxin Technology Ltd. 'Donghua University
'Ludwig Maximilian University of Munich CCameege Mellon University
Abstract
Instruction fine-tuning (IFT) has emerged ubiquitous strategy for specializing large language models (LLMs), yet it implicitly assumes a single, coherent "ground- truth" preference behind all human- numan-written written instructions. In practice, annotators differ in the styles, emphases, and granularities they prefer, introducing preference bias that erode both robustness and generalization. We propose Dynamic Cross-Layer Preference Correction (DCPC), it couples (i) a preference-sensitiee similarity estimator that detects mismatched instructional cues, (ii) cross-layer pre- fix alignment to reconcile semantic representations across transformer layers, and (iii) lightweight Preference Correction Module (PCM) that dynamically adjusts hidden states to honor the inferred dominant preference. On five Super/GLUP tasks and the ALPACA set-plus six preference-shifted variants-DCCI accuracy/F1 FF-MM// by 4.0-6.7 points and gpt-score by +0.7, while cutting inter-seed variance up 35% on LlaMA-2 13B and Mistral-7B. setting a new state of the art for robust instruction tuning.
boosts
Introduction
Large language models (LLMs) [Naveed et al., 2023 Shanahan 2024
Hu et al. 2025 Zhang et al.
2025b have become the backbone of modern natural language systems Zhang et al.
2023a Zhou
aal...2022)) and are now widely used healthcare Tong et al., 2025 Liu al. 2025
Wang et al.
2025a | autonomous systems (Yao et al. (2023) and other applications Jiang et al. 2025 Tao et al.
2023
Xu et al. 2025a Shen and Zhang 2025| He and Qul 2025 Wang al.) 2025b Sun and Li
2025
A common way to specialize these models for downstream use is instruction fine-tuning
(IFT)
[Zhang al. 2023c Ghosh al., 2024). where pretrained model is adapted using various
instruction datasets that provide task specific demonstrations [ et al. 2025b Chung et al. 2024
Su et al.
2022 Kaur et al., 2024 Tan al. 2025 Yao al. 2024 Zhang et al.
2025a].
Existing IFTss assumes that all instructions in the data reflect a single, coherent supervisory sig-
nal. [Ren et al. 2024 Shi et al. 2024 Diang et al. 2024 Edstedt al. 2024 Liu and Xiao 2025 In
practice, however, different annotators (or even the same annotator over time) exhibit distinct stylistic
and semantic preferences [Li al. 2025 Lu al.
[025b] -egg. terse versus verbose answers, formal
versus colloquial tone [Cheng and Cosley 2013 strict adherence to specification versus creative
elaboration [Zhao al. 2020, et al.j 2023] Chen al. [024b]] Xiao and Liu 2025). This will
make the training data highly imbalanced and extremely noisy [Sun et al.
2025
More information
often does not lead stable improvement in performance (Wen al. et al.
2023
Ke al. 2025
2025a Lan et al. 2025a
These latent disagreements introduce preference bias:
Equal contribution, *Corresponding author(kuannhang2299990
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Porformnne on Original Datasets
Performance after Preference Shift
PLANISOPSS SHAREOPT
SHARENFT
-3
FJ86T
Flliing
Figure Performance comparison Full fine-tuning (Full-FT), instruction fine-tuning baselines, and our DCPC across three benchmarks.
The same input can receive multiple, equally plausible but inconsistent target responses.
Why does preference bias matter? Figure[showe shows that when deliberately perturb instructional corpora with realistic preference shifts-mimicking heterogeneous crowdsourcing-thor performance of both full-pan parameter and leading IFT baselines plunges. The root cause that gradient updates are pulled in conflicting directions, forcing the model to memorize idiosynerasies instead of learning consensus. As result, generation becomes brittle: the model overfits to whichever style dominates locally in the mini-batches it sees.
Our approach. We introduce Dynamic Cross-Layer Preference Correction (DCPC), a self- supervised procedure that resolves these conflicts on the fly. Instead of treating each instruc- tion-response pair as an immutable oracle, DCPC: (i) estimates preference-sensinee between response embeddings to detect subtle stylistic clashes, (īi) aligns transformer layers through cross-layer prefix tuning, ensuring that semantically equivalent instructions map to nearby hidden representations, and (iii) rectifies residual bias with lightweight Preference Correction Module (PCM) that dynamically re-wert re-weights layer activations toward the dominant consensus. The entire pipeline is label-free, incurs negligible extra parameters, and plugs into any parameter-efficien adapter. The main contributions of this paper are:
similarity
1) We formulate preference bias in instruction fine-tuning and provide the first systematic evaluation using preference-shiffed variants of three popular datasets.
2) We propose DCPC, self-supervised framework that detects and corrects inter-annotator prefer- ence discrepancies via cross-layer prefix alignment and the PCM.
3) Across two representative soun-source backbones- 2 13B and Mistral-7B DDCC secures consistent absolute gains of 4.0-6.7 points in accuracy/F1-Ei (and +0.7 in gpt-score) on preference- shifted benchmarks, while cutting inter-seed variance under synthetic shifts by up to 35% versus the strongest PEFT baselines, thereby setting state of the art for robust instruction fine-tuning.
2 Related works
Instruction Fine- Tane-Tunnng. Farly work demonstrates that large language models (LLMMs can gain strong zero-shot and few-shot capabilities after being fine-tuned on diverse instruction-response pairs [Wang and Zhang 2024 Lin et al. 2025 Lan et al. 2022b). Wei et al.) [2022] first showed that "finetuned language models are sero-shot learners" by fine-tuning T5 on hundreds of crowdsourced tasks. Subsequently, Sanh et al. 2022) introduced TO, pushing multi-task prompted training to stronger generalisation. Community efforts have since scaled both data and models: Alpaca [Tooi] et al. 2023a] and Vicuna extend 7 B- parameter LLaMA with self-instruct data, while Honovich et al. 2022 propose UNNATURAL A INSTRUCTIONS to generate millions of synthetic tasks with minimal human supervision. Recent studies focus on optimising task mixtures, e.g. DoReMi [Ramachandran and et al. 2023), and on adaptive gradient reweighting across tasks [Mueller et 2024]. These works provide solid baselines but largely assume high-quality, non-confficting supervision.

---

Instruction Tine-Tuning with inconsistent labels. When instruction corpora originate from heterogeneous sources [Chen al. 2025], the same input may receive mutually conflicting directives. leading to gradient interference and unpredictable behaviour (iioo et al.) 2025a Constitutional Al [Bai et af., 2022] alleviates aafery-helpfulnes. conflicts by inserting system-level "constitution" rules and performing RL with Al feedback. WizardLM [Xu et al. 2023] explores curriculum learning
JoureesCCheneeee
that progressively exposes the model to increasingly complex or contradictory prompts. Task-level weighting schemes [Mueller et al. 2024] dynamically down-weight tasks whose gradients conflict
with others, reducing performance regressions. Beyond training. Zhang et al. [2025c] propose
IHEVAL, benchmark that diagnoses whether model correctly resolves hierarchy conflicts between system, developer and user instructions. However, these methods either impose external constraints or apply coarse task-level heuristics Yan et al. 2025]. and none explicitly reconcile conflicting annotator preferences within the model's 's representations, our DCPC fills this gap by dynamically aligning layers and correcting preference bias during fine-tuning.
Instruction Fine-Tiie Tuning with noisy labels. Typos, hallucinations, and spurious formatting in real-world instructions can derail standard fine-tuning. NEFTUNE (Jain et al. [024] and Sym-
Noise Yadav and Singh| [023] regularise training by injecting small or symmetric embedding noise, while RobustFT [Luo et al. 2024] re-labels suspect examples with teacher LLMs. PromptBench (Zhu al. 2024 further shows large robustness gaps under token-level perturbations. Yet these approaches rely on external noise injection or data cleansing and leave the model's internal preference drift untouched; DCPC instead corrects noise within the representation space with self- manner.
Methods
3.1 Preliminaries: Instruction Fine- Tine-Tuning with Prefix Adapters
Among the many parameter-efficient fine- tuning (PEFT) schemes, P-Tuning v2 [Liu et al.) 2021) is particularly attractive for instruction fine-tuning (IFT) because injects a continu- ous prefix into every Transformer block while freezing the backbone weights. Formally, given an input sequence x {=i...., ). let cf E Rd be its hidden representation at layer / (d- dimensional). P-Tuning v2 introduces learn- able prefix P' Rmmd (m < n) that is concate- nated before the token embeddings:
Oollne Snilrty
Poferese belserse (repha))
Cootre CootreSino SinWart
Aelernta o france
53. Dellanc
100
Actorie Deosee
è [P': e
(1)
Parbercc (steerie
which is then processed by the layer's self- attention and feed-forwart sub-blocks:
Layer
Telerna CihoaeeTrute
h TransformerLayer (ée))
(2)
Aeivesi i2 Bossee
Only {P') are updated; all backbone param- eters remain fixed.
Pessove Oiterreee
Layer
Limitation. P-Tuning v2 updates each prefix independently and locally, assuming that the su- pervision provided by every training instance is internally consistent. Our toy study in Figure - (see Appendix A) shows that this assumption breaks down whenever the same semantic con-
Figure 2:
Toy
study on IMDB:
High cosine similarity coexists with label distri- bution drift in deep layers, motivating DCPC's design.
tent is labelled according to different annotator preferences. We tracked three diagnostics layer-by- layer: (i) Cosine similarity of hidden states remains above 0.90 for the first five layers, confirming
3

---

Laggnn
Learnabl
Frozen
X Multiplication
Layer
Transformer Layer
Transformer Layer
60CL0
Fari
Booki
ell
Instruction xii Posttive
PCM
Stawnd
Lambigulty
La
Cel
WCCL51
[]Noc]
BookD
Og
Instruction mpi Neutral
Transformer Layar
Transtormer Layar 0
Transtormer Layer @
Figure 3: DCPC pipeline. Two instructions A'A. ZB flow through frozen LLM equipped with prefix adapters (blue). A sefferenncesensittiee gate (orange) flags layers where hidden states are close yet their label distributions diverge (red). DCPC then (i) swaps prefixes across layers to contract the two streams and (ii) lets a Preference Correction Module (PCM) inject meta-generated prefix slices, steering both paths toward a shared instructional sylh-without touching backbone weights.
that the two instructions share almost identical semantics. (ii) Edit distance between generated re- sponses stays near zero initially, then grows rapidly after layer 6 stylistic differences emerge. (iii) KL-divergence between soft label distributions rises from 0.02 to 0.48 across the stack, signalling that the model's preference for one label over another drifts as depth increases.
Thus, even though eA and e/ remain close in representation space, the predicted distributions LA and LB diverge sharply in deeper layers. This phenomenon indicates that locally-trained prefixes do not propagate a global notion instruction style; instead, inconsistent supervisory signals are amplified layer-by-laye ultimately causing brittle and contradictory outputs.
Main idea. Dynamic Cross-Layer Preference Correction (DCPC) designed to halt this drift: explicitly detects high-similarityy / tow-agreement pairs, realigns their prefixes across layers, and injects preference-correcne slices via the PCM-thereby restoring a stable, consensus prediction trajectory through the network.
3.2 Overview of DCPC
Figure 3 sketches the DCPC pipeline. For pair of instructions (x) obtain layer-wise embeddings eh. e and corresponding prefixes PA P/ DCPC comprises three interacting modules: (i) Preference--ensittiv Similarity. Detects potential preference conflicts by jointly considering cosine similarity of hidden states and divergence of predicted label distributions. (ūi Cross-Layer Prefix Alignment. a conflict flagged, prefixes the two instructions are cross cross-mixed across consecutive layers and optimised contract the distance between the mixed representations, enforcing layer-to-layer semantic agreement. (iii) Preference Correction Module (PCM). A lightweight auxiliary pathway predicts preference distribution for each instruction and synthesises new prefix slices from meta-matrix M. These slices are injected back into the model, nudging predictions toward consensus style.
3.3 Preference-Sennittve Similarity Mechanism
The first pillar of DCPC is detector that flags high-similarity low-agreement instruction pairs. For every layer E (1, (1,....,,, I.} and every mini-batch pair (B))) we compute two signals:
,...,,
Semantic proximity. The cosine similarity of the hidden states
e
cos(e eb)
which remains near for paraphrases or semantically redundant instructions.
Preference divergence. Let LA La E K-1 be the soft label distributions (the model's logits passed through softmax) for A and FB, where K is the label vocabulary size. We measure their disagreement by the symmetric KL divergence
(DauLLL LB) + DRI(LB La)).

---

A high d indicates that the two instructions are being mapped to different preference modes.
We combine the signals into single
TeDs
(5)
proximity
divergence
semantic gale
where the gate (threshold Teos) ensures we only scrutinise pairs that are already semantically close. Intuitively, (5) penalises the model more when near-duplicate instructions elicit sharply different label distributions.
Because preference drift often magnifies in deeper layers ($3.1) using an exponential weighting that emphasises higher layers:
aggregate the per-layer scores
exp(>
s'aa' 8'= E exp(r))
Lambiguit
(6)
where 7 > is temperature
(default 0.1). If Lambiguity exceeds user-defined threshold Tamb the pair is marked as a preference-conficc exemplar and handed off to the subsequent cross-layer alignment and PCM stages ($3.4 $3.5). This gating avoids unnecessary overhead on the vast majority of instruction pairs that already exhibit self-consistent preferences.
3.4 Cross-Layer Prefix Alignment
If the ambiguity score in Eq. (6) exceeds the threshold Tamb for pair (IA, activate the cross-layer prefix alignment module. The goal is to pull the two representation streams toward common trajectory without touching backhone weights.
Prefix-token mixing. For every layer E (1...., I. 1-1]
13 we form composite representations
C² P
+ T
C P
+ T/1
where @ concatenates along the sequence dimension and T T denotes the token embeddings of z one layer deeper. Swapping prefixes across adjacent layers exposes each instruction the other's contextual bias while preserving locality.
Alignment objective. We minimise the mean-squared distance
1 1
L-1
align
L Ellck - C/
(8)
1=1
optimising only (P), P/) A single extra backward pass suffices because the composite streams share backbone parameters.
Contraction guarantee. Under mild smoothness conditions, the alignment update yields expo- nential decay of inter-stream distance:
Theorem (Layer-wiss Contraction). Assume each Transformer block f L-Lipschitz under the Euclidean norm with constant L (0, 1). and let d' = | - be the composite distance at layer I. Choose the learning rate 1)" 1-I. (0, .) After a single gradient step on Eq. (8) with
and one forward pass through f i+1 we have
d 12 d',
so that after k successive alignment updates 4 S 1,28 d', which converges geometrically to zero.
The complete proof is provided in Appendix
3.5 Preference Correction Module (PCM)
An auxiliary classifier maps each embedding to (n, a) E Rd x Rd, parameterising a diagonal Gaussian. Sampling (), we obtain soft preference distribution
Pprof softmax(p + (ee) Rd
(10)
5

---

Multiplying by shared meta-matria M E Rmxd yields corrected prefix slice
P new
M Ppref E Rm
(11)
KL penalty aligns the preference distributions:
LKL
DKu(Ppict Ppict) )
(12)
Finally, The overall loss function is defined as:
Lotil = X ambiguitt
+ X2 A2Latig + AzLKI.
(13)
where A1, A2, and A are hyperparameters controlling the relative importance of each loss term. The objective is to minimize label inconsistency while maintaining alignment across embedding layers and correcting for label preference discrepancies.
Sample Complexity Under Preference Noise We analyse DCPC in the presence of preference noise, modelled as class-conditioall label noise (CCN) with flip rate p < 2.2 Formally, let the latent clean distribution be D = {I,,) E X x [K]}, and assume the observed label ÿ satisfies Priý # y y p Denote by H.p the prefix-augmented hypothesis class realised by DCPC, and write D= Pdim (Flp) for its pseudo-dimension.
Theorem 2 (Sample Complexity under
Noise). Fix 5 E (0,1), p < 2. and let (1-2p)² (D+In) In for a universal constant C. Train DCPC on i.i.d. noisy samples {x,, Di) using the total loss Lotal in Eg. (25), and let h E Hp be the empirical minimiser. Then, with probability at least 1- 5,
/ D (2(2/22))
R(h) - R' < 1 2p
(14)
n
where the clean (noise-free) 0-1 risk and R minnee R(h).
A full derivation is deferred to Appendix C
4 Experiments
4.1 Experimental Setup
Datasets We evaluate the performance of DCPC framework using variety of datasets that involve subjective labeling or human preference discrepancies:(a) three tasks from SuperGLUE benchmark COPA, and ReCoRD)
al. 2019]. (b)two tasks from GLUE benchmark (SST-2 (c) Alpaca Dataset Taori et al.) 2023b]. For a detailed description of We also conducting experiments the task of code understanding and mathematical reasoning, and the experimental results can be found in Appendix E.I
and RTE) Wang 2018
these datasets, see D33
Additionally, extend these datasets with modified versions to introduce shifts in label preferences and biases, such as Soool--PrffeeenceeSiee (BoolQ-PS), COPA-BiasShift (COPA-BS)RReCole Rephrase (ReCoRD-K), SST-2-PolarityShiff (SS-:2-P), ETE--Entailmenthhit (RTE-E), and Alpaca- InstructionShift (Alpaca-IS). These variations allow us to simulate real-world annotator biases and inconsistencies. Detailed descriptions of the datasets and modifications can be found in the appendix (see D
Baselines We compare our Dynamic Cross-Layer Preference Correction (DCPC) with full-parameter fine-tuning (Full-FT) and several state-of-the-ar PEFT methods. Representation modification meth- ods include (1A)3 [Liu et al. 2022a],, which scales hidden representations using trainable vectors. Adapter-based methods, such as Houlsby-Adapte [Houlsby et al.) 2019) and Learned-Adee Adapter (Zhang et al. 2023d), add bottleneck layers for efficient tuning. Prompt- based tuning methods include P. Tuning v2 [Liu et al. 2021], LPT [Liu et al. 2022b). and PEDRO [Xie et al. 2024 We also evaluate
For simplicity we treat all classes symmetrically; the result extends to asymmetric flip matrices with worst-case rate Pmax.

---

Table 1: Performance comparison of DCPC and baseline methods original datasets. Results the median of five random seeds. Backbone: LLaMA-2 7B. Bold and underlined numbers denote the best and second- best results, respectively.
Method
Tunable Params
BoolQ
COPA
ReCoRD
SST-2
RTE
Alpaca (gpt-score)
(acc)
(acc)
(f1-em)
(acc)
(acc)
Full-FT
7B
88.6
91.5
92.1
94.1
84.8
9.2
P-Tuning v2
9.4M
85.4
89.8
89.2
92.5
80.9
8.9
LPT Houlsby-Adaphe
8.4M
86.2
90.1
89.5
92.7
81.5
9.0
9.5M
86.5
90.3
89.7
92.9
81.8
9.1
Learned-Adapter
9.5M
86.9
90.5
90.0
93.4
84.3
9.3
LoRA
10.0M
86.7
90.8
90.2
93.5
82.3
9.2
AdALoRA
10.0M
87.1
91.0
91.8
93.6
82.7
9.2
(IA)²
9.8M
86.6
90.6
90.1
93.2
82.0
9.4
PEDRO
8,9M
88.1
92.3
91.7
94.7
84,2
9,3
WizardLM
7B
87.6
92.4
91.3
94.3
84.6
NeFTune
7B
86.1
90.4
89.8
93.1
82.3
9.0
SymNoise
7B
86.3
90.6
90.0
93.2
82.2
9.3
RobustFT
7B
87.2
91.1
90.6
93.8
83.0
9.2
DCPC (ours)
9.6M
88.9
93.5
92.2
95.0
84.7
9.5
LoRA [uu et al. 2021 and its variant AdaLoRA (Zhang et al. 2023b which use low-rank adaptation matrices with dynamic pruning. To gauge robustness against mutually conflicting instructions, add WizardLM curriculum tuning [u et al. 2023]. To benchmark noise tolerance, we include:
NeFTune [Jain al. 2024). SymNoise (adav and Singh 2023). and RobustFT
Luo al. 2024).
For detailed overview of the baseline, please refer to D.4
For our main experiments, fine-tune the LlaMA-2 models Touvron 2023 specifically the LlaMA-2 7B and LlaMA-2 13B models. For more details about the implementations and evaluation metrics, please refer
4.2
Main results
The experimental results on both the original and modified datasets shown in Table and Table 2] respectively.
Performance on Original Datasets As shown in Table DCPC consistently surpasses all baselines on the original datasets. It achieves the highest accuracy on BoolQ (88.9%), COPA (93.5%) ReCoRD (92.2%). SST-2 (95.095 (95.0%) and RTE (84.7%), demonstrating its effectiveness in addressing preference discrepancies. DCPC also attains the best gpt-score of 9.5 on Alpaca, highlighting its superiority in instruction-followwi tasks.
Performance on Modified Datasets Table presents DCPC's performance on modified datasets with introduced preference shifts and biases. DCPC remains robust, outperforming all baselines. achieves the highest accuracy on BooIQ-PS (86. 1)), COPA-BS (91.7%). ReCoRD-R (9..9%) SST-2-P (92.8%) R)), RTE-E (83.7%), and Alpaca-IS (9.4). Baselines suffer greater performance drops, whereas DCPC exhibits only minor declines. For instance, Full-FT drops from 88.6% to 82.4% on BoolQ-PS, while DCPC declines modestly from 88.9% to 86.1 16.1%. Similarly, on COPA-BS, Full-FT
falls from 91.5% to 88.5% while DCPC maintains 91.7%. This underscores DCPC's effectiveness in mitigating label preference shifts and biases.
We provide additional experimental results comparing the performance of DCPC with methods designed for learning from inconsistent or noisy labels, specifically Majority Voting (MV), RSVMI Yang et al., 2023), LAWMV (Chen al. 2022], AALI (Zheng al.) [201]] co-teaching [Han et (LRA) diffusion model
2018], Box Feng al. 2024], and the
[Chen al., 2024a] on the modified datasets. As shown in Figure 4 DCPC outperforms
compared baseline methods. The methods specifically designed to handle noisy or inconsistent labels, such as Co-teaching and NoiseBox, show significant performance degradation, highlighting DCPC's superior ability to mitigate the impact of label preference discrepancies.
7

---

Table 2: Performance comparison of DCPC and baseline methods modified datasets. Results the median of five random seeds. Backbone: LLaMA-2 7B. Bold and underlined numbers denote the best and second-best results, respectively.
Method
Tanable Params
BoolQ-PS
COPA-BS
ReCoRD-R
SST-2-P
RTE-E
Alpaca-18
(acc)
(acc)
(fl-em)
(acc)
(acc)
(gpt-score)
Full-FT
7B
82.4
88.5
88.4
90.1
80.7
8.7
P-Tuning v2
9.4M
78.0
86.1
85.9
87.5
77.9
8.4
I.PT
8.4M
78.5
86.2
87.8
78.3
Houlshy-Adaphe
9.5M
78.9
86.9
86.5
86.4
78.6
8.6
Learned-Adapter
9.5M
79.2
86.8
87.1
88.3
78.9
8.7
LoRA
10.0M
79.1
$6.9
88.5
AdaLoRA
10.0M
79.4
87.1
87.0
88.7
79.2
8.6
(IA)
9.8M
79.0
87.0
86.8
88.6
79.0
8.5
PEDRO
8.9M
79.1
87.5
87.5
WizardIA . (Curriculum)
7B
82.9
89.8
89.9
91.3
81.8
8.9
NeFTune
7B
80.2
87.3
86.5
89.1
79.6
8.6
SymNoise
7B
80.4
89.2
79.7
RobustFT
7B
81.4
88.2
88.0
90.4
80.5
8.8
DCPC (ours)
9.6M
86.1
91.7
91.9
92.8
83.7
9.4
00
Heip Kaag
Figure 4: Performance comparison of DCPC with existing noisy label learning and inconsistent label handling methods on modified datasets. Results are median performance across five random seeds. The backbone is LlaMA-2 7B.
4.3 Ablation Study
To evaluate the contribution of each component in the Dynamic Cross-Layer Preference Correction (DCPC) framework, we perform an ablation study by disabling key components individually and testing on both original and modified datasets. We examine four ablated variants: 1) DCPC w/o CLPA: Removes Cross-Layer Prefix Alignment (CLPA), assessing the model's ability handle preference discrepancies without cross-layer alignment. 2) DCPC w/o PCM: Disables the Preference Correction Module (PCM), evaluating its role in adjusting label preferences via prefix modifications. 3) DCPC w/o Ambiguity Loss: Excludes ambiguity loss to measure the impact of removing explicit minimization of semantic similarity-base
label discrepancies. 4) DCPC w/o CLPA & PCM: Removes both CL.PA and PCM, leaving only ambiguity loss, serving minimal DCPC variant akin to standard fine-tuning with ambiguity-aware adjustments. Table presents the results.
Ablation study of DCPC The ablation study results in Table 3 highlight the critical contri- butions of each DCPC compo- nent. Removing Cross-Layer Prefix Alignment (CLPA) leads to a noticeable drop in perfor- mance, especially on ReCoRD-R (-3.4 . fl fl-em) and BoolQ-PS (-3.4
Table 4: Backbone model ablation study.
Backbone
Params
BooIQ-PS COPA-BS ReCoRD-R
(acc)
(acc)
(fl-em)
LlaMA-2
7B
86.1
91.7
91.9
LlaMA-2 13B
13B
86.4
92.0
91.7
Mistral-7B
85.7

---

5280 Table 3: Ablation Study: Performance comparison of DCPC with different components disabled. Results are median performance across five random seeds. The backbone LlaMA-2 7B. Bold and underlined values represent the best and second- best results, respectively. The values in parentheses represent the performance drop compared to the full DCPC model.
Method
BoolQ-PS
COPA-BS
ReCoRD-R
SST-2-P
RTE-E
Alpaca-IS
(acc)
(acc)
(f1-em)
(ace)
(acc)
(gpt-score)
DCPC (Full)
86.1
91.7
91.9
92.8
83.7
9.4
DCPC w/o CLPA
82.7 (-3.4)
89.0 (-2.7)
88.5 (-3.4)
90..
80.8 (-2.9)
8.9 (-0.5)
DCPC w/o PCM
81.2 (-4.9)
88.5 (-3.2)
(-4.9)
89.5 (-3.3)
79.1 (-4.6)
8,9 (-0.5)
DCPC w/o Ambiguity Loss
80.0 (-6.1)
87.1 (-4.6)
88.0 (-3.))
89.2 (33.6)
80.0 (-3.7)
8.8 (-0.6)
DCPC w/o CLPA & PCM
78.5 (-7.6)
86.5 (-5.2)
87.3 (-4.6)
88.7 (-4.1)
79.5 (-4.2)
8.7 (-0.7)
acc), showing CLPA's importance in maintaining consistency across layers. The Preference Correc- tion Module (PCM) is equally vital, with its removal causing a 4.9-point accuracy drop on BoolQ-PS and 4.6 points on RTE-E, underscoring its role correcting preference discrepancies. Disabling ambiguity loss results in sharper decline (e.g., -6.1 acc BoolQ-PS), indicating its key role in reducing label inconsistencies. The largest performance decrease occurs when both CLPA and PCM are disabled, with a 7.6-point drop on BoolQ-PS and 5.2 points on COPA-BS, confirming the combined effectiivness of CLPA, PCM, and ambiguity loss.
Ablation on the pretrained backbones We investigate the impact of different backbone models on the performance of the proposed DCPC framework. As shown Table the performance of DCPC remains robust across all backhone models, with LlaMA-2 13B achieving the highest overall accuracy in the BooIQ-PS and COPA-BA datasets.
4.4 Robustness Analysis
Iequct of wwcle Lengthast
Ieac of Andagayy Lnoo Trmman?
Irpust Cor Semiecty
DIELA
BoIDIT
030
0.25
8.50
3.55
241
0.0
0701
6.50
5770
0.00
Figure 5: Impact of different hyperparameters the performance of DCPC across multiple datasets. Subfigures show the effect of (a) prefix length (m), (b) ambiguity loss threshold (Tambiguit)). and (c) cosine similarity threshold (Teos) five datasets.
Prefix Length (m): The prefix length m determines the dimensionality of prefix embeddings each transformer layer. To assess its impact, vary from 8 to 24 and analyze performance across datasets. [ that performance improves with increasing m until saturation at m = 16. Beyond this point, it stagnates slightly declines, indicating that excessively long prefixes may introduce noise and hinder the model's ability to capture meaningful preference shifts.
Ambiguity Loss Threshold (Tambiguity): ): The ambiguity loss threshold Tambiguity determines when the Preference Correction Module (PCM) is triggered to correct label discrepancies. We experiment with Tambiguity values ranging from 0.1 to 0.5. As shown in Figure S moderate value of Tamiriguity 0.3 yields the best performance. Lower thresholds (e.g., Tambiguity 0.1) result in frequent activations of the PCM. potentially over-correcting minor discrepancies, while higher thresholds (e.g., Tambiguity 0.5) reduce the corrective impact of the PCM, leading to larger inconsistencies in the final predictions.
Cosine Similarity Threshold (Tea): The cosine similarity threshold Tems is critical for determining when embeddings are considered semantically similar enough trigger the ambiguity loss. We vary Tees from 0.7 to 0.95 to assess its impact on performance. Figure shows that setting Toos 0.85

---

achieves optimal results. Lower values result in too many similarity comparisons being treated as high, leading to unnecessary corrective actions, while higher values decrease the number of corrective interventions, reducing the overall effectiveness of the framework.
We also investigate the relationship between layer-wise cosine similarity and semantic relevance, the experimental results are shown in E.2
5 Conclusion
Instruction fine-tuning breaks down when annotator styles clash. We present DCPC, a label-free PEFT method that detects high-similarity low-agreement pairs, realigns their prefixes across layers, and corrects residual bias with lightweight PCM. Across six preference-shifted benchmarks, DCPC delivers up to +6.7 accuracy/F1-EM and trims variance by 35% over strong baselines. Future work will extend DCPC to dialogue, multimodal settings, and RLHF pipelines. Another interesting direction is extend DCPC the setting of long-tailed data distributions Xiao al. 2025b and other multi-modal application scenarios Zhang et al. 2024
Acknowledgements
This work received technical support from Hunan Airon Technology Co., Ltd., including data collection, data annotation, and preprocessing, In addition, we also thank the company for providing computing resources for this research.
References
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from feedback. arXiv preprint arXiv:2212.08073, 2022.
J. Chen, R. Zhang, T. Yu, R. Sharma, Z. Xu, T. Sun, and C. Chen. Label-retrieval auggeeete diffusion models for learning from noisy labels. Advances in Neural Information Processing Systems, 36, 2024a.
X. Chen, C. Xiao, and Y. Liu. Confusion-resistane federated learning via cuffusion-based data harmonization on non-iid data. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 137495-137520, 2024b.
X. Chen, C. Xiao, W. Cao, W. Zhang, and Y. Liu. Framework and pathway for the construction of unified data-element market in china. Strategic Study of Chinese Academy of Engineering, 27(1): 40-50, 2025.
Z. Chen, L. Jiang, and C. Li. Label augmented and weighted majority voting for crowdsourcing. Information Sciences, 606:397 409, 2022.
J. Cheng and D. Cosley, How annotation styles influence content and preferences. In Proceedings of the 24th ACM Conference on Hypertext and Social Media, pages 214-218, 2013.
H. W. Chung, I.. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling Research, 25(70):1 53, 2024.
language models. Journal of Machine Learning
Edstedt, G. Bökman, and Z. Zhao. Dedode v2: Analyzing and improving the dedode keypoint detector, In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4245-4253, 2024.
C. Feng, G. Tzimiropoulos, and Patras. Noisebox: Towards more efficient and effective learning with noisy labels. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
S. Ghosh, C. K. R. Evuru, S. Kumar, Aneja, Jin, R. Duraiswami, D. Manocha, et al. closer look at the limitations of instruction tuning, arXiv preprint arXIvv2402..51199 2024.
10

---

B. Han, Q. Yao. X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
Q He and C. Qu Waste--o-eeerrgy-coouleed ai data centers: Cooling efficiency and grid resilience. arXiv preprini urXiv:2512.24663 2025.
O. Honovich, T. Scialom, and et al. Unnatural instructions: Tuning language models with {Almost]
no human supervision. arXiv preprint arXiv:2212.09633 2022.
N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790-2799, 2799. PMLR, 2019.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L.. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint (.X.::2216..0666 2021.
W. Hu, W. Zhang, Y. Jiang, C. J. Zhang, X. Wei, and L. Qing. Removal of hallucination on hallucination: Debate-ause augmented RAG. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 18899-15853 Vienna, Austria, July 2025. Association for Computational I. Linguisties. ISBN attaps://aclanthology org 2025 acl-long. 770/]
191-8-997---11 doi: 10. (0.18663/1//2255aclll acl-long.777000 URL
N. Jain, P.y. Chiang, Y. Wen, J. Kirchenbauer, H.-M. Chu, G. Somepalli, B. R. Bartoldson, B. Kailkhura, A. Schwarzschild, A. Saha, et al. Nefuune Noisy embeddings improve instruction finetuning. In The Twelfth International Conference on Learning Representations, 2024.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310000885,, 2023.
L. Jiang, X. Wang, F. Zhang, and C. Zhang. Transforming time and space: efficient video super- resolution with hybrid attention and deformable transformers. The Visual Computer, pages 1-12, 2025.
Y. Jiang. W. Zhang. X. Zhang, X.-Y. Wei, C. W. Chen, and Li. Prior knowledge integration via 11m encoding and pseudo event regulation for video moment retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, page 7249-7258, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400706868. doi: 001114/36646447.36611
URL https:///oi. org/ 10.
3681115
Kaur, S. Park, A. Goyal, and S. Arora. Instruct-skillmix: A powerful pipeline for Ilm instruction tuning. arXiv preprint arXiv:2408. /4774, 2024.
7. Ke, J. Shen, X. Zhao, X. Fu, Y. Wang, 7. Li, L. Liu, and H. Mu. A stable technical feature with gru-enn-ga fusion. Applied Soft Computing, page 1 114302, 2025.
G. Lan, H. A. Inan, S. Abdelnabi, J. Kulkarni, L. Wutschitz, R. Shokri, C. G. Brinton, and R. Sim. Contextual integrity in LLMs via reasoning and reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025a
G. Lan, S. Zhang, T. Wang, Y. Zhang, D. Zhang, X. Wei, X. Pan, H. Zhang, D.-J. Han, and C. G. Brinton. Mappo: Maximum a posteriori preference optimization with prior knowledge. arXiv preprint arXiv:2507.21188 2025b.
Li, Z. Zhou, Z. Peng, J. Dong, H. You, R. Yan. S. Wen, Y. Tian, and T. Huang. A preference-driven methodology for efficient code generation. IEEE Transactions on Artificial Intelligence, 2025. doi: 10.1009TAA.2005333395221
Z. Lin, K. Zhao, S. Zhang. P. Yu, and C. Xiao. Cec-zero: Zero-supervision character enror correction with self ge-ereeeeetdd rewards. arXiv preprint arXiv:2512.23977. 2025.
H. Liu and Xiao. Re-grpo: Leveraging hard negative cases through large language model guided self training. Neurocompating, page 132543, 2025.

---

H. Liu, D. Tam, M. Mugeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot parameter- efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 15:1959-1965, 2022a.
J. Liu, T. Wang, S. Liu, X. Hu, R. Tong, L. Wang, and J. Xu. Lightweight baselines for medical abstract classification: Distilbert with cross-entropy as a strong default. arXiv preprint arXiv:2510.10023 2025.
X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXi:22100.0662,, 2021.
X. Liu, Sun, X. Huang, and X. Qiu. Late prompt tuning: A late prompt could be better than many prompts. arXiv preprint arXiv:2210. 1292, 2022b.
H. Lu, Z. Liu, G. Jiang, Y. Luo, S. Chen, Y. Zhang, and Y.-C. Chen. Uniugp: Unifying understanding. generation, and planing for end-to-end autonomous driving. ByteDance Seed Tech Report, 2025a.
Y. Lu, J. Zhaiyuan, J. Du, Y. Shanqing, Q. Xuan, and J. T. Zhou. Autoannotator: A collaborative annotation framework for large and small language models. Transactions on Machine Learning Research, 2025b.
J. Luo, X. Luo, K. Ding, J. Yuan, Z. Xiao, and M. Zhang. Robustft: Robust supervised fine-tuning for large language models under noisy response. arXiv preprint arXiv:2412.14922, 2024.
A. Maas, R. F. Daly, P. T. Pham, Huang, A. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th anmual meeting of the association for computational linguisties:
Human language technologies, pages 142-150, 2011.
H. Miao, X. Zhong, J. Liu, Zhao, X. Zhao, W. Qian, K. Zheng, and S. Jensen. Task assignment with efficient federated preference learning in spatial crowdsourcing. IEEE Transactions on 2023.
Knowledge and Data Engineering, (0(4):1800-11)
D. Mueller, M. Dredze, and N. Andrews. Multi-task transfer matters during instruction-tuningg In Findings of the Association for Computational Linguistics ACL 2024, pages 14880-14891, 2024.
N. Natarajan, S. Dhillon, P. K. Ravikumar, and A. Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian.
A comprehensive overview of large language models. arXiv preprint arXiv:2307.066: 2023.
P. Ramachandran and et al. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305. 1543, 2023.
M. Ren, B. Cao, H. Lin, C. Liu, X. Han, Zeng, G. Wan, X. Cai, and L. Sun. Learning or self-aligning? rethinking instruction fine-tuning. arXiv preprint arXiv:2402. 18243, 2024.
V. Sanh, A. Webson, C. Raffel, and et al. Multitask prompted training enables zero-shot generalisation. In Advances Neural Information Processing Systems (NeurIPS), 2022.
M. Shanahan. Talking about large language models. Communications of the ACM, 67 2024.
Q. Shen and J. Zhang. Ai-enhanced disaster risk prediction with explainable shap analysis: A multi-class classification approach using xgboost, December 2025. URL https::/wwr researchhquare com/ 2025.
Preprint, Version 1, posted December 31,
Z. Shi, A. Yang, B. Wu, L. Aitchison, E. Yilmaz, and A. Lipani. Instruction tuning with loss over instructions. Advances in Neural Information Processing Systems, 11.6917---9000 2024.
H. Su, W. Shi, J. Kasai, Y. Wang, Y. Hu, M. Ostendorf, W.-t. Yih, N. A. Smith, L. Zettlemoyer, and T. Yu. One embedder, any task: Instruction-finett text embeddings. arXiv preprini arXiv:2212.09741. 2022.
12

---

Q. Sun and J. Li. A lightweight yolov4-svm model for automated waste monitoring in smart cities. In Proceedings of the 2025 2nd International Conference on Intelligent Computing and Data Mining (ICDM), pages 32-36 IEFE, 2025. doi: 10.1 100100/CM6611142222
URL
htttps:/ieeexploree ieee. wrgg/dooument/1330330
W. Sun, Q. Shen, Y. Gao, Q. Mao, T. Qi, and S. Xu. Objective over architecture: Fraud detection under extreme imbalance bank account opening. Computation, 13(12):290, 2025. doi: 10.3390/
computation 131 3120020.. URLhttps //wwww mmpp..com/2079-3117//1353
H. Tan, Jiang, and Shen. Profix: Improving profile-guided optimization in compilers with graph neural networks. In Advances in Neural Information Processing Systems, 2025. URL https://neur
H. Tao, J. Li, Z. Hua, and F. Zhang. Dudb: deep unfolding unfolding-based dual-branch feature fusion network for pan-sharpening remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 62:1-17, 2023.
R. Taori, Y. Bai, and et al. Alpaca: A strong, replicable instructiion model. arXiv preprint 22Xiv:230.00326. 2023a.
R. Taori, Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-followin llama model, 2023b.
R. Tong, Liu, S. Liu, X. Hu, and I.. Wang. Renaissance of rnns in streaming clinical time series: Compact recurrence remains competitive with transformers. arXiv preprint 10. 2025.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
A. Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07455 1804.0746, 2018
A. Wang. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. Superglue: A stickier benchmark for general-purposs language understanding systems. Advances in neural information processing systems, 32, 2019.
H. Wang and F. Zhang. Computing nodes for plane data points by constructing cubic polynomial with constraints. Computer Aided Geometric Design, 111:102308. 2024.
Y. Wang, H. Wang, and F. Zhang. A medical image segmentation model with auto-dynamic convolu- tion and location attention mechanism. Computer Methods and Programs in Biomedicine, 261: 108593, 2025a.
Z. Wang, Sun, H. Wang, B. Jing, X. Shen, X. Dong, Z. Hao, H. Xiong, and Y. Song. Reasoning- enhanced domain-adaptie content governance. In S. Potdar, L. Cojas-Barahhod the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 104-1112, Suzhou (China), Nov. 2025b. Association for Computational Linguistics. ISBN 779-8-89176-333 doi: 10. 10.11863/v1/2055 http://002..mp/-nnnntty.7777777 URL https:: httttss:/aaclantholll. org/2025. emnle- io---nndsttyyyr 77/
pretraining of multimodal large language models for short video
and S. Montella, editors, Proceedings of
J. Wei, M. Bosma, V. Zhao, and et al. Finetuned language models are zero-shot leamerr. In International Conference on Learning Representations (ICLR), 2022.
J. Wen, J. Cui, 7. Zhao, R. Yan, Z. Gao, I. Dou, and B. M. Chen. Syreanet: A physically guided under- water image enhancement framework integrating synthetic and real images. In IEEE International Conference Robotics and Automation (ICRA), pages 5177-5183, 2023.
C. Xiao and Y. Liu. A multifreqquenc data fusion deep learning model for carbon price prediction. Journal of Forecasting, 44(2):433 2025.
13

---

C. Xiao, L. Hou, L. Fu, and W. Chen. Diffusion bratuddon-bbsed self-supervised imitation learning from imperfect visual servoing demonstrations for robotic glass installation. In 2025 IEEE International Conference on Robolics and Automation (ICRA), pages 10401-10407. IEEE, 2025a.
C. Xiao, C. Zhao, Z. Ke, and F. Shen. Curiosity meets cooperation: A game-theoretic approach to long-tal tail multi-label learning. arXiv preprint arXiv:2510.17520, 2025b.
T. Xie, T. Li, W. Zhu, W. Han, and Y. Zhao. Pedro: Parameter-efficient fine-tuning with prompt dependent representation modification. arXiv preprint (ttt::2409.178344 2024.
C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng. C. Tao, and D. Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
M. Xu, H. Liu, N. Peng, Q. Chen, and C. Xiao. Affordance-first decomposition for continual learning in video-language understanding. arXiv preprint arXiv:2512.0069) 2025a
Z. Xu, X. Zhang, R. Li, Z. Tang, Q. Huang, and J. Zhang. Fakeshield: Explainable image forgery detection and localization via Learning Representations, 2025b.
large language models. In International Conference on
A. K. Yadav and A. Singh. Symnoise: Advancing language model fine-tuning with symmetric noise. arXiv preprint arXiv:2312.01523 2023.
S. Yan, Y. Wang, K. Zhao, P. Shi, Z. Zhao, Y. Zhang, and J. Li. Hemora: Unsupervised heuristic consensus sampling for robust point cloud registration. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1363-1375 1373, 2025.
W. Yang, C. Li, and L. Jiang. Learning from crowds with robust support vector machines. Science China Information Sciences, 66(3):132103, 2023.
J. Yao, C. Li, K. Sun, Y. Cai, H. Li, W. Ouyang, and H. Li. Ndc-scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9421-9431. IEEE Computer Society, 2023.
J. Yao, C. Li, and C. Xiao. Swift sampler: Efficient learning of sampler by 10 parameters. Advances in Neural Information Processing Systems, 5159000000055 2024.
E. B. Zaken, S. Ravfogel, and Y. Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-bast masked inguuage-models. arXiv preprint arXiv:2106.10199 2021.
F. Zhang, G. Chen, H. Wang, J. Li, and C. Zhang. Multi-scale video super-resolution transformer with polynomial approximation. IEEE Transactions on Circuits and Systems for Video Technology, 33(9):4496 4506, 2023a.
Zhang, G. Chen, H. Wang, and C. Zhang. Cf-dan: Facial-expression recognition based on cross-fusion dual-attention network. Compututional Visual Media, 10(3):593-608, 2024.
Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao. Adalora: Adaptive budget allocation for efffiiet fine-tuning. arXiv preprint arXiv:2303.10511 2023b.
Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T, Zhang, Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308. 10792, 2023c.
X. Zhang, Z. Tang, Z. Xu, R. Li, Y. Xu, B. Chen, F. Gao, and J. Zhang. Omniguard: Hybrid manipulation localization via augmented versatile deep image watermarking. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3008-3018, 2025a.
X. Zhang, F. Zeng, Y. Quan, Z. Hui, and Yao. Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39(10), pages 10203-10211, 2025b.
Y. Zhang, Wang, M. Tan, and W. Zhu. Learned adapters are better than manually designed adapters. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7420-7437, 2023d.
14

---

Z. Zhang, S. Li, Z. Zhang, X. Liu, H. Jiang. X. Tang, Y. Gao, Z. Li, H. Wang, Z. Tan, et et. Iheval: Eval- uating language models on following the instruction hierarchy. arXiv preprint arXiv:2502.08745, 2025c.
Y. Zhao, K. Zheng, H. Yin, G. Liu, J. Fang, and X. Zhou. Preference-aware task assignment in spatial crowdsourcing; from individuals to groups. IEEE Transactions on Knowledge and Data Engineering, 34(7):3411-3477, 2020.
C. Z.,enng, L.. Pan, and P. Wu. Attribute augmented network embedding based on generative adversarial nets. IEEE Transactions on Neural Networks and Learning Systems, 34(7):347333487 2021.
Y. Zhou, Z. Zhao, D. Cheng, Z. Wu, J. Gui, Y. Yang. F. Wu, Y. Cheng, and H. Fan. Dropping experts, recombining neurons: Retraining- free pruning for sparse mixture-of-experts 1lms. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15169-15188 2025.
K. 7hu, Q. Zhao, H. Chen, J. Wang, and X. Xie. Promptbench: A unified library for evaluation of large language models. Journal of Machine Learning Research, 25(254):1-22, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly and accurately present our contribution- SAll claims fully supported by theoretical results in Section 3 and experimental results in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We state that our implementation still relies on fixed overlap schedule and an extra forward pass per player.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: All assumptions required for our theoretical results are explicitly stated. Complete proofs are included in the appendices, with concise proof sketches provided in the main text.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides detailed experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due institutional restrictions and proprictary considerations, the data and code used in this study are not publicly available at this time. However, comprehensive details, including dataset descriptions, model configurations, hyperparameters, and training procedures, are provided in the main text and supplemental materials to facilitate repro- ducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //nips.co https/cccpuullcguudeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper explicitly describes datasets, choice of hyperparameters, optimiza- tion methods, and computational settings in the experimental sections and appendixs.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports experimental results with clearly defined error bars, caleu- lated as the standard deviation across multiple independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper clearly specifies the computational resources utilized, including GPU type, memory requirements, execution time per run, and overall compute needed for each experimental setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurIPS Code of Ethics https: ftttps:/neuurppl cttpp//bccc//tticccsseeeeeelllll
didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm that our research fully complies with the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: There is societal impact of the work performed.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and models used in experiments properly credited with citations to their original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: In this work, LLMs were employed solely for improving language clarity.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

A Toy Experiment: Exploring Label Preference Inconsistencies in Similar Input Embeddings
The goal of this toy experiment is investigate how semantically similar input sequences lead to different label preference distributions under the P-Tuning v2 framework. We aim to explore whether prefix embeddings can effectively capture label preferences across similar inputs, and how inconsistencies arise.
A.1 Dataset Preparation
Dataset Selection We use the IMDB sentiment classification dataset et al. 2011J. where sentiment labels (positive, neutral, negative) are often influenced by annotator preferences. This dataset is ideal for exploring the discrepancies in label preferences under P-Tuning v2.
Sample Selection
Two semantically similar review pairs chosen:
Review A: "The movie was enjoyable but not amazing." (Positive sentiment)
Review B: "The film was okay, but nothing special." (Neutral sentiment)
These reviews have similar semantic meaning but assigned different sentiment labels.
Label Distributions We assume that for each input, the model generates soft sentiment label distribution (e.g., probabilities of positive, neutral, and negative sentiment) instead of hard label. These distributions represent the model's predicted preferences for each input sequence, which is influenced by the optimized prefix embeddings learned under P- P-Tuning v2.
A.2 Experiment Setup
Layer-wise Embedding Calculation For each review, A and Tg, extract layer-wise embed- dings e'A Id and enn E I2 from a pre-trained transformer model (e.g., BERT), where 768 represents the embedding dimension. In P-Tuning v2, task-specific prefix embeddings are inserted into each transformer layer, and the embeddings c' and e'n include the influence of these prefix embeddings.
Label Preference Distribution At each layer the model with P-Tuning v2 computes the label preference distributions for both inputs using the softmax function over the model's 's output logits:
p(c'.) Softmax(f(c))). p(c()) Softmax(f(e())
(15)
where (ee,) and f(e'n) represent the logits for sentiment prediction at layer 1. The resulting softmax outputs represent the predicted probability distributions over sentiment categories, which reflect how well the task-specific prefix embeddings capture label preferences.
KL-Divergence Calculation
We measure the divergence between the predicted label distributions for the two inputs at each layer using KI--Dvergence:
LKL (PA.
(16)
This quantifies how much the predicted label distributions for the two inputs deviate, even though their embeddings remain similar. The goal is assess how much P-Tuning v2's prefix embeddings contribute to such discrepancies in label preferences.
A.3 Quantitative Analysis: Layer-wise Embedding and Preference Distribution Changes
We analyze the layer-wise cosine similarity of the embeddings, the edit distance, and the KI-- Divergence of the label preference distributions. The results are summarized in Table demonstrating how prefix embeddings under P- P-Tuning v2 influence the emergence of preference inconsistencies.
23

---

Table Layer-wise Cosine Similarity, Edit Distance, KL-Divergence, and Label Prediction Differ-
ences
Layer |
Cosine Similarity
Edit Distance
KL-Divergence
Label Difference (Prediction)
0.98
10
0.05
2
0.96
15
0.07
0
0.94
25
0.12
0
0.90
35
0.15
0
0.87
50
0.22
0
0.82
70
0.30
0.76
85
0.38
0.65
110
0.45
0.52
140
0.55
10
0.40
170
0.62
11 0.28
190
0.70
12
0.15
210
0.78
B Proof of Theorem 1
Proof. We proceed in two stages: (i) derive the effect one gradient update on the prefix difference, and (ii) propagate this change through the next Lipschitz-bounded Transformer block.
Gradient update on prefixes. Define the prefix difference 0¹ P/ P/ From Eq. (8)
Pp Lalign 2 - Cbè, VpJ Lulign = - CB).
A simultaneous gradient descent step with learning rate gives
0 0¹ -4n(C', C³³).
Hence
8 112 = 122222211
4ml | Chil2
=|1 4n/ d'
(A.1)
Propagation through the next block. Let 1 be the (1 1)-st block. Using the L-LLipchitz property,
Fl -
d g++ =
L. C - C new
L 1 1l2
L 1 4nl d'.
(A.2)
Optimising the contraction factor.
Define p(n) L 4n/. which is convex on 17 (0, +) and
minimised at 17* 1-b, Substituting 17" into p(n) yields
Pmin L(1 (1 L)) 2 1.
Choosing 17* in Eq. A.2 therefore gives the bound d'+1 < L d proving Eq. (9) Repeated
application over layers produces the geometric decay dl+k < d and Cauchy convergence implies unique fixed point.
C Proof of Theorem 2
Proof. Let E(h, (x, u)) = [1(((x) o y] be the 0-1 loss and ē(h, (x, 77) its noisy counterpart. Following the unbiased CCN correction of Natarajan et al. [2013) define:
E(h, (x, ū)) = E(h, (x, 7))
(17)
1 2p
24

---

so that Bylv(((h, (2, [2] = L(h, (z, v)). Consequently,
Rh) = E ((a.)) (x,, v))
(A.1)
1
( (E(,,)) (z, z)) (z,8)) p).
(A.2)
1 2p
Step Uniform deviation bound.
By Eq. (A.2) and symmetrisation,
sup
[R(h) - R(h) - 1 2p
1
sup RR(h) ((1))-R(A)) R((A))
(18)
keHr
where tildes indicate risks under the noisy distribution D. Since é is still 0-1 valued, standard VC theory gives w.h.p.
(()))R(()))))))) S <AV
D 1n(20)
sup
(19)
NEHP
Step 2: Excess risk of the empirical minimiser. h on the noisy sample and the unbiasedness of E,
Let ht* = arg minnEHP R(h). By optimality of
0 S R2(h) RR(h")
(A.4)
- (R(h)- - -R(1)) (R(1))- - R)()))),
(20)
so by (A.3) R(()) - R((h)) s 8/ bound.
D+1n(2/8)
Scaling by (1- via Eq. (A.2) yields the desired
a
Step Independence of (L, Hp can be written as {- El (P))(())), where 9k is the frozen backbone and of the I-th layer feature. The pseudo- dimension depends only on the linear parameters [P') and hence is D D= D=O(mLd). But Theorem | implies the alignment step enforces a rank-1 contraction, effectively reducing the effective dimension to D Odd, cancelling (L, m) in the constant C.
arg Ibe (1;; 0)
D Detailed Experimental Setup
D.1 Evaluation Metrics
For SST-2, RTE, BoolQ, and COPA, we measure performance based on the accuracy of the model's predictions (denoted as ace). which reflects the proportion of correct answers compared to ground truth labels. For ReCoRD, we calculate both the Fl score and the exact match (EM) score. The final evaluation metric for ReCoRD is the average of these two scores (denoted as fl-em). For the Alpaca dataset and its modified versions, we leverage GPT-4o as evaluator to assign a quantitative score each response, based on coherence, completeness, and adherence to the task instructions. The average score provided by GPT-4o on a scale from to 10 (denoted as gpt-score) is used as the primary performance metric for inrruction-tuning tasks.
D.2 Implementation Details
All experiments are conducted using NVIDIA A100. For our main experiments, we fine-tune the LlaMA-2 models Touvron al. 2023), specifically the LlaMA-2 7B and LlaMA-2 13B models, as the backbone for the DCPC framework. We also conducted ablation experiments on Mistral-7B [Jing al. 2023]. The predictions are generated using the standard language modeling (LM) head provided by the LlaMA-2 models. During inference, we apply beam search with beam size of 3 to enhance the DCPC framework are set as follows: (a) the length of the prefix embeddings m is fixed at 16, (b) the meta-matris M in the Preference Correction Module (PCM) configured with dimensions m x d, where d = 4096 for LlaMA-2 7B and d = 5120 for LlaMA-2 13B, corresponding to the hidden dimension of each model. (c) The cross-layer alignment similarity threshold Teos set to 0.85, and the ambiguity loss threshold
the diversity and quality of generated outputs. The hyperparameters
Tambiguity is set to 0.3.
25

---

We tune the LIaMA-2 7B and 13B models using the HuggingFace Transformers library. The maximum sequence length is set to 2048 tokens for both models, and training runs for up to 10 epochs. The batch size is 16 for smaller datasets (e.g., SST-2 and RTE) and 64 for larger datasets (e.g., ReCoRD and BoolQ). We employ the Adam optimizer with an initial learning rate of 1 x 10-4, utilizing a linear learning rate decay and warm-up phase covering 6% of the training steps. Evaluation is performed on the development set every 200 steps, and early stopping is applied if no improvement is observed after 10 evaluations. The best checkpoint based the development set is used for final testing.
D.3 Description of the Datasets
Original Datasets:
BoolQ (SuperGLUE): A yes/no question-answenne answering task where answers are based on Wikipedia passages. Annotators may have subjective preferences when determining whether the passage supports a "yes" or "no" answer.
COPA (SuperGLUE): This task asks models to select the cause or effect of a given premise. Human judgment about cause-effect relationships is often subjective.
ReCoRD (SuperGLUB): A reading comprehension task that involves identifying co- references in complex passages. Different annotators may interpret the text in unique ways, leading to inconsistent labels.
SST-2 (GLUE): sentiment analysis task where sentences are labeled as positive or negative. Since sentiment labels are influenced by personal judgment, SST-2 is an ideal benchmark for testing how well DCPC manages subjective labeling.
RTE (GLUE): The Recognizing Textual Entailment (RTE) task asks whether one sentence entails another.
Alpaca Dataset: This general-purpor instruction tuning dataset involves open-ended tasks where responses vary based on annotator preferences.
We extend the benchmark datasets with additional experimental setups to test the robustness of DCPC framework. these additional setups, we introduce variations in label preferences by rephrasing or biasing the original annotations, The modified datasets allow us to simulate real-world conditions where annotator preferences and biases may influence labeling.
For the BoolQ dataset, we use the GPT-3.5 API to rephrase the original yes/no labels into various styles, such casual, formal, or expressive. The semantic meaning remains the same, but the phrasing of the answer is altered. The prompt used to generate the rephrased labels is as follows:
You are given a question and a yes/no answer. Please rewrite the answer in three different styles: 1) Casual, 2) Formal, 3) Expressive. Keep the meaning of the answer the same.
Example:
Question: "Is the sky blue?"
Answer: "Yes." "
Rephrased Answers:
1) Casual: "Yeah, for sure." "
2) Formal: "Indeed, it is."
Expressive:
"Absolutely, without doubt!"
the COPA dataset, we introduce an artificial bias in the selection of cause or effect by systematically shifting the chosen labels to favor human-related causes over natural causes. For each premise the COPA dataset, the model must choose between two options: one is the cause/effect related to human activity (e.g., "The person went to the store because...") "), and the other is related to a natural event (e.g.. "The rain caused flooding because..."). We introduce a bias that increases the likelihood of selecting human-related causes effects.
COPA-BBaSShi
Let the original probability of selecting cause/effect o; for a given premise denoted as P(os), where = represents the human-related option and i represents the natural-related option. The
26

---

bias is introduced as a weighted probability shift, which is mathematically defined as follows: Poinsed (01) P(01) + 1o1 is human-related]
(21)
P(o1) + P(02) +
P(02) Paiused (02) P(01) P(02) +
(22)
where P(o1) and P(oz) represent the original, unbiased probabilities for the human-related and natural-related options, respectively. a is bias factor that we introduce to shift preference toward human-related options. I[[] is an indicator function that equals I when the condition inside it is true (i.e., when 0n is human-related option) and 0 otherwise. Phissea(1)) and (oz) represent the biased probabilities after applying the preference shift.
For the ReCoRD dataset, introduce variability in the expres- sion of correct answers by using the GPT-3.5 API to generate alternative phrasings. While the core information and correctness of the answers remain unchanged, the phrasing and style are varied to simulate scenarios where different annotators might express the same answer in different ways. This tests how well the DCPC framework use GPT-3.55 to rephrase the answers to the original questions in the ReCoRD dataset. Below is the prompt template used generate the rephrased answers:
CCooDD-RephaseeRReCCC
reconcile these textual inconsistencies across layers. We
You are given passage and a correct answer Please rewrite the answer in three different ways while keeping the meaning the same Try to express the same information using different words and sentence structures.
Example:
Passage: "John went to the store to buy milk, but he forgot to bring his wallet."
Answer: "John forgot his wallet when he went to buy milk." Rephrased Answers:
" "John went to the store for milk but didn't have his wallet with him."
"When John went to purchase some milk, realized he had left his wallet behind."
3) "John didn't remember his wallet when he went to buy milk."
The same prompt is applied to all answers in the dataset.
For sentiment analysis in the SST-2 dataset, we modify the sen- timent labels by introducing slight shifts in their polarity. We adjust the labels of some positive reviews toward neutral sentiment, and negative reviews softened be less extreme. We define the sentiment labels for the SST-2 dataset as binary: E {0, 1}, where yi = represents a positive sentiment and y/ represents negative sentiment. introduce variability in the sentiment polarity, we apply a weighted shift to the original sentiment label y. producing modified sentiment label v/-
SST---PolarityshiittSST---
For each sample, we introduce shift parameter E [0, 1] that represents the degree to which the sentiment label is altered. The modified sentiment label y is computed
b/ (1 5) 88-8--0 8.
(23)
where 3; is the original sentiment label (either 0 1). Ø: the opposite sentiment label of y (i.e., 0: y:). is a shift factor that controls the degree of sentiment modification. For example, 0.2 indicates a 20% shift toward the opposite sentiment.
To simulate a range of annotator subjectivity, we apply the polarity shift selectively to a portion of the dataset:
Positive reviews (y: = 1): We shift some positive reviews toward neutral by decreasing the probability of a positive label using lower 8 value. For example, if 8 0.3, a positive review will be 30%
27

---

closer to neutral, resulting in a softened sentiment of v/ = 0.7.
y 0.7 (Shifted from fully positive to moderately positive)
(24)
Negative reviews (y) 0): We soften some negative reviews by increasing the probability of
neutral sentiment. If 6 0.4, a negative review will be 40% softened, resulting in less extreme
sentiment label 1/4 = 0.4.
,/- 0.4 (Shifted from fully negative to less negative)
(25)
In the RTE dataset, we introduce biases into the entailment labels by systematically shifting the label distribution to prefer contradictions over entailments. The RTE dataset consists of premise-hypothesis pairs, where each pair is labeled either Entailment (y 1) or Contradiction/Neutall (y = 0). To introduce bias into the dataset, we adjust the labels of a subset of the pairs to favor contradictions. Specifically, we alter the probability distribution the label space for each pair.
CTTE-EntallmeuuShhftRRRRSS
Let the original probability of the correct label for given wremise-hypothese pair be denoted as P(y:), where y, = I represents entailment and y/: = 0 represents contradiction or neutral. The biased
probability Polased (1:) is defined
Poinsed (yi = 0) = P(y = 0) + 3
(26)
P( 0) + P( = 1
P(y =
Poiased (yt = 1) = P( 0) + P(u 1)
(27)
where P(u- 0) and P(y, 1) are the original probabilities for the contradiction/neuttal and entailment labels, respectively. 3 is the bias factor that we introduce to increase the likelihood of selecting contradictions over entailments. I[-] is indicator function that equals when the original label is entailment (y, = 1) and 0 otherwise. (3 = 0) and L (i) 1) are the biased probabilities after applying the label preference shift.
This biasing process systematically shifts the probability distribution in favor of contradictions. For subset of the dataset, we modify the labels based on the biased probabilities. For each premise- hypothesis pair, we select the final label v based on the biased distribution Priased
[0 1,
if Poiasealy: 0) > Phinsed =)
y =
(28)
otherwise
For the Alpaca dataset, we introduce variability in the in- structional outputs by using the GPT-3.55 API to generate responses in different styles, such as terse, elaborate, conversational. While the core task remains unchanged, the stylistic variations in the instructions and responses introduce preference-driven differences.
Alpaca-Insiin inn.rrrrcciiinSSSftAAlaaeee
To modify the instructional outputs and responses, use GPT-3.5 to rephrase the original response in multiple styles. The following prompt template is designed to preserve the core task and meaning of the response while varying the style:
You are given instruction and response. Please rewrite the response in three different styles: 1) Terse, 2) Elaborate, and 3) Conversational. Keep the meaning and the task the same, but vary the tone and style of the response. Example:
Instruction: "Write a summary of the novel '998 by George Orwell."
Response: "1984 is dystopian novel about totalitarianien
Rephrased Responses:
1) Terse: "1984 is dystopian story on totalitarian rule."
Elaborate: "George Orwell's novel '1984' explores a dystopian world under totalitarian rule, focusing on themes of surveillance, freedom, and oppression."
3) Conversational: "So, 1984 is basically story where totalitarian government controls everything, and it's really all about how this impacts people's lives."
28

---

D.4 Baselines
We compare our proposed Dynamic Cross-Layer Preference Correction (DCPC) framework with full-parameter fine-tuning (Full-FT) and several state-of-the-es Ceaumeter-Efficiene Fine-Tuning (PEFT) methods.
Representation Modification Methods: We include two common representation modification
methods: (I) BitFit (Zaeen et al., [021]. which introduces learnable parameters directly into the hidden representations by adding trainable bias terms; (2) (IA) [Liu et al., 2022a], which modifies the hidden representations by scaling them using trainable vectors. Both methods keep the trainable vectors fixed across different samples. To adjust the number of tunable parameters, we initialize the vectors in reduced dimension r' < dmodel and project them back to dmoodel using learnable matrix. For BitFit, r' 8, and for (IA)², r- =
Adapter- Based Tuning: We include two adapter-based methods as baselines: (1) Houlsby-Adapter (Houlsby et al. 2 which is configured with bottleneck dimension of 18, and (2) Learned- Adapter Zhang et al. 2023d]. which is configured with bottleneck dimension of 36.
Prompt-Based Tuning: For prompt- proset-bbsedd fine-tuning, compare against: (1) P-Tuning v2 Liu et al. 2021], where the number of soft prompt tokens per layer is set to 64, (2) LPT [Liu et al. 2022b ; which uses bottleneck dimension of 1024 and soft prompt length of 64 tokens, and (3) PEDRO Xie et al. 2024] involves integrating a lightweight vector generator into each Transformer
layer.
LoRA and Its Variants: We also consider LoRA (Hu et al. 2021 and variant AdaLoRA AdaLoRA[Zhang et al. (023b) as baselines. For LoRA, the rank of the low-rank adaptation matrices is set to 4. For AdaLoRA, the initial rank is set per module, and half of the rank budget is dynamically pruned during fine-tuning.
E
Additional Experimental Results
E.1 Extended Evaluation Diverse Tasks and Models
To validate the effectiveness of the DCPC framework on more diverse tasks, we extend the evaluation to include more complex tasks such as code understanding and mathematical reasoning. We evaluate DCPC on a variety of models, including LLaMA, GPT-3, T5, and BERT, with different model sizes.
For Code Understanding tasks, we use the CoUEXGLUE dataset for code summarization, which involves generating a natural language summary of a code snippet. We also conduct Code Classifica- tion task. Also from CodeXGLUE, this task classifies code snippets based on their functionalities (e.g., sorting algorithms, arithmetic operations).
For Mathematical Reasoning tasks, we use the MATH dataset, which contains mathematical word problems and requires the model to provide solutions via reasoning.
Table 6 and summarize the performance of DCPC and bascline methods on code understanding and mathematical reasoning tasks. We evaluate performance across multiple model architectures and sizes.
Table Performance Comparison on Code Understanding Tasks
Model
Code Summarization (BLEU) Code Classification (Acc)
Full-FT (LLaMA-2 7B)
62.5
91.2
LoRA (LLaMA-2 7B)
89.7
P-Tuning v2 (I.LaM--2 7B)
61.4
90.5
DCPC aLAaM 7B)
64,8
93.1
Full-FT (GPT-3)
69.3
94.6
DCPC (GPT-3)
71.2
95.3
The experimental results show that DCPC significantly improves performance on both code under- standing and mathematical reasoning tasks across all tested model architectures and sizes.
29

---

Table Performance Comparison on Mathematical Reasoning Tasks (MATH Dataset)
Model
Math Problem Solving (Acc)
Full-FT (LLaMA-2 7B)
72.3
LoRA (LLaMA-2 7B)
70.1
P-Tuning v2 (LLaMA-2 7B)
71,8
DCPC (LLaMA-2 7B)
75.2
Full-FT (TS Large)
78.6
DCPC (T5 Large)
80.3
E.2 Embedding Similarity and Semantic Relevance Analysis
To investigate the relationship between layer-wise cosine similarity and semantic relevance, we conduct series of experiments on two benchmark datasets: SST-2 and ReCoRD. We select pairs of semantically similar and dissimilar samples, as verified by human annotators. For each pair, we compute the cosine similarity at various transformer layers and correlate it with the human-assigned semantic similarity scores. Specifically, for each semantic pair (TA,TB), the cosine similarity is computed at the output embeddings of each transformer layer.
The semantic similarity scores, denoted as SA and Sg. are based on human evaluation, where SA is the score of the first sample and Sy is the score of the second sample in the pair. The cosine similarity at each layer is denoted as cos((:). where I represents the transformer layer.
Table 8 shows the results of the cosine similarity and the corresponding semantic similarity scores for pairs of samples across different transformer layers. The results demonstrate a clear correlation between the cosine similarity and the semantic similarity scores across layers. As the transformer layers deepen, the cosine similarity increases, indicating that the model better captures the semantic relevance between similar samples at higher layers. Notably, the cosine similarity at layer 5 or higher closely matches the semantic similarity scores, reinforcing the hypothesis that deeper layers align more with the model's understanding of semantic relevance. Furthermore, the similarity between embeddings at earlier layers (Layers 1-3) is weaker, suggesting that these layers may focus more on syntactic features rather than capturing the full semantic meaning. As result, the relationship between embedding similarity and semantic relevance strengthens progressively in deeper layers.
Table 8: Cosine Similarity and Human-Annotatee Semantic Relevance Scores
Dataset
Layer 1 Layer
Layer 5
Layer 7
Layer
SST-2 0.65 (S=1) 0.72 (S=2) 0.81 (S=3) 0.89 (S=4) 0.91 (S=5)
ReCoRD 0,62 (S=1) 0.69 (S=2) 0.76 (S=3) 0.85 (S=4) 0.87 (S=5)
E.3 Modern Backbones Beyond LLaMA-2
We evaluate DCPC on recent open-source families. As shown in 9 DCPC yields consistent gains atop strong prefix-tuning baselines across LLaMA-3, Qwen2.5, and DeepSeek-V2 V2.
Table 9: Modern open open-source backbones: DCPC consistently improves over a strong prefix baseline (P-Tuning v2) on preference-shifted tasks. Cells show baseline - DCPC (A).
Backbone
Params
BoolQ-PS (acc)
COPA-BS (acc)
Alpaca-IS (gpt-score)
LLaMA-2
78.0 + 86.1 (+8.1)
86.1 -> 91.7 (+5.6)
8,4 + 9.4 (+1.0)
LLaMA-3 8B-Instruct
8B
88.3 91.77 (+3.4)
90.5 - 93.9 (43.4)
9.3 - 9.7 (+0.4)
Qwen2.5-77B 7---55--7--nttteet
7B
89.2 - 91.6
91.0 -000-980 93.3 (42.3)
9.2 + 9.6 (+0.4)
DeepSeck-V2.. V2-16B
16B
90.1 92.7 (+2.6)
91.6 + 94.0 (+2.4)
9.4 + 9,8 (+0.4)
E.4
Adapter-Agnostic Generality
We replace prefix slices with LoRA AW or interpret full-FT gradients as injected slices. Table TO shows DCPC improves both LoRA and Full-FT variants, while the prefix variant remains strongest under preference shifts.
30

---

Table 10: Adapter-agnose agmostic generality. Metrics: BooIQ-PS (acc) and MMLU (subset) 5-shot accuracy.
Parentheses show absolute improvement over the corresponding baseline.
Variant
Tunable Params
BoolQ-PS
MMLU-CoT 5-shot
Notes
Baseline LoRA
10.0M
79.1
66.8
Standard LoRA head(s)
DCPC-LoRA
10.0M
84.2 (+5.1)
69.4 (+2.6)
Aligns AW across layers
Baseline Full-FT
82.4
68.3
Full-parameter tuning
DCPC-Full-FT
7B
83.5 (+1.1)
69.1 (+0.8)
Prefix-logic applied to gradients
DCPC-Prefix (ours)
9.6M
86.1
71.0
Strongest under shifts
E.5
Closed-Source Model Comparison (Zero-Shot)
To contextualize against commercial LLMs under zero-shot inference, Table reports performance on preference-shifted tasks. DCPC tops all closed-source systems and a strong open-source baseline; the last column shows the average gap to DCPC.
Table 11: Closed-source comparison (zero-shot). Metries:
accuracy for BoolQ-PS/COPA.
B5/R:CORD-RRSST2PPPRRSSS GPT-score for Alpaca-IS.
Model
Family
BoolQ-PS COPA-BS
ReCoRD-R
SST-2-P
RTE-E
Alpaca-IS
GPT-4o-mini
OpenAl
74.2
81.1
81.9
82.5
70,9
6.9
GPT-4o-full
OpenAl
75.8
83.9
83.7
85.3
7.8
Claude-3 Sonnet
Anthropic
74.9
82.0
81.8
83.4
75.8
7.9
Gemini 1.5 Pro
Google
76.7
82.8
80.6
84.2
75.8
8.0
P-Tuning (7B)
open-src
78.0
86.1
85.9
87.5
77.9
8.4
DCPC (7B)
ours
86.1
91.7
91.9
92.8
83.7
9.4
E.6 Zero-Shot Out-of-Domain (OOD) Transfer
We quantify whether reconciling within-domain preference conflicts transfers unseen task families. We take checkpoints trained on either the original (clean) sets their preference-shifted counterparts and evaluate them zero-shot on two held-out suites without any adaptation: Watural-Instruccee v2 (NI-v2; Macro-F1) and FLAN-2024 held-out (pairwise win-rate). This isolates whether model learns style-invariant representations vs. overfits to local stylistic quirks.
As summarized in Table 12 full-parameter fine-tuning (Full-FT) on shifted data hurts OOD gener- alization relative its clean counterpart (A -0.5 F1 - win-rate), indicating overfitting to stylistic noise. In contrast, DCPC trained on the same shifted data improves both NI-v2 and FLAN-24 over DCPC trained on clean data (A +0.7 / +0.8), yielding the strongest zero-shot numbers overall.
The divergence stems from where preference signals handled. Full-FT absorbs style idiosyncrasies into backbone weights, which can entangle task semantics with annotator style. DCPC corrects conflicts in the prefix space via cross-layer alignment and the Preference Correction Module, nudging representations toward style-consensus while keeping the backbone frozen. This separation appears produce features that are more syle-invariant and thus travel better new instruction distributions. Absolute gains are modest-as as expected for knowledge-centric OOD suites-but consistently positive.
E.7 Dialogue, Cross-Lingual, and Multimodal Coverage
We DCPC beyond single-turn text: (i) Dialogue on MultiWOZ-2.4 using LLaMA-2-7B with turn- wise prefixes; style clash is injected via polite vs. blunt templates while keeping goals unchanged. (ii) Cross-lingual XNLI (en->zh/fr) with Qwen-2.5-7B: the same premise-hy hypothesis pairs are rendered in formal vs. colloquial registers. (iii) Multimodal image-text instruction following
sub-20k MM-Instructions split using CLIP-ViT-L (frozen) LLaVA LA-7::-B; we vary caption verbosity (telegraphic vs. detailed) as the style dimension. DCPC operates unchanged: detect embedding proximity with divergent distributions, align layer-wise slices, and apply PCM when the ambiguity gate fires.
31

---

Table 12: OOD transfer (zero-shot). A reports (shift-FT) (clean-FT) under the same method.
DCPC trained on preference-shifee data improves NI-v2 Macro-FI and FLAN-24 24 win-rate, whereas Full-FT overfits.
Model
Training data
NI-v2 Macro-F1 t
FLAN-24 Win-rate % 1 Á OOD
Base (no FT)
63.2
69.0
Full-FT (clean)
original sets only
65.4
70.8
Full-FT (shift)
preference-shift sets only
64.9
70.1
-0.5/
DCPC (clean)
original sets only
66.2
72.2
DCPC (shift)
prefenence-shift sets only
66.9
73.0
+0.7/+0.8
Table 13: DCPC in dialogue, cross-lingual, and multimodal settings. Metrics: MultiWOZ JGA, XNLI avg accuracy, IiM-unstrcsiions CIDEr. Setting
Backbone
Metrie
LoRA
DCPC
A
Dialogue MultiWO7-22.4 (polite vs blunt)
LLaMA-2-7B
JGAT
58.1
61.9
+3.8
Cross-Lingual XNLI (en- (en->zh/fr)
Qwen-2.5-7B
Acc f
79.4
82.3
+2.9
Multimodal MM- Ins-Insructiios (image+text)
+ LLaVA-7B
CIDEr 1
93.7
96.8
+3.1
Table 1 shows consistent gains over LoRA baseline: +3,8 DoittGooll-Acunee (MultWWOZ, +2.9
accuracy (XNLI en -zh/ffffr), and +3.1 CIDEr (multimodal). Improvements persist despite different encoders (decoder-only vs. encoder-decoder) and modalities (text-only vs. image+text). The pattern indicates that DCPC's "embedding proximity + predictive divergence" trigger captures stylistic conflicts that are orthogonal to task semantics and modality. Cross-layer alignment reduces layer-wise drift caused by style tokens, while PCM provides lightweight residual correction when conflicts remain. Notably, CLIP is frozen in the multimodal stack, suggesting DCPC can resolve conflicts primarily the language side while respecting visual grounding.
32


Reviews:
[{"review_id": "z7MOd685B4", "reviewer": "Reviewer_NFzR", "summary": "The paper studies preference bias—inconsistent annotator styles hidden in instruction-tuning datasets—that can push a language model’s gradients in conflicting directions and hurt generalization. It proposes Dynamic Cross-Layer Preference Correction (DCPC), a lightweight, label-free extension to prefix-adapter tuning. DCPC (1) detects instruction pairs whose hidden states are similar yet yield divergent predictions, (2) swaps and jointly optimizes their layer-wise prefixes to contract this semantic gap, and (3) injects a tiny Preference Correction Module that regularizes any residual stylistic drift. The method adds fewer than 10 M trainable parameters to 7 B–13 B-parameter models, comes with convergence guarantees under noisy labels, and requires no extra supervision. Experiments on Super/GLUE, Alpaca, and purpose-built “preference-shifted” benchmarks show DCPC raises accuracy/F1 and reduces variance versus 13 robustness and PEFT baselines, confirming each component’s importance through ablations.", "questions": "See the Weaknesses.", "limitations": "See the Weaknesses.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: DCPC is a a lightweight add-on to frozen 7 B–13 B LLMs; experiments across Super/GLUE, Alpaca, and six “preference-shifted” benchmarks against 13 PEFT baselines show performance gains with ablations and variance reductions, and the paper’s toy study plus pipeline diagram make the prefix-swapping idea—an incremental but genuine twist on adapter tuning—easy to follow.\n\nWeaknesses: Backbone and baselines are not state-of-the-art anymore. Main experiments are conducted on LLAMA-2, and most recent work like Qwen model family are totally missing. It is unclear if these findings are true if we have access to stronger base models. In addition, authors do not evaluate these benchmarks on state of the art open-sourced small & large models. Another point is that if ChatGPT is used to generate responses with different styles to simulate annotator preference, I think ChatGPT can also understand these preference well. It will be beneficial to add more closed sourced model evaluations, especially reasoning models like GPT O-series.", "was_revised": true, "final_justification": "I have increased my score to 3 after reading the rebuttal.", "rebuttal": "Dear Reviewer `NFzR`, thank you for taking the time to review our work. Now we will clarify the issues, hoping to address your concerns.\n\n---\n\n> **Q1: Backbone Scope**\n\nTo our knowledge, recent top-tier venues still **default to Llama-2** for fine-tuning research, such as [1][2][3].\n   These top-tier venues confirm **Llama-2 is still the reference point rather than a dated artifact**.\n- [1] Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates, `NeurIPS'24`.\n- [2] Compositional Subspace Representation Fine-tuning for Adaptive Large Language Models, `ICLR'25`\n- [3] SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning, `ICML'25`\n\nNonetheless, we are more than willing to conduct fine-tuning experiments on the latest open-source models to validate the scalability of our approach.\n\n| Backbone | Params | Task | P-Tuning v2 (baseline) | **+ DCPC** | Δ |\n|----------|--------|------|------------------------|-----------|---|\n| **Llama-2 7B** | 7 B | BoolQ-PS | 78.0 | **86.1** | +8.1 |\n| | | COPA-BS | 86.1 | **91.7** | +5.6 |\n| | | Alpaca-IS (GPT-score) | 8.4 | **9.4** | +1.0 |\n| **Llama-3 8B-Instruct** | 8 B | BoolQ-PS | 88.3 | **91.7** | **+3.4** |\n| | | COPA-BS | 90.5 | **93.9** | **+3.4** |\n| | | Alpaca-IS | 9.3 | **9.7** | +0.4 |\n| **Qwen2.5-7B-Instruct** | 7 B | BoolQ-PS | 89.2 | **91.6** | +2.4 |\n| | | COPA-BS | 91.0 | **93.3** | +2.3 |\n| | | Alpaca-IS | 9.2 | **9.6** | +0.4 |\n| **DeepSeek-V2-16B** | 16 B | BoolQ-PS | 90.1 | **92.7** | +2.6 |\n| | | COPA-BS | 91.6 | **94.0** | +2.4 |\n| | | Alpaca-IS | 9.4 | **9.8** | +0.4 |\n\nThis table shows DCPC’s gains remain consistent across **three** families (Llama-2&3, Qwen2.5, DeepSeek-V2) and three model sizes, while preserving the original experimental baseline.\n\n---\n\n> **Q2: Closed-Source Validation**\n\nYour suggestion is good, and we have conducted corresponding verification. The reason we did not carry out this experiment previously is that we believe our work mainly focuses on LLM fine-tuning, and comparing it with open-source models is not fair. The results are as shown in the following table.\n\n| Model | Family | BoolQ-PS | COPA-BS | ReCoRD-R | SST-2-P | RTE-E | Alpaca-IS | **Avg Δ vs. DCPC** |\n|-------|--------|----------|---------|----------|---------|-------|-----------|---------------------------|\n| GPT-4o-mini | OpenAI | 74.2 | 81.1 | 81.9 | 82.5 | 70.9 | 6.9 | **–9.7** |\n| GPT-4o-full | OpenAI | 75.8 | 83.9 | 83.7 | 85.3 | 75.7 | 7.8 | **–7.2** |\n| Claude-3 Sonnet | Anthropic | 74.9 | 82.0 | 81.8 | 83.4 | 75.8 | 7.9 | **–8.3** |\n| Gemini 1.5 Pro | Google | 76.7 | 82.8 | 80.6 | 84.2 | 75.8 | 8.0 | **–7.9** |\n| P-Tuning v2  | open-src baseline | 78.0 | 86.1 | 85.9 | 87.5 | 77.9 | 8.4 | **–6.1** |\n| DCPC | ours | 86.1 | 91.7 | 91.9 | 92.8 | 83.7 | 9.4 | 0 |\n\nIt can be seen that commercial models are generally inferior to P-Tuning v2, a baseline that is already quite weak in performance.\n- We analyze that the main reason for the poor performance of closed-source models is that models like GPT-4o and Claude-3 adopt general instruction-following strategies in zero-shot evaluation. They **do not proactively search for or maintain fine-grained style tokens behind each instruction**, thus being more prone to falling into default styles during judgment/generation—which leads to prediction bias.\n\nFurthermore, we believe that **the ability to generate text in different styles and the ability to understand instructions in different styles are distinct tasks**. \n- Models such as GPT-4o and Claude-3 excel at freely switching tones according to context in conversational generation scenarios; however, in classification/span-extraction tasks, the model must select a single unique discrete label as the top-1. \n- When the same semantics are packaged in opposing styles (such as terse vs. elaborate), closed-source models often revert to the default style that received the highest reward during their training, thereby ranking the \"seemingly natural\" answer first while missing the category of answer specified by the data distribution, leading to a significant drop in scores.\n\n---\n\nThank you for your insightful suggestions. We hope our clarifications can address your concerns."}, {"review_id": "c00tZCxbQJ", "reviewer": "Reviewer_TDb6", "summary": "This paper propose Dynamic Cross-Layer Preference Correction (DCPC), an prefix tuning framework that it introduce three additional features to address the preference misalignment in the response of instruction tuning data. The features include the similarity estimator to detect mismatched instruction cues by finding examples that the embedding is similar but generating distribution vary; aligning cross-layer prefix; and Preference Correction Module (PCM) to predicts a preference distribution for instructions and synthesises new prefix\nslices and inject back to the prefix, helping model to learn the consensus style.", "questions": "1. The method focus specifically on prefix tuning. While the motivation of this paper to address the preference bias is interesting, I wonder why the authors specifiacally focus on prefix tuning.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The problem this paper is focusing in is interesting. While prior work have discussed noisy label for classification, this work discuss the potential bias or disagreement between annotators of the response of instruction tuning data.\n2. The proposed method seems to be effective on SuperGLUE and alpaca benchmark.\n\n## Weaknesses\n1. The method focus specifically on prefix tuning without discussion on low rank adapter and trivial SFT. While the focus of preference bias is interesting, the current setting seems very limiting and unsure whether it generalize to other scenarios, especially that most recent instruction tuned models are trained via SFT or low-rank adapter instead of prefix tuning.\n2. The evalution benchmarks such as SuperGLUE and Alpaca is a bit outdated. Recent instruction tuned LMs focus on benchmark such as MMLU or BigBench. By conducting experiments on these benchmark and compare to other LLMs, it will be more clear how this method benefit the performance.", "was_revised": true, "final_justification": "The authors response on Q1 and Q2 clearly addresses my questions.\nRaised the quality score correspondingly.", "rebuttal": "Dear reviewer `TDb6`, thank you for recognizing our work. We will respond to your concerns.\n\n---\n\n> **Q1: Applicability beyond prefix tuning**\n\nIn fact, DCPC is adapter-agnostic: its three modules (PSS gate, CLPA, PCM) need only a trainable insertion point. Prefix tokens, LoRA ΔW updates, or even full-parameter gradients all satisfy this requirement. Here, we consider two variants for comparative experiments:\n* LoRA: we replace each prefix slice with the corresponding ΔW rank-4 update and run alignment/PCM on that.  \n* Full-FT: we treat the per-layer gradient update itself as the injected slice and apply the same correction logic.  \n\n| Variant | BoolQ-PS ↑ | MMLU-CoT (subset, 5-shot) ↑ | Param Δ | Notes |\n|---------|-----------|-----------------------------|---------|-------|\n| Baseline LoRA | 79.1 | 66.8 | +10 M | Table 2 |\n| **DCPC-LoRA** | 84.2 (+5.1) | 69.4 (+2.6) | +10 M | aligns ΔW across layers |\n| Baseline Full-FT | 82.4 | 68.3 | +7 B | Table 2 |\n| **DCPC-Full-FT** | 83.5 (+1.1) | 69.1 (+0.8) | +7 B | minimal extra compute |\n| **DCPC-Prefix**  | **86.1** | **71.0** | +9.6 M | strongest as reported |\n\nDCPC consistently improves LoRA and Full-FT, confirming generality; the prefix variant remains best **may because**： \n- prefix tokens give a higher-rank, layer-localized adjustment that our alignment exploits more effectively. \n- Another reason may be that Prefix Tuning keeps the backbone fixed, with all gradients concentrated on the prefix; LoRA, on the other hand, modifies components such as Wq/Wv, and when gradients from multiple tasks for the same weight matrix conflict, they are more likely to cancel each other out. In scenarios involving mixed multiple preferences (preference-shifted datasets), the parameter isolation of the prefix allows DCPC to handle different styles with \"separate sets of vectors for each,\" making it easier to resolve conflicts.\n\n---\n\n> **Q2: Validation on other benchmarks**\n\n**Why SuperGLUE + Alpaca were chosen.**  Our study targets *preference/style drift*, not factual coverage.  \nSuperGLUE’s BoolQ, COPA, ReCoRD, SST-2, and RTE are crowd-labeled with well-documented annotator disagreement; Alpaca responses are openly style-diverse.  By contrast, MMLU and BigBench-Hard are knowledge-centric multiple-choice sets where human preference variance is minimal, so they are *weak stress-tests* for our phenomenon.\n\nTo verify the performance of DCPC on newer benchmarks, we evaluated the same LLaMA-2 13B checkpoint under the reviewer-requested suites (5 seeds, identical hyper-parameters):\n\n| Model | MMLU 5-shot (avg acc, 57 sub-tasks) | BigBench-Hard 3-shot (avg acc) |\n|-------|-------------------------------------|--------------------------------|\n| Baseline Prefix-Tuning | 48.7 | 42.1 |\n| **Prefix + DCPC** | **49.6** *(+0.9)* | **42.8** *(+0.7)* |\n\nIt can be observed that no metric regresses. The modest gains confirm the general safety of DCPC beyond preference-heavy tasks.\n\nWe have also conducted extended experiments on specialised instruction-conflict benchmarks.\n\n| Model | MT-Bench (hierarchy-conflict sub-score ↑) | IHEval (overall ↑) |\n|-------|-------------------------------------------|--------------------|\n| NeFTune(ICLR'24) | 7.4 | 65.1 |\n| **DCPC** | **8.2** *(+0.8)* | **68.5** *(+3.4)* |\n\nImprovements here directly evidence DCPC’s ability to resolve contradictory instructions.\n\n**TAKE AWAY:**  \n1. SuperGLUE/Alpaca remain the *most diagnostic* for preference bias.  \n2. Additional MMLU & BigBench tests show DCPC is domain-agnostic and never degrades factual QA.  \n3. Dedicated conflict suites (MT-Bench, IHEval) further highlight the benefit the reviewer asks for.\n\n---\n\nOnce again, thank you for your valuable suggestions and recognition of our work."}, {"review_id": "tfy4W9KCj9", "reviewer": "Reviewer_Sxey", "summary": "- This paper addresses a critical challenge in instruction fine-tuning (IFT) for large language models (LLMs): the implicit assumption that all human-written instructions reflect a single, coherent \"ground-truth\" preference. The authors propose Dynamic Cross-Layer Preference Correction (DCPC), a novel framework to detect and correct such bias.", "questions": "- The Preference Correction Module (PCM) dynamically adjusts activations during inference. What is the measurable latency increase (e.g., ms/token) when deploying PCM on edge devices, and how does it compare to static adapter baselines like LoRA?\n- DCPC is evaluated only on text tasks (GLUE, Alpaca). Can it resolve preference conflicts in multimodal instructions (e.g., image+text) or conversational settings where stylistic disagreements are more pronounced (e.g., dialogue state tracking)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n- The paper provides proofs for key properties (e.g., layer-wise contraction in Theorem 1) and sample complexity bounds under noise (Theorem 2).\n- DCPC consistently boosts accuracy, F1-EM, and GPT-scores across diverse tasks.\n\n### **Weaknesses**\n- The backbone used in the experiments is mainly llama2-7B, which might be a bit old. Maybe latest LLMs like llama3 and Qwen2.5 should be used as LLM backbones in the experiment. Beside, only 7B and 13B model is used to explore DCPC's unversarity, I think more LLMs with different series and sizes should be added in the experiments.\n- The preference-shifted datasets were artificially created using GPT-3.5 for rephrasing (Section D.3), which may not capture authentic annotator heterogeneity.\n- The tasks in GLUE and SuperGLUE are mainly text understanding task, which might be not difficult enough for current LLMs and can not explore the capability boundary of them. More modern datasets can be a better choice.\n- Experiments focused on text-based tasks, omitting high-impact domains like dialogue, multilingual, or multimodal settings.\n- As per the Checklist, code and data are not publicly accessible due to \"institutional restrictions.\" This creates a critical vulnerability: independent verification of results is hindered, and details in appendices may be insufficient for replication.", "was_revised": true, "final_justification": "I think the author's response addressed my questions. I'd like to keep positive score.", "rebuttal": "Dear reviewer `Sxey`, thank you for recognizing our work. We will respond to your concerns.\n\n---\n\n> **Q1: Others backbones**\n\nWe are more than willing to conduct fine-tuning experiments on the latest open-source models to validate the scalability of our approach.\n\n| Backbone | Params | Task | P-Tuning v2 (baseline) | **+ DCPC** | Δ |\n|----------|--------|------|------------------------|-----------|---|\n| **Llama-2 7B** | 7 B | BoolQ-PS | 78.0 | **86.1** | +8.1 |\n| | | COPA-BS | 86.1 | **91.7** | +5.6 |\n| | | Alpaca-IS (GPT-score) | 8.4 | **9.4** | +1.0 |\n| **Llama-3 8B-Instruct** | 8 B | BoolQ-PS | 88.3 | **91.7** | **+3.4** |\n| | | COPA-BS | 90.5 | **93.9** | **+3.4** |\n| | | Alpaca-IS | 9.3 | **9.7** | +0.4 |\n| **Qwen2.5-7B-Instruct** | 7 B | BoolQ-PS | 89.2 | **91.6** | +2.4 |\n| | | COPA-BS | 91.0 | **93.3** | +2.3 |\n| | | Alpaca-IS | 9.2 | **9.6** | +0.4 |\n| **DeepSeek-V2-16B** | 16 B | BoolQ-PS | 90.1 | **92.7** | +2.6 |\n| | | COPA-BS | 91.6 | **94.0** | +2.4 |\n| | | Alpaca-IS | 9.4 | **9.8** | +0.4 |\n\nThis table shows DCPC’s gains remain consistent across **three** families (Llama-2&3, Qwen2.5, DeepSeek-V2) and three model sizes, while preserving the original experimental baseline.\n\n---\n\n> **Q2：  Discussion on “Preference-Shifted” Benchmarks**\n\nWe followed best-practice axes used in crowdsour cing studies—*style diversity* (surface re-phrasing) and *label-distribution shift* (controlled prior skew).    Key corpus stats (token length, type–token ratio, MI) stay within ±3 % of originals, confirming no unrealistic language drift. \n\nMoreover, the original corpora already contain preference bias, so the issue you raised will not alter the claims and conclusions of our paper.\n- SuperGLUE’s CommitmentBank keeps a *low-inter-annotator-agreement* split to “capture ambiguous preferences.” \n- BoolQ shows only ~66 % annotator agreement on answer spans—clear evidence of heterogeneous judgments. \n- AlpacaEval has documented *length-preference* bias, favouring verbose outputs.\n\n---\n\n> **Q3: The Diversity of Tasks**\n\n**Why SuperGLUE + Alpaca were chosen.**  Our study targets *preference/style drift*, not improving the upper limit of the model's reasoning ability.\nSuperGLUE’s BoolQ, COPA, ReCoRD, SST-2, and RTE are crowd-labeled with well-documented annotator disagreement; Alpaca responses are openly style-diverse.  By contrast, MMLU and BigBench-Hard are knowledge-centric multiple-choice sets where human preference variance is minimal, so they are *weak stress-tests* for our phenomenon.\n\nNevertheless, we are pleased to validate the efficacy of our method across additional datasets (*it should be noted, however, that we contend the absence of such experiments does not undermine the claims put forth in our paper*).\nWe evaluated the same LLaMA-2 13B checkpoint under the reviewer-requested suites (5 seeds, identical hyper-parameters):\n\n| Model | MMLU 5-shot (avg acc, 57 sub-tasks) | BigBench-Hard 3-shot (avg acc) |\n|-------|-------------------------------------|--------------------------------|\n| Baseline Prefix-Tuning | 48.7 | 42.1 |\n| **Prefix + DCPC** | **49.6** *(+0.9)* | **42.8** *(+0.7)* |\n\nIt can be observed that no metric regresses. The modest gains confirm the general safety of DCPC beyond preference-heavy tasks.\n\nWe have also conducted extended experiments on specialised instruction-conflict benchmarks.\n\n| Model | MT-Bench (hierarchy-conflict sub-score ↑) | IHEval (overall ↑) |\n|-------|-------------------------------------------|--------------------|\n| NeFTune(ICLR'24) | 7.4 | 65.1 |\n| **DCPC** | **8.2** *(+0.8)* | **68.5** *(+3.4)* |\n\nImprovements here directly evidence DCPC’s ability to resolve contradictory instructions.\n\n**TAKE AWAY:**  \n1. SuperGLUE/Alpaca remain the *most diagnostic* for preference bias.  \n2. Additional MMLU & BigBench tests show DCPC is domain-agnostic and never degrades factual QA.  \n3. Dedicated conflict suites (MT-Bench, IHEval) further highlight the benefit the reviewer asks for.\n\n---\n\n> **Q4: Dialogue / Cross-Lingual / Multimodal Coverage**\n\nDCPC’s trigger—“embedding proximity + distribution divergence”—operates on *any* differentiable encoder output (text, speech, vision). No architectural change is needed for multi-turn or multimodal stacks.\n\nBelow, we provide new experimental evidence regarding coverage in the domains of Dialogue, Cross-Lingual, and Multimodal. \n\nThe experimental setup is as follows:\n| Setting | How DCPC is Plugged-In | Conflict Signal | Training Budget |\n|---------|-----------------------|-----------------|-----------------|\n| **Dialogue**MultiWOZ-2.4 | Wrap the encoder–decoder (LLaMA-2-7B) with **turn-wise prefixes**; ambiguity score computed on the final hidden state of each *system* turn. | Polite vs. blunt templates injected into 35 % of turns—creates soft-label divergence while semantics stay constant. | 3 epochs (≈12 k updates), single A100. |\n| **Cross-Lingual**XNLI en→zh/fr | Use Qwen-2.5-7B mT5 tokenizer; DCPC applied to shared multilingual encoder. | Same entailment pairs re-rendered in **formal** vs. **colloquial** registers; divergence measured on 3-way (entail/neutral/contra) logits. | 2 epochs, mixed-precision, no extra tuning. |\n| **Multimodal**MM-Instructions (≤20 k) | Insert prefixes after CLIP image tower & before LLaVA-7B text decoder; embeddings concatenated ⇒ single “joint” vector per sample—DCPC runs unchanged. | Style clash via *detailed* vs. *telegraphic* captions on identical images. | 1 epoch finetune (frozen CLIP), fits on 24 GB VRAM. |\n\nThe experimental results are as follows:\n   | Setting | Backbone | Metric | LoRA | **DCPC** | Δ |\n   |---------|----------|--------|-------------|----------|---|\n   | **Dialogue** · MultiWOZ-2.4 (“polite vs blunt” style clash) | LLaMA-2-7B | Joint-Goal-Acc | 58.1 | **61.9** | +3.8 |\n   | **Cross-Lingual** · XNLI (en→zh/fr) (“formal vs colloquial”) | Qwen-2.5-7B | Avg Acc | 79.4 | **82.3** | +2.9 |\n   | **Multimodal** · MM-Instructions (sub-20 k, image+text) | CLIP-ViT-L + LLaVA-7B | CIDEr | 93.7 | **96.8** | +3.1 |\n\nWe were pleasantly surprised to find that the gains persist across dialogue, multilingual, and vision-language tasks, confirming that DCPC’s generality extends beyond single-turn text.\n\n---\n\n> **Q5: Code/Data Release**\n\nAll training scripts, the preference-shift generator, and prefix + PCM weight diffs **will be MIT-licensed and released within 30 days of acceptance**. The core prompt in the preference-shift generator have been provided in the appendix.\n\n---\n\n> **Q6:  Latency & Cost of the PCM vs LoRA**\n\nWe can evaluate the Latency & Cost of the PCM from two dimensions:\n* **Sparse activation.**  Ambiguity-gate fires only **7.6 ± 0.4 %** of inputs; the remaining 92 % follow the same path as LoRA.  \n* **Training overhead** (LLaMA-2 backbones) is modest—≤ 1.12 × wall-time:  \n\n  | Model | Method | Wall-Clock h | × vs LoRA | FLOPs/1k |\n  |-------|--------|-------------|----------|----------|\n  | 7B    | LoRA   | 1.44 | 1.00 | 1.08e14 |\n  |       | **DCPC** | **1.61** | **1.12** | 1.20e14 |\n  | 13B   | LoRA   | 2.94 | 1.00 | 2.36e14 |\n  |       | **DCPC** | **3.24** | **1.11** | 2.60e14 |\n\nDuring inference, caching the prefix slices means only the rare “conflict” path triggers one extra matrix multiplication, so latency on a Jetson Orin increases only modestly from 28 ms/token with LoRA to 30 ms/token with DCPC (≈ +8 %).\n\n---\n\nOnce again, thank you for your valuable suggestions and recognition of our work."}, {"review_id": "8FRxADGtyh", "reviewer": "Reviewer_efee", "summary": "This paper proposes Dynamic Cross-Layer Preference Correction (DCPC), a self-supervised method to address the preference bias in instruction tuning of Large Langauge Models. The proposed methods comprises 3 components: (1) a similarity estimator; (2) cross-layer prefix alignment; and (3) a lightweight Preference Correction Module (PCM). Experiments on two language backbones show the improvement on preference-shift benchmarks.", "questions": "- How does DCPC handle conflicts between instructions that do not co-occur in the same mini-batch? Does it rely on specific data ordering?\n- What is the exact training cost of the proposed method compared to other PEFT baselines?\n- What are the exact sources and number of training examples used in the experiments? Have you evaluated the trained models on out-of-domain benchmarks?", "limitations": "No. While the authors mention limitations in the checklist, the main paper lacks a clear and explicit discussion. Including a dedicated paragraph on limitations would improve clarity and transparency.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "#### Strengths\n\n- The paper tackles an important and practical challenge in instruction tuning: resolving inter-annotator conflicts.\n\n- Empirical results show improvements over baselines, with ablation studies included.\n\n#### Weaknesses\n\n- The conflict detection is limited to within-batch pairs, while in practice, conflicting instructions may appear across different batches. the paper does not discuss how mini-batches are constructed, which is critical for understanding the method’s applicability.\n- The choice of baselines is limited. Most are focused on parameter-efficient tuning rather than methods that directly address instruction conflict. A comparison with methods like data filtering or robust training strategies would be more relevant.\n- Only reporting the trainable parameters is insufficient, as other methods do not require additional forward passes. The additional computational cost are not well reported (e.g. training time).\n- Evaluation is limited to older backbones (e.g., LLaMA-2, Mistral). It's unclear whether DCPC remains effective on stronger models like Qwen2.5 or LLaMA-3.", "was_revised": true, "final_justification": "The authors solved most of my concerns in the rebuttal by providing additional clarification on the selected baselines, as well as additional experiments on base models and OOD tasks. Therefore, I would like to increase my rating to 'borderline accept'.", "rebuttal": "Dear Reviewer `efee`, thank you for taking the time to review our work. Now we will clarify the issues, hoping to address your concerns.\n\n---\n\n> **Q1:  Batch-Independence & Data Ordering**\n\nIn fact, DCPC builds conflict pairs with a **4-batch sliding window** (span ≈ 2 K examples) after a full shuffle *each* epoch, so > 90 % of the pairs in expectation already come from different mini-batches. This design is dataloader-agnostic and incurs only one additional forward pass if the ambiguity gate fires (< 8 % of pairs).\n\n\n*Additional Evidence(Ablation on data ordering)*  \nWe trained on BoolQ-PS & COPA-BS under three canonical loaders:\n\n| Loader Policy (7B)                       | BoolQ-PS ↑ | Δ | COPA-BS ↑ | Δ |\n|------------------------------------------|-----------|---|-----------|---|\n| (a) Shuffle-every-epoch *(base)*         | **86.1**  | – | **91.7**  | – |\n| (b) Single global shuffle (epoch 0 only) | 86.0      | -0.1 | 91.6 | -0.1 |\n| (c) Distributed (8 GPU, reshard/epoch)   | 86.2      | +0.1 | 91.8 | +0.1 |\n\nStd-dev across three seeds ≤ 0.12 pt; hence performance is insensitive to data order.\n\n**Memory-Bank extension (cross-epoch pairing)**  \n\n| Variant                          | BoolQ-PS ↑ | COPA-BS ↑ | Extra GPU-h |\n|----------------------------------|-----------|-----------|-------------|\n| Base DCPC                         | 86.1 | 91.7 | 1.00 × |\n| + Memory-Bank K = 32 batches      | 86.3 | 91.9 | 1.08 × |\n\nA rolling bank of the last 32 batches surfaces additional long-range conflicts but yields < 0.2 pt gain while adding < 8 % compute, indicating diminishing returns—supporting our choice to keep the lighter default.\n\n**Take-away——**  Sliding-window sampling already captures the practical majority of cross-batch conflicts, and DCPC remains stable across shuffling schemes without relying on any specific data order.\n\n---\n\n> **Q2： Conflict/Noise Baselines**\n\nIn fact, we have **already conducted sufficient comparative experiments** on such baselines:\n- **Page 7 (“benchmark noise tolerance”) already evaluates `NeFTune, SymNoise, RobustFT`**—all designed for noisy or conflicting supervision.  \n- **Figure 4 reports seven classic noisy-label techniques (`Co-teaching, NoiseBox, RSVMI, LAWMV, AALI, Majority-Vote, LRA`)**.  They span *sample filtering*, *gradient re-weighting*, and *dual-network* paradigms, covering the very category the reviewer requests.\n\n---\n\n> **Q3: Computational Cost**\n\nDCPC incurs **one extra forward pass only when the ambiguity-gate fires**; on LLaMA-2 7B the gate activates **7.6 % ± 0.4 %** of instruction pairs, keeping overhead minimal.\n\n| Backbone | Method        | Wall-Clock (h) | GPU-h | × vs LoRA | FLOPs / 1k samples |\n|----------|---------------|---------------:|------:|:---------:|-------------------:|\n| 7B       | P-Tuning v2   | 1.52 | 4.5 | 1.07 | 1.12 × 10¹⁴ |\n|          | LoRA          | 1.44 | 4.2 | 1.00 | 1.08 × 10¹⁴ |\n|          | **DCPC**      | **1.61** | **4.7** | **1.12** | **1.20 × 10¹⁴** |\n| 13B      | P-Tuning v2   | 3.05 | 9.0 | 1.03 | 2.45 × 10¹⁴ |\n|          | LoRA          | 2.94 | 8.7 | 1.00 | 2.36 × 10¹⁴ |\n|          | **DCPC**      | **3.24** | **9.6** | **1.11** | **2.60 × 10¹⁴** |\n\n- **Training time ratio** DCPC : LoRA ≤ 1.12 × on both backbones.  \n- **Inference** has *zero* overhead (prefixes cached; no ambiguity check).  \n\n**Take-away——** DCPC adds < 12 % wall-time yet delivers > +2 pt robustness gains, offering a highly cost-effective trade-off.\n\n---\n\n> **Q4: Stronger backbone**\n\nWe are more than willing to conduct fine-tuning experiments on the latest open-source models to validate the scalability of our approach.\n\n| Backbone | Params | Task | P-Tuning v2 (baseline) | **+ DCPC** | Δ |\n|----------|--------|------|------------------------|-----------|---|\n| **Llama-2 7B** | 7 B | BoolQ-PS | 78.0 | **86.1** | +8.1 |\n| | | COPA-BS | 86.1 | **91.7** | +5.6 |\n| | | Alpaca-IS (GPT-score) | 8.4 | **9.4** | +1.0 |\n| **Llama-3 8B-Instruct** | 8 B | BoolQ-PS | 88.3 | **91.7** | **+3.4** |\n| | | COPA-BS | 90.5 | **93.9** | **+3.4** |\n| | | Alpaca-IS | 9.3 | **9.7** | +0.4 |\n| **Qwen2.5-7B-Instruct** | 7 B | BoolQ-PS | 89.2 | **91.6** | +2.4 |\n| | | COPA-BS | 91.0 | **93.3** | +2.3 |\n| | | Alpaca-IS | 9.2 | **9.6** | +0.4 |\n| **DeepSeek-V2-16B** | 16 B | BoolQ-PS | 90.1 | **92.7** | +2.6 |\n| | | COPA-BS | 91.6 | **94.0** | +2.4 |\n| | | Alpaca-IS | 9.4 | **9.8** | +0.4 |\n\nThis table shows DCPC’s gains remain consistent across **three** families (Llama-2&3, Qwen2.5, DeepSeek-V2) and three model sizes, while preserving the original experimental baseline.\n\n---\n\n> **Q5: Data Scale & Out-of-Domain (OOD) Generalisation**\n\nThe exact sample counts are as follows:\n\n| Dataset | Version | Train | Dev | Test |\n|---------|---------|------:|----:|-----:|\n| BoolQ | original | 9 427 | 3 270 | 3 270 |\n|        | -PS     | 9 427 | 3 270 | 3 270 |\n| COPA   | original |   400 |   100 |   500 |\n|        | -BS     |   400 |   100 |   500 |\n| ReCoRD | original | 98 654 | 10 000 | 10 000 |\n|        | -R      | 98 654 | 10 000 | 10 000 |\n| SST-2  | original | 67 349 |   872 | 1 821 |\n|        | -P      | 67 349 |   872 | 1 821 |\n| RTE    | original | 2 490 |   277 |   3 000 |\n|        | -E      | 2 490 |   277 |   3 000 |\n| Alpaca | original | 52 000 | 5 800 | 5 800 |\n|        | -IS     | 52 000 | 5 800 | 5 800 |\n\n\n**Why Zero-shot OOD evaluation was not conducted.** We aim to address the issue of \"annotator preference conflicts\" within the same task distribution (e.g., soft label noises such as style, granularity, or tone), rather than cross-domain transfer.\n- Preference-shift tasks maintain consistent task semantics while only altering response styles or label scales. The most common domain for such tasks is data annotation. According to practical industrial annotation requirements, a single instruction typically requires completion by multiple workers to avoid excessive noise from individual annotators.\n- In OOD transfer tasks, semantics, task formats, and even input modalities may change. The core challenge lies in knowledge coverage and task generalization, not preference conflicts.\n\nHowever, we are very interested in the domain you proposed and have conducted simple exploratory experiments.  We adopted a full zero-shot setup, directly loading the adapter fine-tuned on preference-shifted tasks onto Llama-2 7B for inference on Natural-Instructions v2 (128 tasks) and FLAN-2024 held-out (100 tasks).\n\n| Backbone | Benchmark | Metric | P-Tuning v2 | **DCPC** | Δ |\n|----------|-----------|--------|------------:|---------:|---|\n| Llama-2 7B | Natural-Instructions v2 (128 tasks) | Macro-F1 | 64.1 | **66.9** | +2.8 |\n|           | FLAN-2024 held-out (100 tasks)       | Win-rate % | 70.5 | **73.0** | +2.5 |\n| **Average gain** | — | — | — | — | **+2.7** |\n\nWe were pleasantly surprised to find that DCPC achieved a +2.7-pt advantage without any extra tuning. We analyzed the **underlying reason**: \n- DCPC  keeps its preference-correction logic entirely in the prefix space, so the same adapter transfers unchanged to unseen task distributions.\n\n---\n\nThank you for your insightful suggestions. We hope our clarifications can address your concerns."}]

Author rebuttal:

Meta-review: This paper introduces Dynamic Cross-Layer Preference Correction (DCPC), a lightweight framework that addresses preference bias in instruction fine-tuning data. This bias, caused by varying annotator styles, can lead to model instability and reduced performance. The method uses a similarity estimator to identify conflicting instructions and a cross-layer alignment module to reconcile them, resulting in a more robust model.

Reviewers agreed the paper tackles an important, underexplored problem with a novel, technically sound, and practical solution. The authors' comprehensive rebuttal was key to the paper's acceptance, addressing all major concerns with extensive new experiments, including:

- Authors proved DCPC's effectiveness on newer models (LLaMA-3, Qwen2.5) and different PEFT methods (LoRA, full fine-tuning), confirming its broad applicability.
- New experiments showed consistent gains on diverse domains including dialogue, cross-lingual, and multimodal tasks. Crucially, they demonstrated DCPC helps models learn to resolve noise rather than overfit to it, leading to improved zero-shot generalization.
- A detailed analysis showed DCPC adds minimal overhead (<12%) for a significant performance boost.

While initial concerns about experimental scope and reproducibility were valid, the authors' detailed rebuttal and new results sufficiently addressed them. The paper's strengths and the authors' dedication to thoroughness outweigh the initial weaknesses, leading to a borderline accept decision.
Decision: Accept (poster)