Title: Whose Instructions Count? Resolving Preference Bias in Instruction Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Instruction Fine-Tuning

Preference Bias

Self-Supervised Learning
Abstract: Instruction fine-tuning (IFT) has emerged as a ubiquitous strategy for specializing large language models (LLMs), yet it implicitly assumes a single, coherent "ground-truth" preference behind all human-written instructions. In practice, annotators differ in the styles, emphases, and granularities they prefer, introducing preference bias that can erode both robustness and generalization. We propose Dynamic Cross-Layer Preference Correction (\textsc{DCPC}), it couples (i) a preference-sensitive similarity estimator that detects mismatched instructional cues, (ii) cross-layer prefix alignment to reconcile semantic representations across transformer layers, and (iii) a lightweight Preference Correction Module (PCM) that dynamically adjusts hidden states to honor the inferred dominant preference.   On five Super/GLUE tasks and the Alpaca set—plus six preference-shifted variants—DCPC boosts accuracy/F1-EM by 4.0–6.7 points and gpt-score by +0.7, while cutting inter-seed variance up to 35% on LlaMA-2 13B and Mistral-7B, setting a new state of the art for robust instruction tuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Dynamic Cross-Layer Preference Correction (DCPC), a self-supervised method to address the preference bias in instruction tuning of Large Langauge Models. The proposed methods comprises 3 components: (1) a similarity estimator; (2) cross-layer prefix alignment; and (3) a lightweight Preference Correction Module (PCM). Experiments on two language backbones show the improvement on preference-shift benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n\n- The paper tackles an important and practical challenge in instruction tuning: resolving inter-annotator conflicts.\n\n- Empirical results show improvements over baselines, with ablation studies included.\n\n#### Weaknesses\n\n- The conflict detection is limited to within-batch pairs, while in practice, conflicting instructions may appear across different batches. the paper does not discuss how mini-batches are constructed, which is critical for understanding the method’s applicability.\n- The choice of baselines is limited. Most are focused on parameter-efficient tuning rather than methods that directly address instruction conflict. A comparison with methods like data filtering or robust training strategies would be more relevant.\n- Only reporting the trainable parameters is insufficient, as other methods do not require additional forward passes. The additional computational cost are not well reported (e.g. training time).\n- Evaluation is limited to older backbones (e.g., LLaMA-2, Mistral). It's unclear whether DCPC remains effective on stronger models like Qwen2.5 or LLaMA-3.\n\nQUESTIONS:\n- How does DCPC handle conflicts between instructions that do not co-occur in the same mini-batch? Does it rely on specific data ordering?\n- What is the exact training cost of the proposed method compared to other PEFT baselines?\n- What are the exact sources and number of training examples used in the experiments? Have you evaluated the trained models on out-of-domain benchmarks?\n\nLIMITATIONS:\nNo. While the authors mention limitations in the checklist, the main paper lacks a clear and explicit discussion. Including a dedicated paragraph on limitations would improve clarity and transparency.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper addresses a critical challenge in instruction fine-tuning (IFT) for large language models (LLMs): the implicit assumption that all human-written instructions reflect a single, coherent \"ground-truth\" preference. The authors propose Dynamic Cross-Layer Preference Correction (DCPC), a novel framework to detect and correct such bias.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n- The paper provides proofs for key properties (e.g., layer-wise contraction in Theorem 1) and sample complexity bounds under noise (Theorem 2).\n- DCPC consistently boosts accuracy, F1-EM, and GPT-scores across diverse tasks.\n\n### **Weaknesses**\n- The backbone used in the experiments is mainly llama2-7B, which might be a bit old. Maybe latest LLMs like llama3 and Qwen2.5 should be used as LLM backbones in the experiment. Beside, only 7B and 13B model is used to explore DCPC's unversarity, I think more LLMs with different series and sizes should be added in the experiments.\n- The preference-shifted datasets were artificially created using GPT-3.5 for rephrasing (Section D.3), which may not capture authentic annotator heterogeneity.\n- The tasks in GLUE and SuperGLUE are mainly text understanding task, which might be not difficult enough for current LLMs and can not explore the capability boundary of them. More modern datasets can be a better choice.\n- Experiments focused on text-based tasks, omitting high-impact domains like dialogue, multilingual, or multimodal settings.\n- As per the Checklist, code and data are not publicly accessible due to \"institutional restrictions.\" This creates a critical vulnerability: independent verification of results is hindered, and details in appendices may be insufficient for replication.\n\nQUESTIONS:\n- The Preference Correction Module (PCM) dynamically adjusts activations during inference. What is the measurable latency increase (e.g., ms/token) when deploying PCM on edge devices, and how does it compare to static adapter baselines like LoRA?\n- DCPC is evaluated only on text tasks (GLUE, Alpaca). Can it resolve preference conflicts in multimodal instructions (e.g., image+text) or conversational settings where stylistic disagreements are more pronounced (e.g., dialogue state tracking)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose Dynamic Cross-Layer Preference Correction (DCPC), an prefix tuning framework that it introduce three additional features to address the preference misalignment in the response of instruction tuning data. The features include the similarity estimator to detect mismatched instruction cues by finding examples that the embedding is similar but generating distribution vary; aligning cross-layer prefix; and Preference Correction Module (PCM) to predicts a preference distribution for instructions and synthesises new prefix\nslices and inject back to the prefix, helping model to learn the consensus style.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The problem this paper is focusing in is interesting. While prior work have discussed noisy label for classification, this work discuss the potential bias or disagreement between annotators of the response of instruction tuning data.\n2. The proposed method seems to be effective on SuperGLUE and alpaca benchmark.\n\n## Weaknesses\n1. The method focus specifically on prefix tuning without discussion on low rank adapter and trivial SFT. While the focus of preference bias is interesting, the current setting seems very limiting and unsure whether it generalize to other scenarios, especially that most recent instruction tuned models are trained via SFT or low-rank adapter instead of prefix tuning.\n2. The evalution benchmarks such as SuperGLUE and Alpaca is a bit outdated. Recent instruction tuned LMs focus on benchmark such as MMLU or BigBench. By conducting experiments on these benchmark and compare to other LLMs, it will be more clear how this method benefit the performance.\n\nQUESTIONS:\n1. The method focus specifically on prefix tuning. While the motivation of this paper to address the preference bias is interesting, I wonder why the authors specifiacally focus on prefix tuning.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies preference bias—inconsistent annotator styles hidden in instruction-tuning datasets—that can push a language model’s gradients in conflicting directions and hurt generalization. It proposes Dynamic Cross-Layer Preference Correction (DCPC), a lightweight, label-free extension to prefix-adapter tuning. DCPC (1) detects instruction pairs whose hidden states are similar yet yield divergent predictions, (2) swaps and jointly optimizes their layer-wise prefixes to contract this semantic gap, and (3) injects a tiny Preference Correction Module that regularizes any residual stylistic drift. The method adds fewer than 10 M trainable parameters to 7 B–13 B-parameter models, comes with convergence guarantees under noisy labels, and requires no extra supervision. Experiments on Super/GLUE, Alpaca, and purpose-built “preference-shifted” benchmarks show DCPC raises accuracy/F1 and reduces variance versus 13 robustness and PEFT baselines, confirming each component’s importance through ablations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: DCPC is a a lightweight add-on to frozen 7 B–13 B LLMs; experiments across Super/GLUE, Alpaca, and six “preference-shifted” benchmarks against 13 PEFT baselines show performance gains with ablations and variance reductions, and the paper’s toy study plus pipeline diagram make the prefix-swapping idea—an incremental but genuine twist on adapter tuning—easy to follow.\n\nWeaknesses: Backbone and baselines are not state-of-the-art anymore. Main experiments are conducted on LLAMA-2, and most recent work like Qwen model family are totally missing. It is unclear if these findings are true if we have access to stronger base models. In addition, authors do not evaluate these benchmarks on state of the art open-sourced small & large models. Another point is that if ChatGPT is used to generate responses with different styles to simulate annotator preference, I think ChatGPT can also understand these preference well. It will be beneficial to add more closed sourced model evaluations, especially reasoning models like GPT O-series.\n\nQUESTIONS:\nSee the Weaknesses.\n\nLIMITATIONS:\nSee the Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}