Debiasing Online Preference Learning via Preference Feature Preservation

Authors: Dongyoung Kim, Jaehyung Kim, Jinsung Yoon, Jinwoo Shin
Primary area: generative models
Keywords: large language model, alignment, bias, preference

Abstract:
While various preferred features determine human preferences, current preference learning frameworks for large language models (LLMs) simplify them with binary pairwise comparisons and scalar rewards. This simplification could make LLMs' responses biased to mostly preferred features such as longer responses which would be exacerbated in online learning scenarios as the biases can be accumulate continuously throughout the iterations.
To address these challenges, we propose a novel framework called PFP (Preference Feature Preservation).  The key idea of PFP is maintaining the distribution of human preference features throughout the online preference learning process. Specifically, PFP first trains a feature classifier using the existing offline pairwise human preference data. 
Then, using this classifier and the distribution preserving optimization, PFP maps appropriate preference features for each input instruction during online learning. 
Lastly, PFP trains LLM using the existing preference learning framework, by incorporating the preference feature of each data into system prompts and enabling LLM to explicitly handle various human preferences. Our experiments demonstrate that PFP successfully mitigates the bias in preference features that arise during online learning, and achieves superior performance compared to previous preference learning methods on general benchmarks including AlpacaEval 2.0 and MT-Bench. We also observe that PFP almost resolves a length bias issue, a long-standing problem of online preference learning, even though it was not specifically designed to tackle this.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730476907877

thread_type:
official_review

<thread_object>
creator: reviewer_vhev

<content>
summary:
This paper introduces a method to mitigate biases that arise during LM alignment -- focusing specifically on online DPO. Authors propose a new framework (PFP) to maintain the ‘distribution of human preferences’ throughout the training process, as they argue that it usually shifts during online DPO optimization.

strengths:
- Addressing biases underlying human preference data is an important problem.
- Experimental results are strong, and the approach is interpretable.

weaknesses:
1. The motivation for the proposed method is unclear, and its description often confusing. Examples:
  - On what basis can we assume to distill the full complexity of human preferences down to discrete unsupervised feature dimensions?
  - What is the motivation behind the process of distribution preservation (paragraph starting L254)?
  - Paragraph starting L173: i am very familiar with this literature but find this explanation confusing. Missing related works here: mention SLiC-HF (Zhao et al., 2023) and online DPO (Calandriello et al., 2023).
  - L307: how can you change the system prompt to be only one of $(s_1, s_2)$? Surely the response which does not correspond to the chosen system prompt is going to be worse?
 - Importantly, the method proposed involves many modeling choices that are not properly ablated, which makes it tricky to know whether all are needed. Some of the following modeling steps are ablated but most are not thoroughly evaluated:
   - Quality of the feature classifier
   - Performance of the distribution matching step
   - Synthesizing system prompt from preference features (see positioning wrt related works below)
   - Double system prompt sampling (see Q above)
   - Curriculum learning via temperature scheduling
  - examples of grammatical issues/ typos:
    - L147: beginning of the sentence
    - L246
    - L248: 'auxiliarly'

2. The experimental setup is weak.
- Weak positioning wrt. rest of the literature. For example, why not compare to prompt optimization strategies such as OPRO (Yang et al., 2023) and RLCD (Yang et al 2024)?
- Since the motivation is to avoid biases, this method requires rigorous evaluation on a task with specific biases... Here we only measure for length bias. Could authors not consider an experimental setting with a clearer bias? Does your approach then accentuate biases present in the preference data?

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605075262

thread_type:
official_review

<thread_object>
creator: reviewer_sywp

<content>
summary:
The paper presents a novel framework, PFP (Preference Feature Preservation), which addresses the issue of bias in large language models (LLMs) during online preference learning. The core content revolves around the innovative approach of maintaining the distribution of human preference features throughout the online learning process. This is achieved by training a feature classifier on existing offline pairwise human preference data, mapping appropriate preference features for each input instruction during online learning, and incorporating these features into system prompts for LLM training. The experiments indicate that PFP successfully mitigates bias in preference features and outperforms previous methods on general benchmarks like AlpacaEval 2.0 and MT-Bench, also nearly resolving the length bias issue.

strengths:
1. Originality: The introduction of the PFP framework is a novel approach to addressing bias in online preference learning for large language models (LLMs). This approach to nearly resolving the length bias issue, which has been a long-standing problem in online preference learning.
2. Rigorous Experimental Design: The paper presents a well-structured set of experiments that validate the effectiveness of the PFP framework. The use of established benchmarks like AlpacaEval 2.0 and MT-Bench adds to the credibility of the results.
3. Significance: The paper's contribution to reducing bias in AI systems is significant. By addressing bias in LLMs, the research has implications for the ethical deployment of AI, which is a critical concern in the field.

weaknesses:
1. Diversity of tasks: The paper primarily uses AlpacaEval 2.0 and MT-Bench for evaluation. While these are established benchmarks, the use of additional or more diverse datasets could strengthen the claims of the framework's effectiveness. For example, the preference features of math or coding tasks may be different, the author should give more insights on various tasks.
2. Comparative Analysis with State-of-the-Art Methods: The paper compares PFP with SFT, DPO and Iterative DPO but does not include a comparison with the latest state-of-the-art methods in bias mitigation for LLMs. Including comparisons with cutting-edge preference learning methods would provide a clearer picture of PFP's performance relative to the most advanced techniques.

questions:
The paper assumes a predefined set of preference features based on certain definitions and classifications. These assumptions may not cover the full spectrum of human preferences, and the preference features of math or coding tasks may be different. How to choose the most proper preference features for different tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662923596

thread_type:
official_review

<thread_object>
creator: reviewer_pyld

<content>
summary:
The paper proposes a novel framework, PFP (Preference Feature Preservation), aimed at mitigating biases that arise in the preference learning process of LLMs.

strengths:
The paper introduces a unique approach, Preference Feature Preservation (PFP), for managing bias in preference learning. By explicitly incorporating preference features in the system prompts and maintaining feature distribution, it provides a fresh angle on bias mitigation that has not been explored in existing work.

weaknesses:
The paper introduces a set of predefined preference features, categorizing them into five distinct classes, which provides a structured framework for evaluating human preferences in various dimensions. However, in the main results, the experiments appear to primarily focus on addressing the length bias issue, leaving it unclear whether similar attention was given to the other identified preference classes. Were any experiments conducted to examine these additional preference aspects?

Furthermore, the evaluation results, including those from AlpacaEval and MT-Bench, rely on GPT-4 as the evaluator. This raises a question about the potential variability in the results due to using GPT-4 for evaluation, particularly since its responses can introduce variance. For the results shown in the paper, i didn't see the significant improvement compared with other baselines, only 1%~2% difference for AlpacaEval, pretty close for MT-Bench.

questions:
Please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688234202

thread_type:
official_review

<thread_object>
creator: reviewer_639n

<content>
summary:
This paper tackles the bias problem as the online preference alignment takes place. The paper is motivated by the bias towards lengthy responses and proposes a system-prompt engineering-based solution. Since system prompts can be combinatorial explosive space, they only try to learn certain attributes (tone, informativeness, etc) and subattributes within them. This makes every instruction and response pair a function of 5 sub-attributes in total. Now, given the input and attribute data, they learn a mapping, which predicts the attributes that can lead to favoring certain responses. 

Having learned this distribution, to create a new dataset for alignment, they first calibrate it with the prior distribution (average distribution over these attributes) for the given training data. Followed by calibration, they generate the new system prompts and then align the model. 

Several ablation studies are conducted to show the efficacy of the proposed method.

strengths:
I like the number of ablations that are being performed, and the comparison with other length-controlled generation-based baselines. The experimental setup seems to be complete.

weaknesses:
Following are my questions that could improve this paper. 

- Notations in this paper can be heavily improved, in particular for the FE part. I think one should use vector notation for the label space, and simplex to denote output of the FE network. 
- I am still not fully sure about the motivation of this work. Can authors highlight cases where certain reward models prefer lengthy responses despite having incorrect answers? In my belief, as long as the answer is correct, and if the system prompt doesn't have instruction to be precise (succinct), then there is nothing wrong in long generations (other than the computation aspect)
- How is length-controlled generation done? 
- Why should SELFEE work? To my understanding, it is some sort of pseudo-labeling based on its current state. If the model is not good enough to begin with, then wouldn't it exacerbate the biases (or if it is incorrect for certain prompts)?
- I am not sure why the output adjustment is needed. Can authors perform a study showing its use? 
- In Fig 4, why is it bad that the distribution for the preference feature is changing (that is KL divergence increasing). I am not fully sure why should that be linked with longer-generation
- Can authors run an experiment that just adds "being concise" in the system prompt, as another baseline?

questions:
Refer to the weakness.

Post rebuttal: I’m increasing my scores.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732535885265

thread_type:
official_comment

<thread_object>
creator: area_chair_t4n1

<content>
comment:
Dear Reviewers,

This is a kind reminder that the dicussion phase will be ending soon on November 26th. Please read the author responses and engage in a constructive discussion with the authors.

Thank you for your time and cooperation.

Best,

Area Chair
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an online preference learning framework named Preference Feature Preservation (PFP). The PFP framework introduces fine-grained preference features and keeps the distribution of preference features consistent across multiple iterations to avoid the potential bias in the model. Experiments show that PFP can mitigate the bias in online preference learning to some extent.



However, the reviewers have pointed out some important weaknesses.

- First, the experiment settings (including tasks, datasets, benchmarks, and evaluation metrics) need significant improvements to show the effectiveness of PFP in reducing various biases. Except for the length bias evaluated using LC win rate and the length measurement, other types of bias need more rigorous and convincing evaluation metrics, not just the KL divergence between feature distributions.



- Second, the authors may want to conduct more comprehensive, rigorous, and adequate ablation studies than those in the current submission to provide insights into the various components of PFP, such as the feature classifier, the distribution matching step, and the sampling of double system prompts.



- Third, the comparisons between PFP and some related approaches (e.g., OPRO [1] and RLCD [2]) are missing. Although the authors claim that these methods are irrelevant to this paper, they do share some similarities with PFP, as they also discuss how system prompts affect the model's responses.





Therefore, I recommend the next version of this paper to incorporate more experiments and insights.



[1] Large Language Models as Optimizers.

[2] RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment.