Preference Data Annotation with Guided Density Ratios

Authors: Guangxuan Xu, Kai Xu, Shivchander Sudalairaj, Akash Srivastava
Primary area: foundation or frontier models, including LLMs
Keywords: RLHF, preference training, reward model, human preference, preference alignment

Abstract:
Preference tuning of large language models (LLMs) relies on high-quality human preference data, which is often expensive and time-consuming to gather. While existing methods can use trained reward models or proprietary model as judges for preference annotation, they have notable drawbacks: training reward models remain dependent on initial human data, and using proprietary model imposes license restrictions that inhibits commercial usage.  In this paper, we introduce Guided Density Ratio, a training-free and highly effective method that leverages off-the-shelf LLMs for preference data annotation. Our approach uses the log-density ratio between a better-aligned LLM and a less aligned LLM as a reward signal. We explores 221 different LLMs pairs and empirically demonstrate that increasing the performance gap between paired LLMs correlates with better reward generalization. Furthermore, we show that tailoring the density ratio reward function with specific criteria and preference exemplars enhances performance across domains and within target areas.

In our experiment using density ratio from a pair of Mistral-7B models, Guided Density Ratio achieves a RewardBench score of 82.6, outperforming the best trained reward functions from same model class and demonstrating competitive performance against SoTA models in Safety (91.0) and Reasoning (88.0) domains. We use Guided Density Ratio to annotate an on-policy preference dataset with which we preference tune \textit{Llama-3-8B-Instruct} with SimPO. Using reward signals from two relatively weak models, our approach pushes Llama-3-8B to achieve a 37.4\% ($+$15.1\%) win rate on ArenaHard and a 40.7\% ($+$17.8\%) win rate on Length-Controlled AlpacaEval 2.0, along with a score of 8.0 on MT-Bench.

Decision: reject
Reviewer scores: 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730630117574

thread_type:
official_review

<thread_object>
creator: reviewer_ojiv

<content>
summary:
The paper investigates the preference tunning in LLM and proposes the challenge that it depends on paired human feedback data and the collection is costly in time and resources. Then the authors propose a novel data annotation technique that improves the density ratio rewards annotation accuracy and enhances the effectiveness of preference tuning using a new annotated dataset.

strengths:
+ The paper explores the challenge of data scarcity in preference tuning, a significant question in the LLM era.

+ The author proposes a novel data annotation technique based on the prompt density ratio, which has advanced community skills and tunning-free advantages.

+ Extensive experiments include multiple settings on the cutting-edged open-source and close-resource MLLM to improve the solidity of the paper.

weaknesses:
- While the proposed prompt-based density ratio method exhibits some improvements, the design of prompt templates for system prompts and few-shot prompts is widely used in broad scenarios with less innovation.

- Prompt-density ratio compared to vanilla-density ratio decreases the model’s bias, but if the model itself with some bias, it will accumulate to the updated modal, it should expand some discussion about the difference of bias between source and target model.

- The experiment section should expand a paragraph to summarise the setting and the experiment description is unclear. While I was confused by some of the questions listed below. 

What is the generative reward in Table 1? 

Why the second to last only include safety and reasoning? 

Why are the results of NousResearch/Nous-Hermes-2-Mistral-7B-DPO not included in the Table, which is also an instruction-tuned model? 

Why is only one setting included with a base or without a base that corresponds to two different weak models?

Does the experiment include a domain-specific prompt that should also include the comparison?

- There is a lack of ablation studies on the functionality of system prompts and few-shot prompts.

- Some sentences contain long-tail expressions, and several equations need improvement, particularly by adding appropriate punctuation marks.

questions:
- How to ensure the annotate quality in the prompt routing for category-specific prompts and preference tunning data creation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691078172

thread_type:
official_review

<thread_object>
creator: reviewer_kbet

<content>
summary:
The paper proposes a scalable, training-free method for preference data annotation in large language models by using guided density ratios between pre-trained models. This approach, which employs domain-specific prompts and few-shot examples, enhances alignment in areas like safety and reasoning without requiring costly human feedback. The method achieves competitive results on benchmarks such as RewardBench and MT-Bench, demonstrating its potential as an effective and resource-efficient alternative for preference tuning in LLMs.

strengths:
1. The results on RewardBench and other benchmarks demonstrate that the proposed method can compete with or even outperform larger, more resource-intensive models, making it a cost-effective solution for preference alignment.

2. The paper includes comprehensive experiments across multiple domains and benchmarks, providing robust evidence for the effectiveness of the method.

3. The paper introduces a scalable, training-free preference annotation method using density ratios, a concept that could significantly reduce the cost and time typically required for human feedback data collection.

weaknesses:
1. While prompt-based guidance improves performance, designing effective prompts and in-context examples for different domains could be labor-intensive and may not generalize well across all tasks.

2. I think evaluation across different model size should be included. For example, including 7B, 13B, and 70B.

questions:
1. Could the density ratio approach be adapted to work with multi-modal data, like vision language models / LLaVA, or is it inherently limited to language models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731217717987

thread_type:
official_review

<thread_object>
creator: reviewer_qnfe

<content>
summary:
The paper interprets the use of a strong v/s weak LLM as a human preference proxy. However, their main contribution is designing prompt templates with relevant domain specific exemplars (IC examples) and domain specific system prompts - to show that the resulting density ratio is more helpful.

strengths:
The paper does a good job in explaining their contribution, and the examples are helpful. The direction of LLM as a preference / human proxy is topical and being investigated by various groups - which makes it aligned to the venue.

weaknesses:
I find the work to be an incremental advancement over previous discussions on using LLMs as a reward proxy. The key contribution of the work is the manually designed prompt templates, T(x), which are composed of system and in-context examples - both of which are domain dependent. The work's key argument is to favor a domain specific prompt design that influences density ratio - with limited discussions on why this template must work. 

Few questions to the authors : 
1. What is the cost of designing these templates? How can one extend this work to an arbitrary domain where the preferences may not be "safety". 
2. In line 048, authors say that human preferences can be complicated. How are the preferences anymore complicated or undefined when they they pen down the preferences as part of their system prompt. Can the authors point out to examples where these prompt templates helped with other user preferences not explicitly mentioned in the prompt. 
3. Can the authors compare the cost of prompt design with collecting more feedback? 
4. What are the guarantees that this prompt template won't work adversarially to the model. For instance, its possible that the system prompt conflicts with some preferences specified in the query. What are some alternatives to the system prompt that the authors considered? The authors provide some discussion in A.1, however, the following questions remain unanswered :  how did the authors generate SYSTEM1, 2... versions; what are the domain specific principles and how were those generated?; what other principles did the authors experiment with? Which principles, while "plausible" and "aligned" with typical human preferences did not yield expected results. 
5. Many a times IC examples have hampered model's performance. The authors discuss ICL example selection strategy in A.1, but the discussion seems to to justify the use of IC examples in their setup anecdotally. 
6. The argument for why a baseline calibration is needed (lines 205 - 211) seem anecdotal. Can the authors provide stronger theoretical / empirical result on what are the properties of this weaker model for such a calibration, and what must the relationship between weaker and stronger model be in the scope of their work.

questions:
Please see Weaknesses section. 

I believe that addition of answers to the above questions can help with a stronger submission.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732057020826

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to ALL reviewers on Major Updates (Part-1)

comment:
We thank all reviewers for their time and valuable feedback. We decided to update our draft with experiments added to address some common concerns of reviewers.

## Improved presentation and additional empirical study for Contribution #1 (model choices)

While all reviewers have pointed out our Contribution #2 (prompting), our Contribution #1 on strong-over-weak model choices with off-the-shelf models in density ratio reward seems to be not well-recognized. 

#### **1. Consolidation of Contribution #1 as the *Strong-over-Weak Hypothesis***
- In response to Reviewer QnfE’s suggestion and to clarify the significance of Contribution #1, we acted on the advice to update our presentation and explicitly stated and framed it as the *Strong-over-Weak Hypothesis*:  
  > "The density ratio reward requires an alignment gap between a strong and weak model."  
  This hypothesis challenges the conventional understanding that density ratio rewards are solely the result of optimizing the DPO objective. Instead, we assert that the alignment gap is a critical determinant of reward performance, providing a new perspective on how density ratio signals are generated and utilized.

#### **2. Empirical Validation Supporting the Strong-over-Weak Hypothesis**
- To substantiate this hypothesis, we conducted controlled experiments, as shown in updated **Figure 1**, using 221 distinct model pairs trained with diverse objectives, including SFT, DPO, PPO, KTO, RRHF, ORPO, SimPO, IPO, and SLiC-HF.  
  - **Findings**:  
    The results reveal a strong correlation between the reward performance and the *Model Alignment Gap* (ArenaHard diff.) between the strong and weak models. This empirical observation underpins the *Strong-over-Weak Hypothesis*.  
  - We further demonstrated this phenomenon in **Figure 2** via a case study involving iterative DPO checkpoints. Key insights include:  
    - **Red-line cells**: When the starting reference model is already well-aligned, the DPO objective struggles to learn a generalizable implicit reward.  
    - **Green-line cells**: Conversely, selecting a weaker aligned model (e.g., the Base model) as the denominator in the density ratio consistently yields the highest reward scores. This practical guidance aligns with the *Strong-over-Weak Hypothesis*.  

#### **3. Practical Implications of Contribution #1**
- Contribution #1 offers actionable insights for identifying successful model pairs when constructing density ratios, providing clear, experimentally validated criteria.  
- Our thesis diverges from conventional beliefs that attribute density ratio rewards exclusively to DPO training. Instead, we demonstrate that these rewards are deeply influenced by the relative alignment between the models involved.  
- This understanding also leads to novel practical benefits:  
  - We introduce *prompt guidance* to exert control over the otherwise under-specified density ratio signal.  
  - This stands in stark contrast to the implicit DPO thesis, where prompting an already optimal reward model would be counter-intuitive.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732058272451

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to ALL reviewers on Major Updates (Part 2)

comment:
## Ablation on Prompt Design and ICL Examples

We appreciate the reviewers’ comments regarding the labor cost involved in obtaining the current prompts and in-context learning (ICL) examples. This is a great question, and we updated our paper to highlight our finding that **density ratio** approach can benefit significantly from **domain-specific instructions**, even with limited effort in prompt engineering.

In response to reviewers KBeT, QnFE, and ojiV’s uniform suggestions, we conducted an ablation study on both the **rules and principles** (Appendix C) and the **ICL examples** used (Appendix C.1). Below, we summarize our findings.

---
### Key Takeaways

1. **Simple instructions** already bring noticeable improvements in reward performance.
2. Various **principles** we tested provide comparable performance and are straightforward to implement.
3. The **ICL examples** we used are not optimized yet still yield significant performance gains, leaving room for further improvements with more advanced techniques.

---

### Ablation on Rules and Principles (Section C)

Using the simple prompt, `"You are a helpful AI assistant."`, we observe a **2.9-point improvement** in the RewardBench score compared to the baseline without any prompt. Interestingly, the largest gains occur in the **Reasoning** domain, which includes tasks such as coding and math.

To better understand the impact of safety principles in prompt design, we conducted an iterative ablation study:

- **safe1**: Includes the first safety guideline.
- **safe2**: Adds a second guideline.
- **safe3**: Adds a third guideline.
- **safe4**: Incorporates all four guidelines.
- **safe5**: Adds a fifth guideline.

#### Observations
1. Adding the first few guidelines (safe1–safe3) consistently improves **Safety** scores.
2. Including the fourth guideline (**safe4**) results in diminishing returns and slight regressions in domains like **Reasoning**.
3. Adding a fifth guideline (**safe5**) leads to performance degradation, indicating that overloading prompts with rules can reduce effectiveness.

**Final Selection**: We chose **safe4** for its balance of comprehensive safety coverage and cross-domain performance. However, leaner prompts like **safe2** or **safe3** deliver comparable results for safety-focused tasks.

| **Prompt**           | **Chat** | **ChatHard** | **Safety** | **Reasoning** | **Overall** |
|-----------------------|----------|--------------|------------|---------------|-------------|
| -                     | 92.2     | 60.5         | 82.4       | 73.8          | 77.2        |
| simple                | 91.1     | 60.8         | 83.5       | **87.8**      | 80.1        |
| safe1                 | 93.8     | 56.8         | 83.9       | 81.2          | 79.0        |
| safe2                 | **94.7** | 57.7         | 89.3       | 82.6          | 81.1        |
| safe3                 | 93.0     | 60.1         | 90.2       | 82.4          | 81.7        |
| safe4                 | 91.1     | 59.2         | **91.6**   | 77.6          | 79.9        |
| safe5                 | 89.4     | 55.9         | 87.8       | 74.9          | 77.0        |

---
</content>
</thread_object>

</thread>
<thread>
creation date:
1732058306511

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to ALL reviewers on Major Updates (Part 3)

comment:
### Ablation on the Effect of ICL Examples (Section C.1)

We designed ICL examples grouped by their primary domain (e.g., ChatHard, Safety, Math/Coding/Reasoning). While many examples overlap domains, each was classified by its primary focus for simplicity.

#### Observations
1. **Performance Increases**: All ICL examples improve density ratio reward performance on RewardBench.
2. **Minimal Differences**: Variations in performance between examples are minor and could result from noise or overfitting to the evaluation set of 2,850 examples.
3. **Simplicity**: Our ICL examples, written based on common design conventions, are unoptimized. This highlights the potential for further gains with advanced techniques.

**Summary of Results**:

| **ICL Example**        | **Chat** | **ChatHard** | **Safety** | **Reasoning** | **Overall** |
|-------------------------|----------|--------------|------------|---------------|-------------|
| -                       | 92.2     | 60.5         | 82.4       | 73.8          | 77.2        |
| **sys+ChatHard**        |          |              |            |               |             |
| ChatHard1               | 91.1     | 69.1         | 88.0       | 85.9          | 83.5        |
| ChatHard2               | 93.0     | 63.6         | 88.7       | 88.2          | 83.4        |
| ChatHard3               | 88.8     | 69.3         | 88.7       | 87.2          | 83.5        |
| ChatHard4               | 89.9     | 66.0         | 91.9       | 86.6          | 83.6        |
| ChatHard5               | 90.5     | 63.8         | 91.7       | 89.7          | 83.9        |
| ChatHard6               | 94.7     | 59.9         | 89.2       | 89.3          | 83.4        |
| **sys+Safety**          |          |              |            |               |             |
| Safe1                   | 88.3     | 61.8         | 91.0       | 87.9          | 82.3        |
| Safe2                   | 90.8     | 64.3         | 89.7       | 86.2          | 82.8        |
| Safe3                   | 91.3     | 60.1         | 91.1       | 87.8          | 82.6        |
| **sys+Reasoning**       |          |              |            |               |             |
| Reasoning1              | 91.9     | 59.9         | 90.1       | 88.7          | 82.7        |
| Reasoning2              | 91.9     | 61.2         | 88.2       | 87.0          | 82.1        |
| Reasoning3              | 90.2     | 64.3         | 90.0       | 85.8          | 82.6        |
| Reasoning4              | 90.5     | 61.8         | 89.5       | 88.7          | 82.6        |
| Reasoning5              | 93.6     | 61.6         | 88.7       | 87.1          | 82.8        |
| Reasoning6              | 91.6     | 58.8         | 88.8       | 87.5          | 81.7        |
| Reasoning7              | 88.3     | 60.1         | 89.9       | 87.0          | 81.8        |
| Reasoning8              | 91.6     | 61.0         | 89.9       | 89.7          | 83.1        |

---

### Conclusion
We hope above ablations can address reviewer concern on the labor cost of prompt design.



### Summary of Revisions to Draft

1. **Abstract and Introduction**:  
   - Updated to frame and clearly present the **Strong-over-Weak Hypothesis**, providing a compelling rationale for our approach.

2. **Method Section**:  
   - Revised to emphasize the **Strong-over-Weak reward function design**, detailing its role as the core innovation of our methodology.

3. **Experiment Section**:  
   - Enhanced to validate the **Strong-over-Weak Hypothesis** with supporting empirical evidence.  
   - Clarified the experiment design to ensure transparency and reproducibility of results.

4. **Appendix**:  
   - Expanded to include an **Ablation on Prompt Design**, offering deeper insights into the effects of various prompts.  
   - Added more detailed examples of the prompts used in experiments to aid understanding and replication.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission addresses the problem of preference tunning in large language models, by introducing a novel data annotation technique based on guided density ratios between pre-trained models. The proposed method achieves competitive performance on RewardBench and MT-Bench. The submission received mixed ratings after rebuttal, including one borderline accept (6), one borderline reject (5), and one reject (3). Reviewer QnfE who gave reject did not engage in the post-rebuttal discussion. Below I summarize the main strengths and limitations of the submission, according to the reviews (after rebuttal discussion) and my own reading of the submission:

*Strengths:*
- The proposed method is scalable and cost efficient, and is of practical importance.
- The submission offers extensive empirical evaluations, demonstrating the effectiveness of the proposed approach.

*Weaknesses:*
- The proposed method employs domain-specific prompts, which could be labor-intensive and may not generalize well (QnfE, kBeT, ojiV).
- Lack of clarity on the submission's claimed contributions (QnfE), and that the proposed prompt template design has been explored (ojiV).

After the rebuttal, the AC shares QnfE and kBeT's remaining concern on the practicality of the prompt design used by the proposed approach, and ojiV's concern on the clarity and novelty of the proposed approach. There was no reviewer championing for the acceptance of the submission. The AC thus believes that the submission is not ready to be accepted.