Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts

## Abstract

Abstract Preference optimization is a standard approach to fine-tuning large language models to align with human preferences.
The quality, diversity, and quantity of the preference dataset are critical to the effectiveness of preference optimization.
However, obtaining a large amount of high-quality and diverse preference annotations is difficult in many applications.
This raises the question of how to use the limited annotation budget to create an effective preference dataset.
To this end, we propose Annotation-Efficient Preference Optimization (AEPO).
Instead of exhaustively annotating preference over all available response texts, AEPO selects a subset of responses that maximizes quality and diversity from the available responses, and then annotates preference over the selected ones.
In this way, AEPO focuses the annotation budget on labeling preference over a smaller subset of responses with diversity and of high quality.
We evaluate the performance of Direct Preference Optimization (DPO) using AEPO and show that it outperforms models trained using a standard DPO with the same annotation budget.
Our code is available at https://github.com/CyberAgentAILab/annotation-efficient-po .

## 1 Introduction

Large Language Models (LLMs) trained on massive datasets are capable of solving a variety of tasks in natural language understanding and generation [, , , ].
However, they have been shown to generate texts containing toxic, untruthful, biased, and harmful outputs [, , , , , ].
Language model alignment aims to address these issues by guiding LLMs to generate responses that aligns with human preferences, steering them to generate responses that are informative, harmless, and helpful [, , , ].

The most common strategies to align an LLM are Reinforcement learning from human feedback (RLHF) and Direct Preference Optimization (DPO) [, , ].
RLHF and DPO use the human preference dataset to train a reward model or a language model directly.
The performance of these algorithms is highly dependent on the choice of the preference dataset.
However, building a human preference dataset requires human annotations, which are expensive to collect.
Thus, the main bottleneck in building a preference dataset is the annotation cost.

A large number of works have investigated the synthesis of preference data using a powerful LLM (e.g., GPT-4) to distill the knowledge of human preferences [, , , , , , , ].
However, human preferences are known to be diverse and pluralistic, and they are unlikely to be represented by the opinion of a single model [, , , , , , , , , , , ].
Several papers have pointed out that LLMs may exhibit bias toward aligning with people from a particular background [, , ].
For example, Cao et al. [] reports that ChatGPT has a strong alignment with American culture, but adapts less effectively to other cultural contexts.
In addition to cultural biases, previous work suggests that even a highly capable model (e.g., GPT-4) still has biases such as length bias [, ], style bias [], and positional bias [].
Thus, human annotation is desirable to align and personalize an LLM with diverse and unbiased human preferences [, , ].
The efficiency of annotation is critical to making LLMs accessible and useful to people from diverse backgrounds, who may have only a small amount of preference feedback data to work with.

The question is how to generate an effective preference dataset with a limited annotation budget.
Previous work has shown that the following three features are desirable for a preference dataset to be effective [, ]:

- 1.
Quantity and Diversity of instructions. Quantity and diversity of the instructions are desirable [, , , , , , , , , ].
- 2.
Quality of responses. Higher quality responses is desirable [, , , , ].
- 3.
Diversity of responses. A set of responses with higher diversity is desirable [, , , ].

To achieve all three desiderata with a limited annotation budget, it is desirable to annotate preference over diverse and high quality responses with a minimum amount of annotation required per instruction.

Figure: Figure 1: Annotation-Efficient Preference Optimization (AEPO) is a process for generating a preference dataset with diverse and high-quality responses with fewer annotations. See Section 3 for details. Here we set $k=2$ and select two responses from the generated responses to annotate.

To this end, we propose Annotation-Efficient Preference Optimization (AEPO), a preference optimization with a preprocessing step on the preference dataset to reduce the required amount of annotation (Figure 1).
Instead of annotating the preference over all $N$ responses, AEPO selects $k(<N)$ responses from $N$ responses. We deploy a sophisticated method to select a set of response texts with high diversity and quality.
It then annotates the preference for the selected $k$ responses. In this way, AEPO uses all $N$ samples to select a subset of responses with high diversity and quality, while requiring only an annotation over a subset of responses.

The strength of AEPO is threefold (Table 1).
First, it is applicable to human feedback data. Compared to Reinforcement Learning from AI Feedback (RLAIF) [], our approach can be applied to both human and AI feedback. RLAIF is a scalable approach in terms of both instructions and annotations, but it is known that the feedback from existing language models is biased in various ways [, , , , ].
Second, it is scalable with additional computational resources. By generating larger amount of responses, AEPO can find more diverse and higher quality responses to annotate, resulting in more effective preference dataset with a fixed amount of annotation (Figure 2).
Third, it requires less annotation to generate an effective preference dataset. Unlike an exhaustive annotation strategy which requires a large annotation effort (e.g., West-of-N strategy [, , ]), AEPO can reduce the annotation cost through the subsampling process.

We evaluate the performance of DPO using AEPO on the AlpacaFarm, Anthropic’s Helpfulness, and Anthropic’s Harmlessness datasets [, ].
With a fixed annotation budget, we observe that the performance of vanilla DPO degrades as the number of responses per instruction increases above a certain threshold.
In contrast, AEPO improves with a larger number of responses and outperforms vanilla DPO on AlpacaFarm and Anthropic’s Harmlessness dataset.
Our ablation study on AlpacaFarm shows that AEPO achieves higher performance with varying LoRA hyperparameters and loss functions (Appendix B).
The result shows that AEPO can scale the performance of preference learning with a larger number of computations, even under a fixed number of annotations.

AEPO improves the diversity and the quality of the responses of the preference dataset (Figure 3).
The experimental result shows that both diversity and quality are important for the responses to be effective for preference optimization.

**Table 1: Comparison of annotation strategies for preference dataset.**
| Preference dataset | Human feedback | Scalable | Annotation-efficient |
| --- | --- | --- | --- |
| Human feedback | ✓ | ✗ | ✗ |
| RLAIF [] | ✗ | ✓ | ✓ |
| West-of-N [] | ✓ | ✓ | ✗ |
| AEPO | ✓ | ✓ | ✓ |

## 2 Background

In this section, we describe the RLHF and DPO and review how the preference datasets are constructed for these algorithms.

### 2.1 Preference Optimization

Let $\mathcal{D}_{p}$ be a pairwise preference dataset $\mathcal{D}_{p}=\{(x,y_{c},y_{r})\}$, where $x$ is an instruction, $y_{c}$ is the chosen response, and $y_{r}$ is the rejected response, that is, $y_{c}$ is preferred to $y_{r}$.
Reinforcement Learning from Human Feedback (RLHF) uses a reward function $R(x,y)$ learned from $\mathcal{D}_{p}$ to train an SFT model $\pi_{\mathrm{ref}}$ [, , , ]. Typically, the RL process is formulated as the following optimization problem:

$$ $\pi_{\mathrm{RLHF}}=\operatorname*{arg\,max}_{\pi}\mathop{\mathbb{E}}_{x\sim \mathcal{D}_{p}}\mathop{\mathbb{E}}_{y\sim\pi(y|x)}[R(x,y)]-\beta\mathbb{D}_{ \mathrm{KL}}[\pi(\cdot|x)||\pi_{\mathrm{ref}}(\cdot|x)],$ (1) $$

where $\beta$ is a hyperparameter that controls the proximity to the SFT model $\pi_{\mathrm{ref}}$.
Since the objective is not differentiable, reinforcement learning algorithms are used for optimization.

Direct Preference Optimization (DPO) trains the language model to directly align with the human preference data over the responses without using reward models [].
Although DPO is based on supervised learning rather than reinforcement learning, it uses essentially the same loss function as RLHF under the Bradley-Terry model [].
The objective function of the DPO is the following:

$$ $\pi_{\mathrm{DPO}}=\operatorname*{arg\,max}_{\pi}\mathop{\mathbb{E}}_{(x,y_{c} ,y_{r})\sim\mathcal{D}_{p}}[\log\sigma(\beta\log\frac{\pi(y_{c}|x)}{\pi_{ \mathrm{ref}}(y_{c}|x)}-\beta\log\frac{\pi(y_{r}|x)}{\pi_{\mathrm{ref}}(y_{r}| x)})],$ (2) $$

where $\sigma$ is the sigmoid function.

### 2.2 Preference Dataset

The performance of preference optimization largely depends on the choice of the preference dataset $\mathcal{D}{}_{p}$.
Existing approaches explore the use of high-performance models (e.g., GPT-4) to synthesize high-quality instructions, responses, and preference feedback [, , , , , ].
It is also shown to be useful to use the training model itself to synthesize a preference dataset [, , ].

Several papers have investigated annotation-efficient learning by reducing the number of instructions rather than synthesizing more [, ].
Su et al. [] suggested selecting examples to annotate from a pool of unlabeled data to improve the efficiency of in-context learning.
Zhou et al. [] shows that fine-tuning a model with carefully selected and authored instructions can improve performance.
Chen et al. [] points out that public instruction datasets contain many low-quality instances and proposes a method to filter out low-quality data, resulting in more efficient fine-tuning.

Regarding the selection of the response texts, several works have proposed to use the West-of-N (WoN) strategy [, , ]. The WoN strategy randomly samples $N$ responses ${\{y_{i}\}_{i=1}^{N}}$ for each instruction $x$. Then, it annotates the preference scores *over all $N$ responses*. The response with the highest preference is labeled as chosen (win) $y_{c}$ and the one with the lowest preference is labeled as rejected (lose) $y_{r}$:

$$ $y_{c}\leftarrow\operatorname*{arg\,max}_{y\in\{y_{i}\}_{i=1}^{N}}R(x,y),\quad y _{r}\leftarrow\operatorname*{arg\,min}_{y\in\{y_{i}\}_{i=1}^{N}}R(x,y).$ (3) $$

The strategy is shown to be more efficient than random sampling with the same number of instructions.
However, it requires $N$ annotations per instruction to run, making it inapplicable when the annotation budget is limited.

## 3 Annotation-Efficient Preference Optimization (AEPO)

We propose Annotation-Efficient Preference Optimization (AEPO), a method for efficiently learning preferences from a large number of responses *with a limited budget on preference annotations* (Figure 1).
The procedure of AEPO is described in Algorithm 1.
We assume that a set of $N$ responses is available for each instruction: $\mathcal{D}{}=\{(x,\{y_{i}\}_{i=1}^{N})\}$. Instead of annotating the preference over all responses in $\{y_{i}\}_{i=1}^{N}$, AEPO subsamples $k$ responses (e.g., $k=2$) with high diversity and quality (line 3).
Then, it deploys the WoN strategy (Eq. 3) on the subsampled subset of responses $Y^{*}$ instead of all $N$ responses $\{y_{i}\}_{i=1}^{N}$. It annotates the preference over $Y^{*}$ to select the best and the worst responses as the chosen and the rejected responses, respectively (lines 4, 5).
In this way, we can allocate the annotation budget only to labeling diverse and high-quality responses. AEPO achieves to build a preference dataset with diverse and high-quality responses using small amount of annotation effort, which is exactly the characteristics desired for preference annotation methodology we discussed in Section 1.

Figure: Algorithm 1 Annotation-Efficient Preference Optimization (AEPO)

The performance of AEPO is highly dependent on how we subsample a subset $Y$ from the candidate set of responses $\{y_{i}\}_{i=1}^{N}$.
We choose the subset $Y$ that maximizes the Diverse Minimum Bayes Risk (DMBR) objective []:

$$ $Y^{*}\leftarrow\operatorname*{arg\,max}_{\stackrel{{\scriptstyle Y\subseteq\{y _{i}\}_{i=1}^{N}}}{{|Y|=k}}}\sum_{y\in Y}\left(\frac{1}{N}\sum_{y^{\prime}\in \{y_{i}\}_{i=1}^{N}\setminus\{y\}}u(y,y^{\prime})\right)+\lambda d(Y),$ (4) $$

where $u(y,y^{\prime})$ is a similarity between the two sentences.

DMBR is the state-of-the-art algorithm for generating a set of diverse and high-quality outputs []. It is shown to be effective in various directed text generation tasks including machine translation, text summarization, and image captioning.
Using state-of-the-art methodology, AEPO can effectively select diverse and quality responses from the candidates.

The DMBR objective consists of two objectives. The first term $\sum_{y\in Y}\left(\frac{1}{N}\sum_{y^{\prime}\in\{y_{i}\}_{i=1}^{N}\setminus
\{y\}}u(y,y^{\prime})\right)$ is the Minimum Bayes Risk (MBR) objective, which corresponds to the quality objective [, ]. Many works have shown that by maximizing the MBR objective, we can generate high quality texts in many directed text generation tasks [, , , , , ]. Therefore, by maximizing this objective, we expect to find high quality responses.
The latter term $\lambda d(Y)$ is the diversity objective, with $\lambda$ controlling the trade-off between quality and diversity.
We follow Jinnai et al. [] and use the following pairwise distance as the diversity metric $d(Y)$:

$$ $d(Y)=-\sum_{y\in Y}\sum_{y^{\prime}\in Y\setminus\{y\}}\frac{1}{|Y|}u(y,y^{ \prime}).$ (5) $$

where $u$ is the same similarity function as the one used in the MBR objective.

We use the cosine similarity of the embedding by the all-mpnet-base-v2 sentence BERT model as the similarity function $u$ of DMBR, as it has shown to be effective for a variety of sentence embedding tasks [, ].(^1^11[https://huggingface.co/sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2))(^2^22[https://sbert.net/docs/pretrained_models.html](https://sbert.net/docs/pretrained_models.html))

AEPO is applicable to any preference optimization method that uses a preference dataset with multiple responses per instruction. In our experiments in Section 4, we focus on DPO because of its effectiveness and efficiency in preference optimization.

## 4 Experiments

We evaluate the performance of AEPO on DPO using the AlpacaFarm dataset [] and Anthropic’s hh-rlhf datasets.(^3^33[https://huggingface.co/datasets/tatsu-lab/alpaca_farm](https://huggingface.co/datasets/tatsu-lab/alpaca_farm))(^4^44[https://huggingface.co/datasets/Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf))
We use mistral-7b-sft-beta (Mistral) [, ] as the language model to generate responses.(^5^55[https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta](https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta)) We generate up to $N=128$ responses per instruction with nucleus sampling ($p=0.9$) []. The temperature of the sampling algorithm is set to $1.0$ for all experiments.
The number of subsampled responses is set to $k=2$ and the diversity hyperparameter is set to $\lambda\in\{0.0,0.5,1.0,2.0\}$.
We evaluate DPO with the WoN strategy as a baseline.
Since it uses $N/2$ times more annotations per instruction than AEPO with $k=2$, we reduce the number of instructions for WoN to $2/N$ so that the number of required annotations is the same as for AEPO. Note that we assume that the cost of annotating the preference rank for $N$ responses is linear in $N$. This assumption favors WoN because it becomes increasingly difficult to annotate preference rank over a larger set of options []. WoN and AEPO use the same set of responses to ensure a fair comparison.
As a naive baseline with no response selection strategies, we include the results of a random sampling of $N=2$ responses using nucleus sampling with $p\in\{0.8,0.9,1.0\text{ (ancestral sampling)}\}$ [].

We use the OASST reward model to annotate the preference over the responses.(^6^66[https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2))
Although it is ideal to use human annotations to evaluate the performance of the algorithms, human annotations are expensive and difficult to reproduce. To this end, we use existing open source reward models as preference annotators for the experiment.

We train the same model that generates the responses (Mistral) using Direct Preference Optimization (DPO) with Low-Rank Adaptation (LoRA) [, ]. We set the LoRA’s $r=64$ and $\alpha=r/4$ (See Appendix B for the result with $r=32$ and $r=128$).
Other hyperparameters for the training process are described in Appendix A.
For the AlpacaFarm dataset, we use the alpaca_human_preference subset as the training set and use the alpaca_farm_evaluation subset as the evaluation set.
For the Anthropic’s hh-rlhf datasets, we use the first 5000 entries of the training set of the helpful-base and harmless-base subsets as the training set. Then we evaluate the trained model on the first 1000 entries of the test set of the helpful-base (Helpfulness) and harmless-base (Harmlessness) subsets.
For WoN, we reduce the number of instructions evenly for the two subsets so that the dataset always has the same number of instructions from the two subsets.

We evaluate the quality of the trained models by sampling a response using nucleus sampling ($p=0.7$). The model output is evaluated using Eurus-RM-7B (Eurus) and PairRM [, ].(^7^77[https://huggingface.co/openbmb/Eurus-RM-7b](https://huggingface.co/openbmb/Eurus-RM-7b))(^8^88[https://huggingface.co/llm-blender/PairRM](https://huggingface.co/llm-blender/PairRM))
Both reward models are shown to have a high correlation with human annotations in RewardBench [].

All the experiments are run using Huggingface’s Transformers library [] and Transformer Reinforcement Learning library []. DMBR is implemented using the code by Jinnai et al. [].(^9^99[https://github.com/CyberAgentAILab/diverse-mbr](https://github.com/CyberAgentAILab/diverse-mbr))

Figure: Figure 2: Evaluation of AEPO and West-of-N strategies for DPO with fixed number of preference annotations and varying number of responses per instruction in AlpacaFarm. The line represents the average reward score and the bar shows the standard deviation over three runs. With a fixed amount of annotation, the performance of West-of-N decreases with large $N$, while AEPO continues to improve with increasing $N$. Note that WoN and AEPO are equivalent to random sampling with $N=2$.

Table: Table 2: Evaluation of preference dataset annotation strategies for DPO on AlpacaFarm using Mistral. The best score is in bold, and the second best is underlined. The mean and standard deviation of three runs are reported. $|\mathcal{D}{}|$ is the total number of instructions and #Insts is the number of instructions used for training. For a fair comparison in the annotation cost, we keep the number of annotations (#Annots) at $2|\mathcal{D}{}|$. w% shows the win rate against the SFT model.

#### Main Results.

Figure 2 shows the Eurus score of the DPO models on AlpacaFarm using AEPO ($\lambda=1.0$) and WoN with different numbers of responses.
WoN with $N=4$ outperforms the random sampling baselines (i.e., WoN with $N=2$), even though it uses only half of the available instructions, which is consistent with the results of Song et al. [].
However, WoN’s score drops significantly for $N\geq 8$ as the number of instructions decreases.
In contrast, AEPO scales with the number of responses $N$ and outperforms WoN (Figure 2 and Table 2).
We observe a similar result on Anthropic’s Harmlessness dataset (Table 4). Interestingly, we observe that the WoN strategy with $N=8$ is effective for Anthropic’s Helpfulness dataset, even with a much smaller instruction set (Table 3).
We hypothesize that this may be because helpfulness is an objective to maximize, which can be learned from a small set of high-quality sentences rather than a large and diverse set of medium-quality sentences. On the other hand, harmlessness is an objective to minimize, which can be learned from a diverse set of medium-quality sentences rather than a small set of high-quality sentences.
The other possible reason is that AEPO finds more diverse and higher quality sentences than random sampling in the Harmlessness dataset (Figure 6 and 6 in Appendix E). AEPO may be effective in the Harmlessness task because the DMBR objective successfully finds an appropriate response even in an adversarial setting, improving the effectiveness of DPO training.

Table: Table 3: Evaluation of preference dataset preparation strategies for DPO on Anthropic’s Helpfulness using Mistral. The best score is in bold, and the second best is underlined. The mean and standard deviation of three runs are reported.

Table: Table 4: Evaluation of preference dataset preparation strategies for DPO on Anthropic’s Harmlessness using Mistral. The best score is in bold, and the second best is underlined. The mean and standard deviation of three runs are reported.

#### AEPO generates a diverse and high quality preference dataset.

We evaluate the diversity and quality of the preference dataset generated by AEPO with $k=2$, $N\in\{2\text{ (Random)},4,8,16,32,64,128\}$, and $\lambda\in\{0,0.3,0.5,1.0,2.0\}$. To measure the semantic and lexical diversity of the responses, we use Sentence BERT and distinct-n [].(^10^1010[https://github.com/neural-dialogue-metrics/Distinct-N](https://github.com/neural-dialogue-metrics/Distinct-N)) We use the same Sentence BERT model (all-mpnet-base-v2) as AEPO to evaluate the average cosine similarity between the selected pairs of responses. Distinct-n counts the number of distinct n-grams in a sentence divided by the length of the sentence.
The quality of the responses is measured by the average reward score of the responses.
The result is shown in Figure 3. By using a larger number of responses $N$, AEPO manages to generate more diverse and higher quality response pairs than a random sampling with the same number of annotations, measured as Sentence BERT and distinct-n. See Appendix C for examples of the preference data generated by AEPO.
We observe similar trends in the results on distinct-2 and distinct-3, as well as the results on the Anthropic’s datasets (Appendix E).

Figure: (a) Sentence BERT

Table: Table 5: Evaluation of AEPO for off-policy learning. We generate responses with Mistral and use the sampled responses to train Dolly. The best score is in bold, and the second best is underlined.

#### Both diversity and quality of the preference dataset are important for preference learning.

The question is what contributes to the improved performance of AEPO.
We evaluate AEPO with $\lambda\in\{0.0,0.5,1.0,2.0\}$ to investigate the importance of quality and diversity of responses to the performance of the fine-tuned models.
The performance of AEPO with $\lambda$ of $1.0$ outperforms the performance of AEPO with higher or lower $\lambda$ (Table 2 and 5). The result shows that both the diversity and the quality of responses are important for the preference dataset, which is consistent with the observations in previous work [, , , ].

#### AEPO is also effective for off-policy learning.

Several studies have shown that using responses generated by the training model itself (on-policy learning) is more effective than using responses generated by other models (off-policy learning) [, , , , , , ].
Nevertheless, off-policy learning is advantageous in resource-constrained settings because it can leverage existing public resources to train arbitrary models. To this end, we investigate the use of AEPO for off-policy learning.

We use the preference dataset $\mathcal{D}_{AE}$ generated by Mistral’s responses $\{y_{i}\}_{i=1}^{N}$ on AlpacaFarm to train dolly-v2-3b (Dolly).(^11^1111[https://huggingface.co/databricks/dolly-v2-3b](https://huggingface.co/databricks/dolly-v2-3b)) We set the LoRA’s $r=32$ and $\alpha=r/4$ for Dolly. Other experimental settings are the same as the experiment on Mistral.
Table 5 shows the results of the off-policy learning.
AEPO with sufficiently large $\lambda$ outperforms vanilla DPO measured in Eurus.
The result shows the potential of AEPO to improve the efficiency of off-policy learning.

#### AEPO can improve out-of-domain performance.

Previous work has shown that training on a diverse set of instructions improves the performance on out-of-domain tasks [].
The question is whether we can achieve a similar robustness with a diverse set of responses generated by AEPO.
We evaluate the Mistral models fine-tuned with the AlpacaFarm dataset on ARC [], HellaSwag [], TruthfulQA [], and WinoGrande [] using the language model evaluation harness [].
Table 6 summarizes the scores and the standard errors of the trained models on these benchmarks. Overall, AEPO scores slightly higher than WoN, except for the ARC.
The result shows that AEPO outperforms WoN in the AlpacaFarm domain not because it overfits to the task, but because it improves on a wide range of tasks.

Table: Table 6: Evaluation of DPO models trained with AlpacaFarm on out-of-domain benchmarks. Means and standard errors are reported.

## 5 Related Work

#### Minimum Bayes risk decoding.

The DMBR objective we use is derived from the MBR objective which was originally proposed as a decoding strategy.
MBR decoding is a text generation algorithm that selects the sequence with the highest similarity to the sequences generated by the probability model.
It has been proven to produce high-quality text in many text generation tasks, including machine translation, text summarization, and image captioning [, , , , , , , ].
The novelty of our work is to introduce the DMBR objective as a strategy to subsample representative responses from many candidates so that the annotation budget can be used efficiently.

#### Active learning.

Annotation-efficient learning has long been a challenge in natural language processing [].
Active learning is an approach that aims to achieve training with fewer training labels by proactively selecting the data to be annotated and used for learning [, , ].
There are roughly two active learning strategies used in NLP []. One uses the informativeness of the data instances, such as uncertainty and disagreement of the models [, , , ]. This approach has proven to be efficient in many text classification tasks.
The other strategy is based on the representativeness of the data instances [, , , ].
By annotating instances with high average similarity to all other instances, it can cover a large portion of the dataset with few annotations.
Another approach is to select instances that maximize the diversity of labeled instances [, , ].
Our approach is more related to representativeness as we can consider the DMBR objective as a combination of similarity and diversity measures. The novelty of our study lies in applying these ideas to preference learning, aiming to reduce the annotation cost.

## 6 Conclusions

We propose Annotation-Efficient Preference Optimization (AEPO), an annotation-efficient dataset subsampling strategy for language model alignment.
The selection strategy is derived from DMBR algorithm, the state-of-the-art decoding strategy for optimizing the diversity-quality tradeoff, and can efficiently select diverse and high-quality responses from the candidates.
By focusing the annotation effort on the selected responses, AEPO achieves efficient preference optimization with a limited annotation budget.
Experimental results show that the proposed strategy outperforms the baselines on AlpacaFarm and Anthropic’s Harmlessness dataset. We also evaluate AEPO in off-policy learning and out-of-domain tasks and show that it is effective in these settings as well.
We believe that AEPO is a critical contribution to promoting preference optimization research by addressing the severe obstacle, the cost of creating better preference data.

## Appendix A Hyperparameters

Table 7 lists the hyperparameters we use to run DPO.
Table 8 lists the hyperparameters we use to generate the texts for evaluation.

**Table 7: DPO hyperparameters.**
| Parameter | Value |
| --- | --- |
| Training epochs | 3 |
| Batch size | 4 |
| Regularization factor ($\beta$) | 0.1 |
| Optimizer | RMSProp |
| Learning rate | 1e-5 |
| Learning rate scheduler | linear |
| Warm up steps | #instructions / 80 |
| Max instruction length | 512 |
| Max new tokens | 512 |
| Max total length | 512 |

**Table 8: Generation hyperparameters on evaluation.**
| Parameter | Value |
| --- | --- |
| Max instruction length | 512 |
| Max new tokens | 512 |
| Temperature | 1.0 |
| Top-$p$ | 0.7 |

## Appendix B Ablation Study

We describe the ablation study to evaluate the effect of AEPO in different settings.

### B.1 LoRA Hyperparameters

We evaluate the effect of the LoRA hyperparameters on the performance of AEPO.
We run DPO once with LoRA’s $r\in\{32,128\}$ and $\alpha=r/4$.
All other experimental settings are the same as in Section 4.
Tables 9 and 10 show the experimental results.
We observe that AEPO outperforms WoN in reward scores as in Section 4 regardless of the choice of the LoRA’s $r$.

Table: Table 9: Evaluation of AEPO on AlpacaFarm using Mistral with LoRA’s $r=32$ and $\alpha=r/4$.

Table: Table 10: Evaluation of AEPO on AlpacaFarm using Mistral with LoRA’s $r=128$ and $\alpha=r/4$.

### B.2 Loss Function

Several variants of loss functions are proposed to replace the sigmoid loss function of DPO [, , ]. The experimental results of AEPO using hinge loss and KTO loss are given in Tables 11 and 12. We use LoRA $r=32$ and LoRA $\alpha=r/4$. Other experimental settings follow the settings in Section 4.
We observe that AEPO outperforms the baselines regardless of the choice of the loss function.

Table: Table 11: Evaluation of AEPO on AlpacaFarm with Mistral using *hinge loss*.

Table: Table 12: Evaluation of AEPO on AlpacaFarm with Mistral using *KTO loss*.

## Appendix C Generation Examples of AEPO

We show examples of the dataset entries generated by AEPO and WoN in Tables 13 and 14. AEPO with $\lambda=0.0$ selects two very similar or even identical responses. In contrast, the two responses selected by AEPO with $\lambda=1.0$ are sufficiently diverse and of high quality.

Table: Table 13: Example of a preference dataset entry generated by AEPO and WoN on AlpacaFarm dataset. The input prompt is: Paraphrase this sentence to have the same meaning.\n\nWe have had a prolonged period of extremely cold weather.

Table: Table 14: Example of a preference dataset entry generated by AEPO and WoN on AlpacaFarm dataset. The input prompt is: Write two sentences with the same meaning but with different wordings\n\nHe went to the store.

## Appendix D Evaluation of West-of-N Strategy

We do not claim that AEPO outperforms WoN in every situation. WoN is an effective strategy when an abundance of annotations is available. Table 15 shows the performance of DPO with the WoN strategy using $N$ annotations per instruction without reducing the size of the instruction set.
As shown in previous work [, ], the WoN strategy significantly improves the performance of the resulting DPO models at the cost of additional annotations. The win rate against the SFT model is shown in Figure 4.

Table: Table 15: Evaluation of DPO with the WoN strategy on AlpacaFarm using Mistral. The results of $N=2,128$ are the average of three runs, while the rest are of a single run.

Figure: Figure 4: Evaluation of the DPO using the WoN strategy without limiting the annotation budget on AlpacaFarm using Mistral. The Eurus reward model is used for evaluation. With more annotations, WoN outperforms AEPO.

## Appendix E Additional Results

Tables 16 and 17 show the results of training Mistral on AlpacaFarm including the results of the proxy reward model (OASST).
The results of Mistral on Anthropic’s Helpfulness and Harmlessness are shown in Tables 18, 19, 20, and 21.
Table 22 is the result of training Dolly.

Figure 5 shows the diversity (distinct-2 and distinct-3) and quality of the preference dataset $\mathcal{D}{}_{AE}$ generated by AEPO. AEPO successfully makes use of the set of responses to select diverse and high-quality responses to be labeled by the annotator, making the annotation process more efficient.
Figures 6 and 7 show the results of Anthropic’s Helpfulness and Harmlessness datasets. AEPO achieves higher diversity and quality in both datasets.

Table: Table 16: Reward score of the AEPO on AlpacaFarm using Mistral. The best score is in bold, and the second best is underlined. The mean and standard deviation of three runs are shown. Note that OASST is used as a proxy reward model to annotate the preference of the training dataset.

Table: Table 17: Win rate against the SFT model (Mistral) on AlpacaFarm. The best score is in bold, and the second best is underlined. The mean and standard deviation of three runs are shown. Note that OASST is used as a proxy reward model to annotate the preference of the training dataset.

Table: Table 18: Evaluation of AEPO on Anthropic’s Helpfulness dataset using Mistral. The mean and standard deviation of three runs are shown. Note that OASST is used as a proxy reward model to annotate the preference of the training dataset.

Table: Table 19: Win rate against the SFT model on Anthropic’s Helpfulness dataset. The mean and standard deviation of three runs are shown. Note that OASST is used as a proxy reward model to annotate the preference of the training dataset.

Table: Table 20: Evaluation of AEPO on Anthropic’s Harmlessness dataset using Mistral. The mean and standard deviation of three runs are shown. Note that OASST is used as a proxy reward model to annotate the preference of the training dataset.

Table: Table 21: Win rate against the SFT model (Mistral) on Anthropic’s Harmlessness dataset. The mean and standard deviation of three runs are shown. Note that OASST is used as a proxy reward model to annotate the preference of the training dataset.

Table: Table 22: Evaluation of preference dataset configuration strategies for off-policy learning. We generate responses using Mistral and use the generated responses to train Dolly. LoRA hyperparameters are set $r=32$ and $\alpha=r/4$. Note that OASST is used as a proxy reward model to annotate the preference of the training dataset.

Figure: (a) $\uparrow$Distinct-2

Figure: (a) $\downarrow$Sentence BERT

Figure: (a) $\downarrow$Sentence BERT

## Appendix F Computational Resources

Text generation and DPO training run on an instance with an NVIDIA A100 GPU with 80 GB VRAM, 16 CPU cores, and 48 GB memory.
A single run of DPO takes approximately 50-55 minutes on the A100 instance for the AlpacaFarm and hh-rlhf datasets.
AEPO runs on an NVIDIA A2 GPU with 8 GB VRAM, 8 CPU cores, and 24 GB memory.
AEPO takes about 46 hours on the A2 instance to run with $N=128$ and $k=2$ for all of the AlpacaFarm and hh-rlhf training data.

## Appendix G Limitations

Although our method is motivated by the situation where the annotation is needed to align the language model, our experiments are conducted using a reward model to annotate preference on training datasets.

We use automated evaluation metrics to evaluate the models. Although we use one of the most accurate publicly available reward models to evaluate the performance of the models [, ], it would be desirable to perform a human evaluation.

Our focus is on developing a method to generate a diverse and high quality set of responses.
The preparation of diverse and high quality instructions is also an important task to generate an efficient dataset [, , , , ].
Our method is orthogonal to methods for generating high quality instructions and can be combined. Comparing and combining AEPO with methods for generating diverse instructions is future work.

All experiments are performed using LoRA. The evaluation of AEPO with full parameter fine-tuning is a future work.
Our experiments are limited to the evaluation on DPO. Evaluating AEPO on other preference optimization algorithms (e.g., RLHF, LiPO) is future work.

The performance of AEPO depends on the choice of the hyperparameter $\lambda$. We observe that $\lambda=1.0$ is a good choice throughout the experiments, but developing a strategy to find an effective $\lambda$ for a given dataset is future work.

One of the objectives of DMBR is to find a set of high-quality responses (Eq. 4).
However, previous work shows that DPO is more efficient using a low reward response as a rejected response than using a high reward response [, ].
Optimizing the objective function for DPO to find a low quality response as a rejected response is future work.

## Appendix H Impact Statement

We believe that this work will have a positive impact by encouraging work on AI systems that work better with a diverse set of people.
LLMs would be more useful if they could adapt to the preferences of diverse groups of people, even if little preference annotation is available from their communities.

We foresee our method being useful for personalizing LLMs [, , ].
Personalized LLMs could have far-reaching benefits, but also a number of worrisome risks, such as the propagation of polarized views. We refer to Kirk et al. [] for a discussion of potential risks and countermeasures for personalized LLMs.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Would it be possible to apply this approach to selecting *instructions* as well as generations? That might yield a more interesting and useful algorithm.

What is the size of the candidate set for the results in figure 5?
soundness: 1
strengths: The authors identify an important problem, which is the need to make best possible of use of human preference annotation effort. The paper offers a broad range of evaluations and a comprehensive discussion of the recent preference optimization literature. The method is described clearly and would be straightforward to implement, although the authors also make code available.

For me, the most persuasive result is AEPO at N>2 vs AEPO at N=2, which shows that the proposed method of example selection offers some improvement over naively sampling two random responses.
summary: The paper proposes to select generations for preference annotation by optimizing an objective that combines representativeness (in the sense of the selected generations being similar on average to the full set of generations) and diversity (in the sense of the subset of selected generations being dissimilar to each other). This method is compared primarily with "West-of-N", in which preferences are annotated between all pairs of generations.
weaknesses: From a conceptual perspective, the idea of selecting diverse *and* representative training examples is not original, and the application to preference learning does not change the fundamentals in a significant way. See, e.g. (Wei et al 2006 "Submodularity in Data Subset Selection and Active Learning"; Bıyık et a; 2019 "Batch active learning using determinantal point processes"), which propose more principled approaches to the same goal. In fact, the situation is significantly simplified in the submission because the subset size is limited to 2, so that it is possible to completely enumerate the maximization over subsets in equation 18. A related point is that when selecting larger subsets, the approach of *averaging* over all similarities/distances may not be meaingful. Instead, it may be better to ensure that every point in Y^cand is well-covered by at least one example, but there is no need to cover it more than once. This is the intuition behind the Coreset baseline (which I was glad to see included), but it only becomes meaningful for larger subsets.

Empirically, the comparison with West-of-N balances the number of preference annotations (which is reasonable) by offering WoN dramatically fewer instructions. It is unsurprising that WoN does worse when offered up to 64 times fewer instruction examples, and it seems likely that for any fixed annotation budget B, we would do best by offering B/2 instructions with exactly two generations each.

Theoretically, the approach is motivated by an information theoretic argument in section 3. However, without a probability model underlying H[R], this argument is too informal to shed much light on the proposed approach.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness

Consider to cite those relevant works: 
https://arxiv.org/abs/2402.00396
https://arxiv.org/abs/2406.12168
soundness: 2
strengths: 1. Annotation efficiency is an important research problem in preference learning.

2. AEPO empirically has some improvements over DPO or DPO variant.
summary: Authors extended DMBR decoding into preference annotations and propose annotation-efficient preference optimization (AEPO). AEPO annotates a subset of responses that maximize diversity and representatives from the available responses. AEPO was emperically shown to outperform DPO under the same annotation budget.
weaknesses: 1. The AEPO approach appears to be a straightforward extension of Diverse Minimum Bayes Risk (DMBR) decoding (https://arxiv.org/pdf/2401.05054) applied to preference annotations. The objective function remains largely identical to that in DMBR decoding.

2. Lack of Comparative Baseline with DMBR: Since the DMBR paper already claims to improve data quality in generated responses, it’s essential for the authors to evaluate AEPO directly against DMBR decoding. This would clarify whether AEPO actually enhances annotation efficiency or simply benefits from improvements in data quality.

3. Hyperparameter Sensitivity and Generalization: The model’s performance appears to be highly sensitive to the choice of λ, as shown in Figure 5. This heavy dependence on tuning raises concerns about the method’s generalizability across different tasks and datasets, as selecting an optimal λ may require extensive trial and error.

4. Lack of baselines using two selection principles: If you want to focus your contributions on your two data selection principles. You should at least explore more implementations. For example, you can do diversity beam search and apply MBR as a baseline. To prove those two principles are general and not constraint on a single implementation.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. The sum in (13) is a bit surprising. Isn't this misaligned with the intuitive goal of distinctness. A group of very similar examples with one extraordinarily dissimilar one could get a higher distance sum here than a group of uniformly quite dissimilar examples. Does this matter? I am not sure whether this can arise, and it could be that the min/max sample step in (2) makes it irrelevant.

2. The paper says on line 144 that the dataset is the main factor in outcomes in this space. What is the evidence for this, as opposed to tracing results to interactions between the dataset, the model, and the objective?
soundness: 4
strengths: AEPO seems like a valuable technique targeting an important area, and the results show that being smart about which preference pairs to use can be more effective than simply having a lot of data, at least for the configurations studied in the paper.

The experiments seem to be carefully done and comprehensively reported, and the gains seem significant.
summary: This paper develops a method called Annotation-Efficient Preference Optimization (AEPO) for creating more efficient and effective preference datasets by subsampling responses using criteria that seek to maximize diversity and representativeness, where these quantities are estimated based on embedding similarity. The experiments use Mistral-7b-sft-beta as the model, DPO as the core preference learning objective, and AlpacaFram and Anthropic's Helpfulness/Harmlessness as the datasets. The annotations are done by the OASST reward model. In these experiments, AEPO outperforms random sampling, and random sampling with only the best and worst kep (West of N). Ablation studies show that both the diversity and representativeness make important contributions to these results.
weaknesses: 1. The opening of the paper emphasizes how important it is to use human annotators, so it is surprising to find, in section 4, that no human annotation was used for the experiments.

2. The above is part of a larger tweak to the narrative that I would suggest: there is evidently no reason to focus on cost in particular, since the expensive settings seem to do very poorly.

3. The choice of DPO as the objective is significant, I believe. In Figure 3, the blue line goes up a bit and then goes down as the number of responses increases. Evidently, this process is actually hurting. A good explanation of this is that the AlpacaFarm data are just overall much worse than the model (Mistral), so that even the winning AlpacaFarm examples are actually bad from the perspective of the model. This happens with DPO because DPO just wants the winner to be more likely than the loser -- it doesn't care what the likelihoods of either are before learning begins. If this is a correct explanation, then what AEPO is doing is not about cost but rather about quality. For different objectives than DPO, we might see a different pattern.

4. I can't find a version of Figure 3 for the Anthropic dataset. That would be significant because I suppose Mistral has the opposite relation to that dataset -- it is weaker than whatever model generated the Anthropic examples. Please do let me know if I have missed something.

5. The Related work is pretty sparse. I suppose we can give a pass, but it would be good to at least cite other work that has sought to improve these preference datasets. [Wang et al. 2024](https://arxiv.org/abs/2407.16216) is a valuable survey of methods in this area -- it could be cited, as could closely related papers it cites.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - How do you optimize Eq. 18?
- In line 188, it is mentioned that after AEPO, West-of-N is applied to select 2 candidates, is this necessary? Is there ablations on this?
soundness: 2
strengths: The motivation behind this paper is highly relevant to the research community. In scenarios where human annotations are costly, reducing the number of annotations can substantially cut down expenses. Leveraging information gain as a criterion for sample selection in the context of alignment presents a promising approach.
summary: This paper proposes a strategy, annotation-efficient preference optimization (AEPO), to select a subset of preference pairs to be sent for annotation. Collecting human annotations is costly, therefore, if we can select only a subset of preference pairs and achieve close performance to that of annotating the whole set it can cut down costs significantly. The authors propose to select a set of responses from the pool of candidate pairs that have the highest information gain. They propose two heuristics for selecting the subset that might lead to a higher information gain: (1) selected elements should be, on average, close to all candidate pairs, and (2) selected elements should exhibit diversity.  AEPO is benchmarked against other subsampling strategies by using the subsampled data with DPO and evaluate the performance of the aligned model.
weaknesses: **Methodology:** There is a type mismatch between the definition of information gain (which applies to sets of responses) and its application to preference pairs (which are sets of response pairs). Additionally, the paper introduces information gain broadly but quickly shifts to the two heuristics without a formal mathematical connection between them. A constructive addition would be to use toy examples to measure information gain and demonstrate how the two heuristic metrics correlate with it.

Maybe I have missed it, but the authors mention they optimize Eq. 18, without mentioning how they do that. Does this involve iterating all N choose k pairs and picking the best one? How efficient is the selection algorithm? Discussing the efficiency of the selection algorithm is an important factor that is missing in the paper.

**Implementation:** The paper distances itself from approaches focused on generating quality synthetic data, aiming instead to design a subsampling strategy that minimizes human annotation requirements. Given this, the use of a reward model in place of human preferences in the experiments does not make sense. If the primary objective is to evaluate the algorithm's efficiency in selecting representative samples for human annotation, direct use of human preferences appears essential. Alternatively, one could consider using a large reward model as a proxy for human judgment and a smaller reward model for methods like West-of-N that are specifically for synthetic data generation.

Similarly, the comparison to West-of-N may not be entirely fair. The authors use fewer prompts for West-of-N, which could unfairly disadvantage its performance; it’s possible that West-of-N is highly effective, and the reduced prompts are the primary factor impacting results. Figure 11 in the appendix seems to support this interpretation. A potential solution is to use a reward model for West-of-N while using human annotation for AEPO, or to use a weaker reward model for West-of-N and a stronger one for annotating the final two AEPO candidates while using the same number of prompts for both methods.