Deep Bayesian Active Learning for Preference Modeling in Large Language Models
Authors: Luckeciano Carvalho Melo
Panagiotis Tigas
Alessandro Abate
Yarin Gal
Venue: NeurIPS 2024 poster (2024)
Keywords: Bayesian Active Learning
Preference Modeling
Language Models
Primary area: natural_language_processing

Abstract: Leveraging human preferences for steering the behavior of Large Language Models (LLMs) has demonstrated notable success in recent years. Nonetheless, data selection and labeling are still a bottleneck for these systems, particularly at large scale. Hence, selecting the most informative points for acquiring human feedback may considerably reduce the cost of preference labeling and unleash the further development of LLMs. Bayesian Active Learning provides a principled framework for addressing this challenge and has demonstrated remarkable success in diverse settings. However, previous attempts to employ it for Preference Modeling did not meet such expectations. In this work, we identify that naive epistemic uncertainty estimation leads to the acquisition of redundant samples. We address this by proposing the Bayesian Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition policy that not only targets points of high epistemic uncertainty according to the preference model but also seeks to maximize the entropy of the acquired prompt distribution in the feature space spanned by the employed LLM. Notably, our experiments demonstrate that BAL-PM requires 33\% to 68\% fewer preference labels in two popular human preference datasets and exceeds previous stochastic Bayesian acquisition policies.

Deep Bayesian Active Learning for Preference Modeling in Large Language Models
Luckeciano C. 1.
Panagiotis Tigas'
Alessandro Abate Yarin Gal
OATML, University of Oxford
OXCAV, University of Oxford
Abstract
Leveraging human preferences for steering the behavior of Large Language Models (LLMs) has demonstrated notable success in recent years. Nonetheless, data selection and labeling are still a bottleneck for these systems, particularly large scale. Hence, selecting the most informative points for acquiring human feedback may considerably reduce the cost of preference labeling and unleash the further development of LLMs. Bayesian Active Learning provides a principled framework for addressing this challenge and has demonstrated remarkable success diverse settings. However, previous attempts to employ it for Preference Modeling did not meet such expectations. In this work, we identify that naive epistemic uncertainty estimation leads to the acquisition of redundant samples. We address this by proposing the Bayesian Active Learner for Preference Modeling (BAL-PM),, a novel stochastic acquisition policy that not only targets points of high epistemic uncertainty according to the preference model but also seeks to maximize the entropy of the acquired prompt distribution in the feature space spanned by the employed LLM. Notably, our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference labels in two popular human preference datasets and exceeds previous stochastic Bayesian acquisition policies.
Introduction
Preference Modeling is key component to aligning unsupervised pre-trained Large Lan- guage Models (LLMs) towards human prefer- ences M44. It is often performed by collecting human feedback for set of prompt-completion pairs and then leveraging the data steer the behavior of such models, either directly [5] or via reward models [6]. Nevertheless, human feedback generation is laborious 77). especially when requires specialized knowledge 9 Furthermore, the quality of the prompts has crucial impact on the performance of fine-tuned models [10]. Hence, selecting the most infor- mative points to gather feedback is essential to reduce costs and enable better L.LMs.
Log Likelihood
0661
-066
33% fewer samplesi
0.63
Random Sampling
BALD
BAL-PM (ours)
2000
8000
12000 Acquired Data
16000
20000 24000
Figure 1: Log-Likelihood of learned preference models the Reddit TL;DR dataset [1. Our method, BAL-PM, reduces the volume of required human feedback by 33% over random acquisition.
Despite its substantial impact, data selection for Preference Modeling poses significant chal- lenge. The prompt-completion pool is arbitrarily
Correspondence to: htt.eiiin..aaaalllmmmeeecoo.ooee
Denotes equal supervision.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Preferance Fool Pisee
Base LLM
Feature Space
X
Feature Space Entropy Score
ago heeeg p009 ens with our cblle The situun
H( U (aft)
Imarady stoudd (-3 Prior my Ivrhod dissnoo was 10 milee Paae
Phompt
Acquired Prompt
Tino
YeAY 61 bu. (i (e))
Laing HIMl cacte,
(wwC display picture my HD OVR HDMI cuble Nead In...
(a).vis od)
Fset to C. 21 talked inso
hall
my
(o" ol)
and ended up by trinn aha freecation...
Preference Model
Base LLM
Boyesian Preference Modial
Epistemicc Uncerbainty Score
An illustration of how BAL-PM works. For each tuple (z, V1, V2) DD Dooot we obtain features for the prompt and prompt-completio pairs by computing the last layer embeddings of the base LLM. We leverage the prompt feature space to estimate the entropy score of the acquired prompt distribution, (Xrainn U {x))) Similarly, we use the prompt-completion features input for the Bayesian Preference Model, which used to estimate task-dependent epistemic uncertainty scores, (I, y1, 32). BAL-PM selects the tuple that maximizes the linear combination of both scores.
Figure 2:
large and semantically rich. Additionally, human feedback is inherently noisy, with low agreement rates among labelers, typically observed between 60% 75% for these settings 66] 1 [2]. Lastly, the intrinsic scale of LLM development requires parallelized labeling and makes frequent model updates prohibitively expensive, limiting the applicability of many active learning schemes that rely on single-point acquisition
[K]
Bayesian Active Learning provides a principled approach to data selection [4 which has demonstrated remarkable success across different fields [1719]] However, its application in Active Preference Modeling is not straightforward. Past attempts of employing the framework in this setting reported no benefits over random selection (20], arguably due poor uncertainty estimation in the context of LLMs, which is indeed an open challenge and active area of research (21).
We identify two reasons for this phenomenon. First, the inherent bias of approximate Bayesian inference deep learning models, particularly for LLMs. Second, and more nuanced, the current intractability of epistemic uncertainty estimation methods in Preference Modeling for LLMs, a context that intrinsically requires batch acquisition. Proper estimators for this setting present combinatorial complexity, and even greedy approximations are still computationally demanding and impractical This limitation leads to relying on simpler single-point acquisition schemes such as BALD 233 (as in Gleave and Irving (20)), designed to acquire individual points followed by model updates. However, these assumptions are far from realistic for the scale Preference Modeling in LLMs, and naively applying such methods for batch acquisition leads to the selection of redundant samples.
13 [22].
In this work, we argue that leveraging the information available from the feature space spanned by the LLM a task-agnostic]] source of epistemic uncertainty alleviates these problems. We propose Bayesian Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition policy that not only targets points of high epistemic uncertainty according to the preference model but also seeks to maximize the entropy of the acquired prompt distribution in the feature space. This entropy score encourages the active learner to select prompts from low-density regions, effectively reducing the feature space epistemic uncertainty [24]. As a result, it promotes diversity in the acquired training set, preventing the selection of redundant samples and also helping in learning better Bayesian preference model and its task-dependent epistemic uncertainty estimates for subsequent acquisitions. Figure Segurre]ilusten iillustrates how BAL-PM works.
We conduct active learning experiments in the Reddit and CNN/DM preference datasets 25 26
to validate our method. BAL-PM demonstrates strong gains over random sampling, reducing by approximately 33% (as shown in Figure T and 68% the volume of feedback required to learn the
>ttask" refers to Preference Modeling, A task-agnostic estimation is independent of preference labels.

---

preference model in the considered datasets. It also consistently surpasses other strong stochastic Bayesian acquisition policies [22]. Finally, we further analyze the acquired prompt distribution to show that BAF-PM prevents redundant exploration and effectively balances the contribution of the two sources of epistemic uncertainty.
2 Related Work
Bayesian Active Learning is established form of active learning that leverages the uncertainty in model parameters to select the most informative points [14] [27 [6]. demonstrating relevant impact In this work, we apply this technique for Preference Modeling [30, 31 in LLMs. Given the requirements of such a problem setting, we focus on batch acquisition [4. 13, particularly in the design of stochastic acquisition policies, similarly Kirsch et al. [22]. However, our work fundamentally differs from theirs in the strategy of incorporating stochasticity. The policies introduced by Kirsch et al. [22] directly sample from the distribution determined by the single-point, task dependeenddnt epistemic uncertainty scores. In contrast, our method maximizes the entropy of the acquired data distribution, which allows leveraging task-agnostic source of epistemic uncertainty, alleviating the effect of biased task desendennddntt uncertainty scores.
in several applications [19] [8] 17 28 29
Active Preference Modeling leverages active learning techniques reduce the feedback needed for (27). There has been a recent surge of interest in the area [3]
Preference Modeling
20 36 given
the impact of Preference Optimization in fine-tuning LLMs 2 10 6 A portion of this line of work focuses on query generation to directly optimize preferences. Mehta et al. (32] theoretically formalizes
the problem and proposes method that generates one completion to maximize the uncertainty of the triple in a kernelized setting. Das et al. [35] proposes a method based on confidence bands, accounting for both completions in the triple and relaxing linearity assumptions on the reward function. Ji et al. 33 constructs an optimistic estimator for the reward gap between completions and selects those with the least gap, using an uncertainty estimator to reduce query complexity. Lastly, Dwaracherla et al. 36 generate completions using double Thompson Sampling, representing epistemic uncertainty with an Epistemic Neural Network [37. similar to our Bayesian model. Overall, while having the shared goal of reducing the volume of human feedback, query generation is orthogonal to our problem setting. Instead, we focus on the pool-based setting, as in Gleave and Irving 20), which allows us to leverage real human feedback in experiments rather than relying synthetic preference simulators. Gleave and Irving 200 was the first attempt of Bayesian Active Learning in this setting, and it directly applies BALD acquisition (23) for Preference Modeling, using fully fine-tuner deep ensemble for epistemic uncertainty estimation. In contrast, our work proposes a new objective that extends BALD acquisition to account for the entropy of the acquired prompt distribution to encourage the acquisition of more diversified samples, and formulates the Bayesian model as an ensemble of adapters.
Task-Agnostic Uncertainty Estimation refers to set of techniques that quantifies uncertainties based on density estimation of the input in a learned latent feature space [88] 39). In this context, distant points from the training set offer more information about the input space, which is useful for -utrof-distribution detection R9] and unsupervised active learning (40). Similarly, leveraging information from the feature space via entropy maximization is a common approach in Reinforcement Learning for state exploration 4M44 While our method relies on the same principles acquiring more information about the feature space our problem setting and methodology differ substantially. as we focus on Active Preference Modeling in the context of LLMs.
3 Preliminaries
Problem Statement. We formulate our setup as an active inverse variant of the contextual ducling bandit problem [45] 46). We assume a prompt space X, an action space y and a language policy T: X' x x + [0, oc). Given x ~ X, this language policy T (e.g., an LLM) selects actions yı, 8/2 7(. z) (also referred to as completions), generating dataset of tuples Dpoot = fx', yi. v J. Crucially, x is sampled with replacement, i.e., we may generate multiple completions for the same prompt. Then, we define policy π : X' x y x y + [0, 20), which select tuples (I. 31, V2) E D Dooo to query for human binary preference over completions f/1 > y/, forming a preference dataset Derain = {x², vi. V22 y1 > ) Finally, Derain is used to learn preference model po > y22 Jr, VI, 12). parameterized by 0, which aims to recover the human preference function. The goal is to find x that minimizes the amount of samples B required to learn po(y1 y/2 x, V1, 92).
3

---

Preference Modelingg In this work, we assume that the preferences y/2 are generated by an unknown latent reward model r(x, y). We model y1 > Y/2 following the Bradley-Tery (BT) model
47
expr(x. v1)
ply > 92 23 91. 122
expr + (x,
The BT model is often implemented by learning parameterized latent reward model re(a.y) and op- timizing 9 via maximum likelihood estimation. This means minimizing the negative Log-Likelihood with respect to the human preference labels.
Bayesian Active Learning. We adopt Bayesian Model, which assumes a probability distribution over the parameters 0, such that, given a classification setting with inputs æ ~ X and labels Y the predictive preference distribution is given by:
ply Φ) Exre (p(( x 0)].
For active learning, follow the methodology introduced by Lindley (3), namely BALD (Bayesian Active Learning by Disagreement), which proposes that the utility of a data point : ~ X is given by the expected information gain about the parameters 0 with respect the predictive distribution, proxy of epistemic uncertainty:
(xx) = I(0, y Dirain: x)
H(p(y æx Dirain)) - Epp(0 Deresn) [H(p(y xx 0)).
(3)
Berachennk--eoneniie Entropy. The KL entropy estimator [48] is non-parametric, particle-based estimator that leverages the k-nearest neighbors distance. Given random variable X and set of N i..d particles {æ,} 24 ~ X, the KL entropy estimation for is defined as:
HxL(X)= dx Σ log (i( log Udx + +NN) (k),,
where dx is the dimension of X, Fdy is the volume of the dx dimmennsson unit ball, is the digamma function, and (1) is twice the distance between the particle x; to its k-nearest neighbor.
4 Bayesian Active Learner for Preference Modeling
We now introduce our method for Active Preference Modeling, BAL-PM, illustrated Figure Our desiderata is to design an acquisition policy that addresses the shortcomings of naive epistemic uncertainty estimation - such as the acquisition of redundant samples - by leveraging an unsupervised source of epistemic uncertainty that encourages diversity in the acquired training distribution.
Objective. Based on the above, we propose the following objective:
arg max E2
B fx)),
(5)
where Üz, 31, 82) is the preference model epistemic uncertainty estimate for the tuple (I, V1, Y2) and H((err U {x)) is the entropy estimate for the acquired prompt distribution, assuming the policy selects x. Xtr acquired training set. Lastly, 3 is hyperparameter to balance the contribution of each term. Crucially, the first term represents task-dependent source of epistemic uncertainty, since it refers to the learned preference Model. In contrast, the second term represents task-agnosttr source, it solely relies on the information available in the feature space spanned by the base LLM.
slight abuse of the notation that refers to the set of prompts in the previously
Preference Model Epistemic Uncertainty Estimation. We first describe our Bayesian Preference Model. Assuming a prior distribution over parameters p(0), the posterior predictive distribution over preferences after observing Dtrain is given by:

---

plv1 > y/2 2, M/: M2, D Derain) = p(3/1 > M1, U2, O)p(O D Deruin )d0,
(6)
where the likelihood term p(y1 > y2 x, 31, 92. 0) follows the BT model in Equation | Consid- ering deep models, solving this inference problem is intractable, given the large parameter space. Nonetheless, we may assume a simple yet effective posterior approximation via deep ensembles
49 51
K 5 0x)
p(0) D Dtrain)
k=
Equation 8 approximates the posterior distribution over parameters p(0 ( (erain)) as mixture of delta functions, where K is the number of ensemble models and 02 the MAP estimate of model k. The posterior predictive distribution is then computed via the following approximation:
p(y1 > y/ xr U1 U2: Dtrain)
plyn > y2 x, v1. U2, 0x), Oz ~ p(0 Dtrain)-
(8)
Equations and allow approximate inference by training multiple preference models separately. However, this is challenging in the context of L.Mss, as fine-tuning billions of parameters several times computationally expensive and impractical in many settings. Alternatively, we employ an ensemble of adapters (52) [3], which consists of multiple lightweight networks (with few million parameters each) on top of the frozen LLM that works as feature extractor. This allows us to generate the LLM features offline and use them required for training and Bayesian inference. This also enables using very large base models, with dozens hundreds of billions of parameters in single GPU setting. Finally, based on the previous modeling assumptions, we can estimate the epistemic uncertainty term employing Equation 3
dataset, considerably reducing the resources
K
(x, V1, (1,11,822) = H( Ep(uv p(ul - y2 x 31. U2, 0x))
Σ
H(ply) > y2 x.
2. U1. U2. 0k)). (9)
K
Feature Space Entropy Estimation. Equation requires estimating the entropy of the acquired prompt distribution, H( H((Xtrain). For this matter, employ a kNN-based entropy estimator. We represent each prompt in the pool as the last- layer embedding vector generated by the base LLM, leveraging the semantic representations learned during unsupervised pre- mreitrinnng..
X Available Prompt
Acquired Prompt
INN k
MNN K=5
However, naively applying the KL estimator from Equation 4 has major drawback: the training set Drrain initially contains very few data points and does not provide support to rep- resent the probability density, introducing bias to the estimates and affecting the data selection.
(e) KL
(b) KSG
Figure 3: Illustration of entropy estimators. The green point maximizes the entropy estimation of the prompt distribution (according the employed estimator). Dashed lines represent its k-NN dis- tance. In (a), the KL estimator (Equation 4 does not account for the available prompts in the pool (in red) and underestimates the density regions not covered by the acquired set (in blue). In (b), the KSG estimator (Equation 10) uses all data points, leading to better estimation and effectively select- ing the point that maximizes the true entropy.
For illustration, we show the scenario of Fig- ure 3.. In this case, we estimate the entropy using Equation with à: 3. Since does not account for the available points in the pool, it underestimates the density around the top clus- ter and ends up selecting the green point as the one that maximizes the entropy of the feature space, while the point that does is in the bottom cluster. In extreme where all the points in
5

---

Algorithm 1 BAL-PM
Require:
Pool set D poot (x', y 1/ training set Drrain {xr, vi 22. y1 > y2'}
Require:
Base LLM T, Bayesian Preference Model p(y > y/2 xt yr, 92)
Compute feature sets for poot and Dirain by performing forward passes on
Compute KNN distances for points in pooo U D Derain
while true do
Train Bayesian Preference Model (ensemble) in Dtrain assuming Equations and
Compute Epistemic Uncertainty Estimates Üzz U1, 12) via Equation
Initialize nxx,, (ar) by counting {u E DDrain A (Ilx ul I ((x) Vx E poo!
Initialize Batch: B = 0
while batch not full do
Compute entropy term: e(z) log D(x) xx v (x) +
Select tuple (x", 122 following πT arg max
(z, V1, 12) Be(x)
(2. poot
BU (x*,, V/: 1/5 Update counts: Vx E 'D pool, fl. (x) = n.x.. (z) + 1 if end while
Update Pool and Batch: D Pooot Dpool (x" Vi V2
x²1 S 2
Collect human feedback for B and update training set D Dtrain UB end while
the top cluster are the same, this bias leads to acquiring duplicated points. In Appendix we formally derive the KL entropy estimator and show how the low-data regime challenges its main assumptions.
Alternatively, we may use the available unlabeled pool, often much larger than the acquired set. Following the argument introduced by Kraskov et al. [54], the key insight is to notice that Equation 4 holds for any value of and it does not require fixed & over different particles for entropy estimation (we provide more details in Appendix E Therefore, we can find the distance to the k-th nearest neighbor in the joint space spanned by the pool and the acquired set and map the corresponding neighbor (denoted as nx.,,) in Xtrain to estimate the marginal entropy. This results in the KSG marginal entropy estimator but repurposed to our setting:
HxSG(X)= = dx N
N log (1) + log Udx + (NN) N
N (nx,, (2) 1),
(10)
where (i) is now computed in the joint space and vxx,-
(i) is the number of points in Derein whose distance x; is less than Dz (i)/2. Figure 3 (b) illustrates the data selection by following this alternative estimation, leading to more diversity in the feature space.
Implementation. Firstly, we simplify the entropy term by dropping the constant terms with respect to xz:
1 - (x) dx
arg max (ar waa EPol!
((()U(x))
arg max log D(x) (m.an (r)
(1))
Equation| 11 acquire points by computing D(x) (based on the kNN distance) and the counter n.,. related to prompt x only. Furthermore, as ((x) accounts for the full dataset in the KSG estimator, it does not change over training. Hence, we may compute it offline once, and potentially scale to very large datasets (55). Lastly, BAL-PM acquisition scheme builds a batch of data by successively selecting points following Equation 5 Crucially, while BAL-PM keeps the preference model uncertainty estimates the same over the batch. it updates the entropy term after in-batch iteration. This operation boils down updating the counter X. lightweight operation. In Algorithm [ we present the pseudocode for BAL-PM. Appendix H further describes its computational cost.
5 Experiments and Discussion
In this Section, we aim to evaluate how BAL-PM performs in Active Preference Modeling. Our central hypothesis is that leveraging the information available the feature space spanned by the

---

Log Likelihood
Log Likelihood 1
-0.63
-0.61
0.62
-33 fewer samplesi
-0.66
69% fewer samples
-0.63
Random Sampling
Random Sampling
BALD
BALD
BAL-PM (ours)
BAL-PM (ours)
-0.69
2000
8000
12000
16000
20000
24000
2000
8000
12000 16000 20000 24000
Acquired Data
Acquired Data
(a) Reddit TL;DR (Test)
(b) CNN/DM Dataset (OOD)
Figure 4: Comparison with baseline methods in Active Preference Modeling. BAL-PM consider- ably reduces the number of samples required for preference modeling, achieving 33% and 68 % of reduction the Reddit TL;DR test split and CNN/DM News datasets, respectively. The shaded area corresponds to the standard error computed over five seeds.
base LLM - task-agnostic source of epistemic uncertainty addresses the problem of acquiring redundant samples, natural pathology of relying task-dependent epistemic uncertainty estimators designed for single-point acquisition schemes. BAL-PM, our proposed stochastic acquisition policy, promotes this diversity by maximizing the entropy of the acquired prompt distribution, besides selecting points for which the preference model presents high epistemic uncertainty.
Experimental Setup. We consider pool-based active learning setup. Each experiment consists of several acquisition cycles, where each iteration performs a batch acquisition in the currently available pool. The training set starts with the size of acquired batch and leaves the remaining data for the pool set. Following previous works [3] [4] we reinitialize the model after each acquisition to decorrelate subsequent acquisitions. We train the ensemble of adapters previously acquired data and employ early stopping based on the Log-Likelihood of held-out validation set. We evaluate the preference model after each acquisition loop and report the average Log-Likelihood of the test sets. Appendix G discusses why the test average Log-Likelihood is a proper metric for Preference Modeling. Finally, Appendix Cedetails hyperparameters and tuning methodology used in this work
Model Architecture. As described in Section we employ an ensemble of adapters on top of base LLM. Each adapter is multi-layer perceptron with non-linear activations. In most experiments, the base LLM is a 7-billion parameter model, although we also employed 70-billion and 140-billion parameter ones when analyzing the effect of scaling the base LLM. All considered models are only unsupervised pre-trained and have not undergone any preference fine-tuning.
Datasets. Following previous work 200, we considered prompts from the Reddit TL;DR dataset of Reddit posts (25) and the CNN/DM News dataset (26). We leverage the generated completions and human feedback collected by Stiennon et al. 11. The Reddit dataset contains train/eval/test splits, and we adopt the train split (92,858 points) for the pool and training sets, the eval split (33,083 points) for validation, and report results in the test set (50,719 points). The CNN/DM dataset contains single split (2,284 points), and we use for the Out-tt-Distiibttion (OOD) evaluation.
Comparison Methods. We considered Random Sampling and BALD 23 as baselines. BALD selects points based on the utility function of Equation 3 and equivalent the acquisition function used by Gleave and Irving (20]. We also compared BAL-PM with other stochastic acquisition policies
[22], namely SoftmaxBALDD. SoftRankBAL.D, and PowerBALD. We refer to Kirsch et al. [22 for a detailed description of these methods.
5.1 Experiments
We highlight and analyze the following questions to evaluate hypothesis and proposed method.
We release code httttt::/gtttee http:////ghthhb...m//ccccc.aaaat hct:://ccceeiiaao///LLLLP.P
7

---

Log Likelihood
Log Likelihood 1
-0.61
-0.64
0.62
-0.63
SoftmaXBALD
SoftmaxBALD
SoftRankBALD
SoftRankBALD
BAL-PM (ours)
BAL-PM (ours)
PowerBALD
PowerBALD
-0.66
2000
8000
12000
16000
20000
24000
4000
8000
12000
16000
20000
24000
Acquired Data
Acquired Data
(a) Reddit TL;DR (Test)
(b) CNN/DM Dataset (OOD)
Figure 5: Comparison with Bayesian stochastic acquisition policies for Active Preference
Modeling. BAL-PM consistently outperforms other policies in Test and OOD settings.
Does BAL-PM reduce the volume of feedback required for Preference Modeling? We start evaluating how BAL-PM performs against standard random acquisition and BALD, presented in Figure 4 BAL-PM considerably reduces the volume of data required to learn the preference model. Particularly compared with random sampling, it reduces the number of required samples in 33% for the Reddit TL:DR dataset and 68% for the out-of-distribution setting of CNN/DM News dataset, representing substantial reduction in the human feedback needed. BALD does present any benefits over random sampling in the TL:DR dataset, which aligns with previous work (20). Interestingly, BALD also presents an interesting improvement over random sampling in the OOD setting, but BAL-PM consistently outperforms BALD with more data.
How does BAL-PM compare with other stochastic acquisition policies? Next, analyze BAL- PM in comparison with other Bayesian stochastic acquisition policies. These policies address the acquisition of redundant samples by relying sampling points from the distribution determined by the task-dependent epistemic uncertainty scores. BAL-PM consistently surpasses all variations in both datasets, suggesting that leveraging the information available in the prompt feature space as task-agnostic source of epistemic uncertainty - is more effective in encouraging diversity for batch acquisition in the considered setting.
Does BAL-PM encourage diversity and prevent the acquisition of redundant samples? We evaluate the exploration approach of the considered methods by analyzing the statistics of the acquired prompt distribution, particularly the number of unique prompts over the course of training.
Figure 6 presents three different perspectives on the acquired distribution. On the left, it presents the number of unique acquired prompts over learning, which indicates diversity the training set. much faster rate than random sampling and BALD. Naturally, this rate saturates when the selection exhausts the number of distinct prompts available in the pool (approximately 14,000). The rate is also not equivalent to the data acquisition rate since BAL-PM does not simply select different prompts but also prioritizes points with high epistemic uncertainty.
BAL-PM selects new prompts
The middle plot shows the ratio of unique prompts in each active learning loop, and BAL-PM acquires batches with all distinct prompts during almost the whole training. BALD only maintains rate of 70%, which means substantial number of duplicated prompts. In Appendix K we present the first batch sampled by BALD and BAL-PM for a qualitative analysis. Lastly, the plot on the right shows the ratio of unique prompts across all training. While random sampling presents a high unique prompt ratio each separate batch, it consistently samples duplicated prompts throughout learning. In contrast, BAL-PMM maintains high ratio of unique prompts during most of the training. Again, this rate progressively decays as BAL-PM exhausts the pool of different prompts and due to the influence of the epistemic uncertainty prioritizing particular prompt-completion pairs.
How does BAL-PM scale to larger LLMs? As highlighted in Section 4 ouur design choices allow us to scale our experiment for very large base LLMS in a single GPU setting. We investigate the effect of scaling the base LLM in BAL-PM performance, considering 70-billion and 140-billion parameter models in their 4-bit quantized versions. Naturally, the preference model performance improves substantially against the 7-billion parameter model. More interestingly. BAL-PM presents similar

---

Unique Acquired Prampts
Unique Prompts Ratio Acquired Batch
Prompts Ratio Acquired
14000
1.0
1.0
0.9
10000
0.8
5000
0.7
0.4
0.5
8000 16000
24000
8000
16000
24000
0
8000 16000 Acquired Data
24000
Acquired
Acquired Data
Random Sampling
BALD
BAL-PM (ours)
Figure 6: Statistics of acquired prompt distribution. We present the total number of unique acquired prompts (left), the ratio of unique acquired prompts per active learning loop (middle), and the ratio of unique acquired prompts over training. BAL-PM consistently acquires novel prompts and encourages diversity in each acquired batch and the full training set.
Log Likelihood
Log Likelihood
-0.59
-0.60
-0.61
-0.61
31% fewer samp
325 fower samples
0.62
-0.63
Random Sampling
Random Sampling
BALD
-0.55
BALD
BAL PM (ours)
BAL PM (eurs)
-0.64
2000
8000
12000
16000
20000
24000
2000
8000
12000
16000
20000
24000
Acquired Data
Acquired Data
(a) 70b Parameter Model
(b) 140b Parameter Model
Figure 7: The effect of scaling the base LLM. We analyzed how increasing the size of the base LLM affects BAL-PM performance in the Reddit TL:DR dataset. We considered (a) 70-billion parameter model and (b) 140-billion parameter model. Interestingly, we find approximately the same gains (31 %-33% reduction of required samples) across all models.
gains across all scales, with around 31 31% reduction of required samples compared to random sampling. In contrast, BALD still does not present benefits over random sampling, suggesting that the scale of the base LLM is not the prevailing factor for its negative result.
Ablations and Further Analysis. We conduct ablation studies in the key components of the proposed method in Appendix [ More concretely, we ablate the components of the objective to show that both preference model epistemic uncertainty and entropy scores play relevant role in BAL-PM. We also ablate the type uncertainty and the employed entropy estimator. Furthermore, we conduct further empirical analysis in Appendix Fo investigate how each component of Equation ( contributes to the data selection, and conduct robustness analysis for the 3 hyperparameter in Appendix ! Lastly, we provide comparison with additional data selection baselines in Appendix[]
Closing Remarks
In this work, we present BAL-PM, Bayesian Active Learning method for Preference Modeling in Language Models. BAL-PM is stochastic acquisition policy that selects points for which the preference model presents high epistemic uncertainty and also maximizes the entropy of the acquired prompt distribution. We show that leveraging the information available on the feature space spanned by the base LLM via this entropy term has a crucial role preventing the acquisition of redundant samples. BAL-PM substantially reduces the volume of feedback required for Preference Modeling and outperforms existing Bayesian stochastic acquisition policies. also seales for very large L.LMs and effecttvely balances the contribution of both considered sources of uncertainty.

---

Limitations, Despite its encouraging results, BAL-PM presents some limitations. For instance, it heavily relies on the quality of the feature representations provided by the base LLM. Particularly. it might be subject to the Noisy-TV problem [56] and provide high-entropy scores to nonsensical prompts if those are spread in the representation space rather than collapsed into a single region.
Fortunately, we expect this limitation to be progressively addressed by better LLMs.
Future Work may evaluate BAL-PM larger preference datasets with millions or billions of data points. Another direction analyzes how the learned models perform in the Preference Optimization setting. Lastly, future work may extend BAL-PM to consider recent prediction- orrientd methods of epistemic uncertainty estimation 57 in contrast to parameter- sarameter-based methods such as BALD.
Acknowledgments and Disclosure of Funding
We thank Jannik Kossen for the insightful discussions in the early stages of this project. We also thank the reviewers for providing insightful feedback. Luckeciano C. Melo acknowledges funding from the Air Foree Office of Scientific Research (AFOSR) European Office of Aerospace Research Development (EOARD) under grant number FA8655-211-17701
References
[1] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems. neurrps Paper.
volume 33, pages 3008 3021. Curran Associates, Inc.. 2020, URL cc/ papppppfilllee
[2] Long Ouyang, Jeffrey Wu. Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35. pages 27730-27744. Curran Associates, Inc., 2022. httpstss:://///ett h/ttooeep///ooosiiingg... www.ooofeeen....ooomomo pdf
cc/paper
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Danny Hernandez, Tristan Hume, Scott
Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatficld-Dodee Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https:/// ///xxiiv wrt./ba//////eeee.
abs/ab///// http:/2244.05555
[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lec, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta. Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoging Ellen Tan, Binh Tang, Ross Taylor. Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models, 2023.
[5] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. In Thirth-sevnnthh Conference on Neural Information Processing Systems, 2023. MtlL/httss://peeeeeeee wwwp:/ennt//rrssa
[6] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu. Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909v:1909.08800 2019. https:/: //attps://arxiv...m org/ abs/aa/s/.9.. 1000b//9990099933333
10

---

[7] Stephen Casper. Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang. Samuel Marks, Charbel- Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan. Max Nadcau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadficld-Mencll Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL Survey Certification.
https://openrevier wwt.//orurridddb222R......
[8] Michael Bailey, Saeed Moayedpour, Ruijiang Li. Alejandro Corrochano- Navarroo, Alexander Kötter, Lorenzo Kogler-Anele, Saleh Riahi, Christoph Grebner, Gerhard Hessler, Hans Matter. Mare Bianciotto, Pablo Mas, Ziv Bar-Joseph, and Sven Jager. Deep batch active learning for drug discovery. July 2023. doi: 10.1 101000023.0..665550200
URL http://///xxxxx.m doi org/ 10. 11011112122222 07. 26. 550653
[9] Steven C. H. Hoi, Rong Jin, Jianke Zhu, and Michael R. Lyu. Batch mode active learning and its application to medical image classification. In Proceedings of the 23nd International Conference on Machine Learning, ICML, '06, page 417-424, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/1143844.1 143897. URL httllhtts::///t
org/ /0/g 0144/114114444 004443844.1148888 1143897
[10] Meta Llama Team. Introducing Meta Llama 3: The most capable openly available LLM to date al cm.meta.com https://ai. meta. com/ homm/blogh bwtpb//lo///////eeeeeeee/ee 1lama- 3/| [Accessed 13-05-2024).
[11] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36. pages 30039-30069, Curran As- sociates, Inc., 2023. URL https::/ppoooeeedtt http:///....t.///////oooooooooon (ccT7006eb300877fd000baaaa----b3 Confereeeee .ddf
[12] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrevi.e nett/forumsSddddejtWWaaaa.p
[13] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In H. Wallach, H. Larochelle, A. Beygelzimer. d'Alché-Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019, URL httt::::/ppoccediing neur cctppppp.cppppp...pppppp hcc//paper_files/ppppereet ftllss/paper/219//11.. 71772666662224500aaa226666663363 Papeeee-----pPP..p.. Paper.
[14] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 1183-1192. JMLR.org. 2017.
[15] Andreas Kirsch. Advanced deep active learning and data subset selection: unifying principles with information-theorry intuitions. PhD thesis, 2023. URL https:///oraa.... 3799959f 1119959f--139-110
ottp:::://rra.o....... ox. tttx///jjctt//// /
97744800775
[16] David J. C. MacKay, Information-based objective functions for active data selection. Neural Computation, 4(4):590-66 604, 1992. doi: 10. 1661/cc// www.e/ccoo........ 1995.4.4.590.
[17] Aditya Siddhant and Zachary C. Lipton. Deep Bayesian active learning for natural language processing: Results of large-scale empirical study. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun 'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods Natural language Processing, pages 2904-2909, Brussels, Belgium, Nctober-November 2018. Association for Computational Linguistics. doi: 18.18653/v 1/D18- 1-D18-118 1318 URL httpt:s:saclaattholloooooo oww//oo-g/////.eooo/eooo
[18] Biraja Ghoshal, Stephen Swift, and Allan Tucker. Bayesian deep active learning for medical image analysis. In Allan Tucker, Pedro Henriques Abreu, Jaime Cardoso, Pedro Pereira Rodrigues, and David Riaño, editors, Artificial Intelligence in Medicine, pages 36-42, Cham, 2021. Springer International Publishing. ISBN 978-3-030-77211 -721112.---6
[19] Jianxiang Feng, Jongseok Lee, Maximilian Durner, and Rudolph Triebel. Bayesian active learning for sim-to-real robotic perception. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10820-10827, 2022. doi: 00.10999000777111222229.2.
[20] Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models, 2022. URL
ftttps///axivv org/ htwp////......ospp 07472
11

---

[21] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Reprexentations, 2023. WRLLittps//opperrriies wwt.://tww.oooonnddnooooo httppppr/ppppprrreeeeeeefffffftr
[22] Andreas Kirsch, Sebastian Farquhar, Parmida Atighchchian, Andrew Jesson, Frédéric Branchaud- Sranchaud-Charron, and Yarin Gal. Stochastic batch acquisition: A simple baseline for deep active learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
wtw.a/rrwwwee..eeeoouiii..i.i... net/forum?id
vcHwQyMBjj Expert Certification.
[23] D. V. Lindley. On a Measure of the Information Provided by an Experiment. The Annals of Mathematical Statistics, 27( - 1005, 1956. doi: 10. /114///////177///0000 URL httpp:///doi.. org/10. 1214/ aoms/ 1
[24] Jishnu Mukhoti, Andreas Kirsch, Joost R. van Amersfoort, Philip H. S. Tom, and Yarin Gal. Deep deterministic uncertainty: new simple baseline. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 24384 24394, 2021. htLphtts:://appp www.sticsmcooooooeeeooeee org/
1D:
[25] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL:DR: Mining Reddit to learn automatic summarization. Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18663/1/W11-45500 URL https
//aclanthology org/ S0///1172000 W11--40888
[26] Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS'' 15, page 1693-1701 1701, Cambridge, MA, USA, 2015. MIT Press.
[27] Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference leamning, 2011.
[28] A. Gilad Kusne, Heshan Yu, Changming Wu, Huairuo Zhang. Jason Hattrick- Simperss. Brian DeCost, Suchismita Sarker, Corey Oses, Cormac Toher, Stefano Curtarolo, Albert V. Davydov, Ritesh Agarwal, Leonid A. Bendersky, Mo Li, Apurva Mehta, and Ichiro Takeuchi. On-the-fly closed-loop autonomous materials discovery via bayesian active learning. Nature Communications, 1 11(5966), 2020.
[29] Christopher Tosh, Mauricio Tec, Jessica White, Jeffrey Quinn, Glorymar Ibanez Sanchez, Paul Calder, Andrew L. Kung. Filemon Dela Cruz, and Wesley Tansey. A bayesian active learning platform for scalable combination drug screens. bioRxiv, 2023. doi: 10. 101/2002,,22 18.52.57 URL https
7/uuw. http://wwwr/.ooooeee////////////
12. 18. 572245
[30] Johannes Fümkrraz and Eyke Hüllermeier. Pairwise preference learning and ranking. Europeon Confer- Corg/Coruuulii ::: 4735672
ence on Machine Learning, 2003. URL h/tptp:://ap....maatttteee
[31] Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the
22nd International Conference on Machine Learning, ICMI '05, page 137-144, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805. doi: 10. 10. 10235.0921,.,.002233 URL
https://doi. org/ 10. 01444//11111111111
[32] Viraj Mchta, Vikramject Das, Ojash Ncopane, Yijia Dai, Ilija Bogunovic, Jeff' Schneider, and Willic Neiswanger. Sample efficient reinforcement lcarning from human foedback via active exploration, 2024. Rttlstttp://ooeeeriiii
[33] Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement learning from human feedhack with active queries, 2024.
[34] Anonymous. DUO: Diverse, uncertainty-aware, wwwp://wwwwwaaaooooloiiciccycooe query generation and selection for reinforcement learning from human feedback. 2024. URL
[35] Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Provably sample efficient rihf active preference optimization, 2024.
[36] Vikranth Reddy Dwaracherla, Scyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for 1ms. ArXiv, abs/2402.00396, 2024. URL httpps:/appt semanticscholar. .rg/
Corpus 1D:23694:26734948
12

---

[37] lan Osband. Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, MORTEZA IBRAHIMI Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine. editors, Ad- vances in Neural Information Processing Systems, volume 36, pages 2823. Curran Asso- ciates, Inc., 2023. URL https::/proceeeeee weweediinnsnuuuiiiinnee neur ctpuurips.cc///e wwtpppp..ccpppp.pppppppppp /tt.s///pppa////2/////. (rtba696be66014e080333cdd Pawp:rrrr23--------------r Pwwer-Confeeeeeeeeeeeeeennae Conference..eecoeoo pdr
[38] Janis Postels, Hermann Blum, César Cadena, Roland Y. Siegwart, Luc Van Gool, Federico Tombari. Quantifying aleatoric and epistemic uncertainty using density estimation in latent space. ArXiv, abs/2012.03082, 2020. URL https://// semanticscholar org/CorpusID
[39] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan tection. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 21464-21475. Curran Asso- ciates, Inc., 2020. URL httt::://proceedingg neun www.reerrccccccceoo.ee.e wwtpppp..cpppp..ppppppppp 0atp22268000aaa22211777996666666
Energy-based oul-ofddistrbuttion de-
[40]
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9690 9700. PMLR, 13-18 Jul 2020. BRLLtttp::/proceeddin mlr alr.pre press/vi19) van amersfoorte www...rtta0.... html
[41] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th Interna- tional Conference on Machine Learning. volume 97 Proceedings of Machine Learning Research, pages 2681-2699 PMLR, 09-15 Jun 2019. Btthhttps: pttp:////coceeeeesseeeee mlr. waa...press/eee press/v97/ wtw////a//aaaa aaw.nn9a.....aaaaaaaaaaaaaaaaaaa html
mmr.
[42] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL httpt:tsp//// htttt:::/pprcceeiinng
mlr. prw.prresssssee Trress/v00/haarae haao://aaaaooaaaaaa html
[43] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. In Thirty-seventh Conference on Neural Informaion Processing Systems, 2023. httpsstts/////enrreeee
[44] Hao Liu and Pieter Abheel. Behavior from the void: Unsupervised active pre-training. In M. Ranzato, A. Beygelzimer, Y. Dauphin,
P.S. Liang, and J. Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 18459-18473. Curran Asso- ciates, Inc., 2021. URL httpp::/ppocceeeeeee 990faaa5aaa166777600111111222222
neur ips. wttppppp.ppppppppppppppppppppppe htilee//apprr222//1112
[45] Aadirupa Saha. Optimal algorithms for stochastic contextual preference bandits. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 30050 30062. Curran Asso- ciates, Inc., 2021. URL htttps://procceeiings heurips wwwprips.cppppe__tiiee http:/pppe..llleeeppp/////////// facr:555aaBB2eeee7772222eeeeee
[46] Siddartha Y. Ramamohan, Arun Rajkumar, Shivani Agarwal, and Shivani Agarwal. Dueling bandits: Be- yond condorcet winners general tournament solutions. In D. Lee. M. Sugiyama, U. Luxburg. Guyon, and Garnett, editors, Advances Neural Information Processing Systems. volume 29, Curran sociates, Inc., 2016. URL https: http::::/p//cceeeeeseee neurips. wtt.apps.ccpppp...ppp http://ppr_..llee//ppprrr/////// cicb3cdcSacc14466e//o 7tee770001117765660 4560c026-66666 Paper. .pf
[47] Ralph Allan Bradicy and Milton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324- 345, 1952. ISSN 00063444, URL http:///: jstor org/ htt///table///e 2tttable23330229 2334029
[48] L. F. Kozachenko and N. Leonenko. Sample estimate of the entropy of random vector. Problemy Peredachi Inform, 1987. URL
[49] Kevin P. Murphy, Probabilistic Machine Learning: Advanced Topics MIT Press, 2023. URL http: //probml. github. 1o/b2
13

---

[50] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and II Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 4697-4708. Curran Asso- ciates, Inc., 2020, URL htattp://proocediigg.. neurips. wttp:pss.ccpppperratee http://pppr...lleaaaa//////////e 22216266666667723556f44444al.333 Papeeaaa---ppPrr.......
[51] Pavel Izmailov, Sharad Vikram, Matthew D Hoffmman, and Andrew Gordon Gordon Wilson. What bayesian neural network posteriors really like? In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4629 4640. PMLR, 18-24 Jul 2021. httphtttppop:oeeei//
mlr press/ press/v139) /
izmailov211 html
[52] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL htttp::/pproceeedingn neurips. meeurps..c/pappeeee http://ppppp..fleeeepeeeeeeee htt.le//pape///177////1/ pdf
[53] Sylvestre-Alvise Rebuffi, Andrea Vedaldi, and Hakan Bilen. Efficient parametrization of multi-domain deep neural networks. In 2018 IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 8119 8127, 2018. doi: 10.1109/CVPR.20 8.00877.
[54] Alexander Kraskov, Harald Stoegbauer, and Peter Grassherger. Estimating mutual information. Physical review E, 696))066133, 2004.
[55] Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. Spann: Highly-efficient billion-scale approximate nearest neighborhood search. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan, editors. Ad- vances in Neural Information Processing Systems, volume 34, pages 5199-5212. Curran Asso- ciates, Inc., 2021. URL
neuripe ipsrrpps...o httpp/pppe....lee/////////////// Paper.pd.
[56] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. WtRLitttts://pperreeet het/forum?id-Hi1
[57] Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. Prediction-orieee orientee bayesian active learning. In Francisco Ruiz, Jennifer Dy, and Jan An-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 Proceedings of Machine Learning Research, pages 7331-7348. PMLR, 25-27 Apr 2023. URL https: h/ppse//ppoceeding.. mlr, wwwpss/w0w//////eeeeeee wtt.///iic/oorre/m..... hhmml
[58] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison. Andreas Kopf. Ed- ward Yang, Zachary Vitioo Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and Gamett, editors, Advances Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL httpp: //ppoe/e///occeiing... hwwrise.erpppp.. wtcpippppccppppepeeeeee fileapperfileeeepeee///aa /019///////// (bc
pdf
[59] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Mai, Pierric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-ar natural language processing. 2020.
[60] David D. Lewis and William A. Gale. A sequential algorithm for training text classifiers. In Bruce W. Croft and C. J. van Rijsbergen, editors, SIGIR '94, pages 3-12, London, 1994. Springer London. ISBN 978- 978-1-4471-0 14441-99999
[61] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5753- 5763. PMLR, 18 24 Jul 2021. URL http::s:///occceiiing www.mmmmppeeeeeeeee wer.ppesss/vve 11113399//sse21... html html
14

---

[62] Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6405-6416, Hook, NY, USA, 2017, Curran Associates Inc. ISBN 9781510860964
[63] Yarin Gal Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-10555 1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URt/ttps: htttps://proceeedinggg mlr. www.prreee/eeee v48/ galll.llll.... html
[64] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835-10866. PMLR, 23-29 Jul 2023. URL.
https://proceeddings mlr. srees//v02// httpl//gg233hhh.... html
[65] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[66] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Eneko Agirre, Marianna Apidianaki. and Ivan Vulić, editors, Proceedings of Deep Learning Inside Out (DeeLIO 2022): The Workshop on Knowledge Extruction and Integration for Deep Learning Architectures, pages 100 114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: 10. N...8863//11222222e0
1.10. URL https
//aclanthology- .rrgey..rgg/ee. w02.. deelio-1.10
[67] Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10144- Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.1853/11/2023 h20.://.nnnnnnneeeee..... www.iinnn-eemnlplee eemngg---mmlp.67.. URL //aclanthology..... .wwggy..o.oooo............... 2023. Sindings-emmlp...
15

---

A Impact Statement
Preference fine-tuning has become erucial step in aligning LLMs toward human preferences and demonstrated a real-world impact in many open-source and production systems 2 I0] Nonetheless, collecting human feedback very expensive and coneuconsumnn, posing substantial bottleneck for further development of LLMs. In this work, we approach the problem of Active Preference Modeling, which aims to reduce volume of feedback required for learning preferences. We show that our proposed method, BAL-PM, requires 33% to 68% fewer labels in the human preference datasets considered. We believe that these results point out strong impact in the process of acquiring labels, and estimate economy of hundreds of thousands of dollars and months of labeling work the current scale of LLMs. This scenario represents faster cycles preference optimization, potentially leading to better-aligned and safer models. Therefore, we believe our work poses a relevant positive societal impact for the upcoming years.
Reproducibility Statement
Code Release. To ensure the reproducibility of research findings, release code PML-PM Our implementation is based on Py [58] and HuggingFace [59]. All baselines are available the released code.
Experiments Reproducibility. We detail our methodology in Section 4 and our experimental setup in Section 5 We provide all hyperparameters used in this work well the selection strategy in Appendix We plan to release all the raw experiment logs and feature datasets generated this work. For all experiments in this paper, we report the results five seeds with standard errors. For better visualization, we applied smoothing for the curves considering two past observations.
Datasets. All preference datasets are open-source and available online for academic use 1
Compute Resources. We execute all active learning experiments in single A100 GPU, and each experiment takes approximately one day. For the base LLM feature generation, we also use a single A100 GPU, taking few hours for the 7-billion parameter model and approximately four days for the 70-billion and 140-billion parameter models.
16

---

C Hyperparameters
In T we share all hyperparameters used in this work. We specifically performed hyperparam eter search on the entropy term parameters and baselines. The search strategy was a simple linear search on the options in Table considering each parameter in isolation. The selection followed the final performance on a held-out validation set. For baselines, we mostly considered the values presented in prior work [22]. For the proposed method, we also considered dx as a hyperparameter and found that smaller values often work better than using the dimensionality of the base LLM embeddings.
Hyperparameter
Value
Acquisition Batch Size
320
Initial Training Set Size
320
Initial Pool Size
92,000
Active Learning Iterations
75
Adapter Network Hidden Layers
[2048, 256]
Adapter Network Activation Function
tanh
7b Model Name
OpenHerrmes-255-Mstteee
70b Model Name
Llama3-70b
140b Model Name
Mixtral-8x22B--v.11
Early Stopping Patience
3
Training Batch Size
32
Learning Rate
3e-5
Learning Rate Scheduler
Cosine
Optimizer
AdamW
Entropy Term |
0.01
Entropy Term k
13
Entropy Term dx
1.0
SoftmaxBALD
10,000
SoftRankBAL.D 3
1.0
PowerBALD /
8.0
Table 1: Training Hyperparameters.
Hyperparameter
Search Space [0.0001, 0.001, 0.01, 0.1, 1.0]
Entropy Term 8
Entropy Term &
[1, 13, 19, 25]
Entropy Term dx
[4096, 2048, 1024, 256, 64, 32, 8, 4, 1, 0.5]
SoftmaxBALD
[0.25, 1.0, 2.0, 100.0, 1000.0, 5000.0, 10,000] [0.25, 1.0, 4.0, 8.0]
SoftRankBALD /
PowerBALD 3
0.25, 1.0, 2.0, 4.0, 8.0, 10.0, 12.0
Table 2: Hyperparameters search space.
17

---

D
Ablation Studies
This Section presents and discusses the results of the ablation studies, We focused on three different aspects: the components in the objective of Equation 5 the nature of the uncertainty considered; and the entropy estimator.
Objective Components. We considered three different versions for ablating components: BAL-PM (ours), which follows Equation [ exactly; version with No Uncertainty Score in the objective; and another version with No Entropy Score. Figure Shows the findings. In the datasets considered, both terms of the objective play crucial role in the performance of BAL-PM. Disregarding the entropy score fundamentally means solely following BALD, which acquires several redundant samples. On the other side, disregarding the uncertainty score prevents the learner from acquiring points where the model lacks information.
Log Likelihood
Log Likelihood
-0.64
No Uncerlainly Score
No Uncerlainly Score Entropy Score BAL-PM (aurs)
No Entropy Score
BAL-PM (ours)
2000
8000
16000 Acquired Data
12000
20000
24000
4000
8000
12000
16000 Acquired Data
20000
24000
(a) Reddit TL:DR (Test)
(b) CNN/DM Dataset
Figure 8:
The ablation study of the components in the BAL-PM objective. We considered BAL-PM, a version without the uncertainty score, and a version without the entropy score.
Type of Uncertainty. In Machine Learning, we identify two different sources of uncertainty: epistemic and aleatoric. Epistemic Uncertainty refers to the uncertainty in the parameters of the model, often due the lack of information from some regions of the parameter space. In contrast, aleatoric uncertainty refers to the uncertainty in the data, originating from the inherent noise of the data generation process. We reduce epistemic uncertainty by acquiring new data, while aleatoric is irreducible.
common practice in Active Learning is to select points based on high Predictive Uncertainty, which is often referred to as "Uncertainty Sampling" [60]. This type represents the total uncertainty, i.e., accounts for both epistemic and aleatoric sources. Therefore, we expect that following Predictive Uncertainty underperforms in datasets with high label noise, as the objective may favor points with high alcatoric uncertainty and low epistemic uncertainty.
Figure 9 compares using Predictive and Epistemic uncertainties the objective of Equation 5 Selecting points based epistemic uncertainty strongly outperforms the other variant, which aligns with the fact that preference datasets contain high levels of label noise as mentioned in Section I. the agreement rate among labelers is low, typically between 60% and 75%. This ablation also highlights the importance of a Bayesian preference model for epistemic uncertainty estimation.
Entropy Estimator. In Section 5 we argue for using the KSG entropy estimator (rather than the KL estimator) since it leverages the full dataset and better estimates the probability density in the feature space, leading to less biased entropy estimates. In this ablation, we compare both estimators to measure the impact of this design choice.
Figure 10 presents the results of this ablation. In the Reddit TL;DR dataset, the KSG estimator consistently outperforms the KL estimator, requiring approximately 20% fewer samples. In the OOD setting, both estimators performed equally. This is expected once that the available pool and training set does not provide support the regions of the feature space with out-s--iistiibttion samples.
18

---

Log Likellhood
Log Likelihood
-0.61
-0.64
0.63
-0.66
0.67
-0.68
Predictive Uncertainty Epistemic Uncertainty (ours)
Predictive Uncertainty
Epistemic Uncertainty (ours)
2000
8000
12000 16000 20000 24000
2000
8000
12000 16000 20000 24000
Acquired
Acquired Data
(a) Reddit TL:DR (Test)
(b) CNN/DM Dataset
Figure 9: The ablation study of the type of Uncertainty in the BAL-PM objective. Leveraging Epistemic Uncertainty substantially surpasses Predictive Uncertainty since disregards the effect of the high Alcatoric Uncertainty from preference datasets.
Log Likelihood
Log Likelihoan f
0.64
0.62
15%
Estimator
KL Estimator
KSG Estimator (ours)
KSG Estimator (ours)
-0.63
-0.65
4000
8000
12000
16000
20000
24000
5000
8000
12000
16000
20000
24000
Acquired
Acquired Data
(a) Reddit TL:DR (Test)
(b) CNN/DM Dataset
Figure 10: The ablation study of the type of entropy estimator in the BAL-PM objective. Using the KSG estimator requires approximately 20% fewer samples than the KL estimator in the Reddit TL.DR dataset.
19

---

E KL Entropy Estimator: Review and Assumptions
In this Section, review the derivation of the KL. entropy estimator and highlight the main assump- tions and how they impacted the design choices for BAL-PM. We mostly follow the derivation from Kraskov al. [54].
We start defining X as a continuous random variable in a vector space where the Euclidean norm || x=1 is well-defined (x: and are two realizations of X). Let (u(x) represent the density of X over this vector space. The Shannon entropy defined as:
-f u(z)) log u(z)dx.
= = -Eu -
(12)
To build an estimator, we can approximate Equation 12 via Monte-Carte sampling:
N
1 N
H(X)=
Σ
log p(x)),
(13)
where N is the number of samples for approximation and (ax)) an estimate of the density X. The goal of kNN-based entropy estimators primarily to provide a good approximation for the density. For this matter, first define probability distribution Px (c) for the distance between any realization 24 and its k-nearest Neighbor. We start highlighting the first assumption:
Assumption 1. The probability Ps(c)de is equal to the chance of existing one point such that || - x= < c/2, k other points with smaller distances, and N k - points with larger distances.
Following this assumption we can obtain the following trinomial distribution:
Pa(c)
= k:
N k
dp.(c)
dc P. ' (1
pc.) N-111
(14)
where pt(e) is the probability mass of the e-ball centered in x4. The expectation of (:(e) is:
I 00
Pa(e) = k (N (N-1) (x Pi 21 N logp: (15)
Bllogp;(e))
= -
where yb denotes the digamma function. We now highlight the second assumption: Assumption 2. The density u(z) is constant in the e-ball.
Assumption 2 allows us to approximate pc(e) s cae u(x;). where d is the dimension of æ and ca is the volume of the d-dimensional unit ball. Using Equation in this approximation and rearranging terms, we have:
log (xx)) = (ok) - (NN - dE [log d log Ca-
(16)
Finally, using this estimator in Equation T33 we obtain the KI. entropy estimator in Equation
Remarks. Now, analyze how this derivation and assumptions impact entropy estimator. First, Assumption [ models the probability based on the choice of k. For the low-data regime (i.c., N is small), this could lead considerably large e-balls where the Assumption 2 does not hold, and, therefore, it is not a good approximation. Thus, naively applying the KL estimator in the acquired training set may lead strongly biased entropy estimates.
Secondly, in Section 4 we raise the key insight that Equation 4 holds for any value of k. and it does not require fixed & over different particles for entropy estimation. Indeed, the density estimation (ix:)) is estimated for each particle æ; in isolation (Equation T6p. Therefore, we may choose different & for each particle to ensure that Assumptions and 2 valid.
20

---

BAL-PM Objective Empirical Analysis
Balancing Dekendentt and Task-Agnostic Epistemie Uncertainty for Active Learning. Since considering the information in the feature space is crucial for Active Preference Modeling, relevant question arises: how should an algorithm balance the contributions between the Bayesian preference model epistemic uncertainty and the prompt feature space uncertainty? Excessive reliance on the task-dependent term leads to acquiring redundant points. Similarly, the exacerbated contribution of the task-agnostic term prevents the acquisition of the points that reduce the uncertainty in the preference model. Thus, we investigate how BAL-PM balances these two terms over active learning. In Figure [T] show the ratio of the entropy and preference model epistemic uncertainty scores in the first selected point of each acquired batch. Interestingly, BAL-PM automatically adjusts the contribution of each term. It progressively decays and converges the influence of the entropy score (task-age aask-agnostic source) as the novelty in the prompt feature space reduces due to the acquisition of new points. Similarly, it increases the relevance of the preference model uncertainty estimates (task- dependent source). positive downstream effect that BAL-PM switches to a more lask-dependent selection as it improves the Bayesian model and, consequently. its epistemic uncertainty estimates.
Score Ratio
1.0
0.8
0.6
0.4
0.2
Ú(x, V1, Y2)
7((x U {x})
0.0
0
8000
16000 Acquired Data
24000
Figure 11: Ratio of entropy and preference model uncertainty scores. This plot represents the normalized contributions from the terms of Equation 5 on the first selected point of each batch. BAL-PM automatically adjusts the contribution based on the information available in the pool set.
21

---

G Is Log-Likelihood a proper performance measure for Preference Modeling?
In this Section, we argue why the Average Log-Likelihood on the test set is good performance
measure for Preference Modeling. Given test set Diest {(x, yl, y/, y1 1/2] ]aa and the learned preference model po (y
> 1/2 x. y1, (2), the average Log- bog-Likelihood given by:
LL(Dtest 0) (a
waash wa.nn WD [log (1 > 1/2 2, M1 11. y2)]
(17)
Equation 17 is exactly the objective maximized in standard binary classification (or, equally, the minimization of the negative Likeliikdd loss) but computed over the test data. In other words, this is the negative "test loss".
Average LL is typical metric in the Active Learning and Uncertainty Quantification literature [61- 63.. For Preference Modeling, is very relevant as LL directly accounts for the preference strength to rank models: given triple (Ir, y1, 1/2 where all raters agree that y/ is preferable over y/, LL allows us to measure that model A predicting Pa(y1 > 1/2 x,, V/ 12) = 0.9, (LL -0.1) better (in that data point) than another model B predicting P3(11 > 1/2 xr y1 .22 0.6 (LL R -.5).. Accuracy would provide an equal score for both models since it only accounts for the binarized prediction. LL provides more "fine- sine-grained" measure.
Another crucial point is that LL factors in the aleatoric uncertainty in the label-generating process. For instance, in scenario where only 70% of the raters agree that is preferable, LL better ranks models whose predictions are closer to p = 0.7, respecting the ground truth preference strength, which is not possible with accuracy.
G.1 Do the models better ranked by Average Log Likelihood (LL) lead to better fine-tuned policies?
In the context Preference Modeling, tine-tuning LM policies currently a very relevant downstream task. The Preference Modeling optimization objective and model selection protocol adopted in this work follow exactly the prior influential work on the topic 6 I. which provides evidence that better preference models (in terms of validation loss) lead to improved downstream policies. Thus, expect our models to behave similarly under the same conditions.
Preference Log Likelihood and Finetuned Policy Win Rate
1.0
Preference Log Likelihood vs Finetuned Policy Win Rate
Points
1..0
-0.1
0.9
Regression on Line
0.9
Log Likellhood
Policy Win Rate
0.8
-0.3
0.7
Win Rate
0.7
0.6
-0.5
0.5
0.5
Log Likelihood
-0.7
Policy Win Rate
-0.3
0.4
0.3
0
50
100
150
200
Acquired Data
-0.7
-0.6
-0.5
Log Likellhood
-0.4
-0.3
-0.2
-0.1
(a)
(b)
Figure 12: The Relationship hetween the Test Log-Liiel of Preference Model and the Performance of the corresponding fine-tuned Policy. We show that, under simple conditions, there a strong correlation between these two performance measures.
As additional evidence, we empirically illustrate the relationship between Log-Likelihood and policy performance on simplified setup (Figure [2]. Here, prompts x and completions are real numbers in [0, 1). The ground-truth reward function given by Gaussian density function r(I,y) = (x + y In this setup, we progressively increase the training set size (the x-axis in Figure 12a) at which train the preference models. This process generates different models with increasing levels of
1.0, 0.4), and true preferences follow the Bradley-Terry model.
22

---

test-set average Likeeiiiddoood. Then, similar to [64,, we optimize the base policy via a Best-of-N optimizer by leveraging each of these learned preference models. Finally, we report the rate at which the fine-tuned policy completion is preferable over the base policy completion according to the ground truth reward model ("win rate"). Although simple, this setting allows us to bypass several optimization and distributional challenges and solely focus on evaluating the relationship between average Log-Likeli and the performance of the fine-tuned policy. Figure] reports the Log-Likelihood (red) and the win rate against the base policy (blue). Figure 12b directly plots both measures and fits a regression line. We observe a strong correlation, which aligns with our point: a higher test-set average Log-Likelihood means that the preference model is better at predicting the ground truth preferences, assigning higher rewards for better completions, and, therefore, improving fine-tuned policies that maximize such reward scores.
H BAL-PM Computational Cost Description
In this section, we describe the computational cost of executing BAL-PM. We argue that computational tractability is one of the main contributions of our method. We start by providing some context: our work focuses on (Bayesian) Active Learning, which is naturally more computationally demanding than simply training predictive models. This because we require models that express epistemic uncertainty to acquire informative labels for efficient training. This also requires models to constantly update their uncertainties based on the new data via re-training. The key that Active Learning reduces the number of labels needed to train a better model, which considerably overcomes the extra computational cost. Labeling is significantly more expensive and laborious.
As described in Section [ Preference Modeling in LLMs requires batch acquisition - it is impossible request the label of single point, re-train the model, and repeat this process. Still, tractable methods rely on these Langlepoiin acquisition objectives. Thus, what BAL-PM does computationally is to replace B model re-trainings per acquired batch with computing entropy estimates (considerably cheaper, as explained below). B is the batch size, and we set B = 320 in our experiments.
BAL-PM does not require training inference LLMs during the active learning loops. This considerably reduces the computational cost and allows us to scale up to 140b models in a single A100 GPU. Comparatively, even fully fine-tuning several A 100 GPUs. LoRA methods (65) also require new LLM inferences for every model update, while BAL-PM only requires single time.
bo-parmmeter model currently requires least
The computation of BAL-PM has three pieces: offline processing (LLM inference and kNN computa- tion), updating adapters, and entropy estimation. LLM inference is done only once before Active Learning, which is the minimum for LLM adoption. Furthermore, we may compute the features used for the preference model and sentropy estimation in the same forward pass: every prompt- completion input concatenates promp//completion texts. Thus, we can extract prompt features as the last layer embedding right after the last prompt token and the prompt-completion features right after the completion's ' last token. Hence, there is no extra cost to extract features for entropy estimation. The cost of updating adapters is minimal: it boils down to updating MLPs with two hidden layers, which is reasonably cheap for LLM research. Lastly, the entropy estimation only requires computing the di-gamma function (Equation 11) in the pool.
23

---

I 3 Robustness Analysis
In this Section, we introduce a robustness analysis for the hyperparamettrr (Figure considering the values in the search space, described in T As presented in Equation this hyperparameter balances the effect of the epistemic uncertainty and entropy scores.
Log Likellhood
Log Likelihood
-0.63
-0.61
-0.64
-0662
1.100
8 - 1.0
-0.63
=
0.65
8 0
==0,1
8 0=0.01
A 0.000
0 = 0.001
= 0.0001
0 = 0.0001
8000
16000 Acquired Data
12000
20000
24000
4000
8000
Acquired Data
(a) Reddit TL;DR (Test)
(b) CNN/DM Dataset (OOD)
Figure 13: 3 Robustness Analysis. We considered different scales of the 3 hyperparameter in the BAL-PM objective (Equation 5).
The impact of the choice is more noticeable when values are 100x greater/lower than the optimal choice. Values around 10x greater/lower still perform well, suggesting good room for choosing this byperparameter. Furthermore, we employed the same value of 3 across the different datasets and LLMs, suggesting robustness across different relevant dimensions. Crucially, B trades off the contribution two different terms. As such, provides a spectrum objectives and may recover the two extremes presented in the ablation of Figure 13 Naturally, different choices of 3 will F (i.e., the contribution of each term
change the uncertainty score ratio in Figure II on Appendix after convergence). Nevertheless. and most importantly, the behavior of the curves the entropy contribution progressively reducing and converging and the relevance of epistemic uncertainty estimates increasing should remain.
24

---

Further Baselines
In this section, we provide additional baselines for further comparison.
Log Likelihood
-0.1
J.1 Full Dataset Baseline
-0.6
We start by evaluating the performance a pref- erence model trained in the full dataset. Fig- ure [4 presents this result purple, with the shaded area representing the standard error com- puted across five seeds. BAL-PM achieves on- par performance to this baseline, although it only requires 24000 data points (the full dataset contains 92,858 points). This result is another strong evidence of the sample efficiency of our method.
T- 13% fewer samplesi
-0.63
Full Dalaset
Random Sampling
BALD
BAL-PM (ours)
8000
1200D
16000
Acquired Data
Figure 14: Comparison with Preference Model
trained on the full dataset.
J.2
Additional Data Selection Methods
Log Likelihood
Log Likelihood f
-0.61
-0.64
0.62
Entropy
Low Perpiexity
-0.67
BAL-PM (ours)
High Perplexity
Min
Perplexity MC-Dropout
MC-Drropuu
Low Perplexity (ours)
0.69
-0.68
2000
8000
12000 16000 20000 24000 Acquired Data
4000
8000
12000
16000
20000
24000
Acquired Data
(a) Reddit TL:DR (Test)
(b) CNN/DM Dataset (OOD)
Figure 15: Comparison with several additional baselines for Active Preference Modeling, These baselines focus on different notions of uncertainty and diversity for acquiring samples.
We considered the following additional baselines:
Entropy Minimizer: Inspired by Liu et al. [66], we consider an objective that, addition to selecting points with high epistemic uncertainty, also selects points that are semantically similar the current training points. This is equivalent to selecting points that increase the entropy the prompt distribution the least, thus the name "Entropy Minimizer". It serves as a check for our central hypothesis that entropy maximization leads to better batch active learning.
Entropy Minimizer:
Inspired by Gonen et 667, we consider an objective that selects points based on the perplexity of the base LLM. We consider two versions: one that chooses points with lower perplexity (Low Perplexity) and another with higher perplexity (High Perplexity). This is an interesting baseline since perplexity is equivalent to the predictive entropy of the token distribution. Therefore, it helps to analyze how much the base L.MM "knows what does not know" in terms of preference modeling.
Perplexity:
This method performs approximate Bayesian inference via executing dropout time generate different parameter hypotheses. Therefore, express epistemic uncertainty, which is used to select the most informative points.
MC Dropout 68
Latent Reward Uncertainty (LRU): This method computes a reward distribution over the data points by leveraging the latent reward model Iearned via the Gradley-Tert model.
25

---

Then, it selects extreme points (too high or too low rewards) as proxy for the uncertainty of the model.
Log Likellhood
Log Likelihood
-0.61
-0.63
-062
0.63
-0.65
MC-Dropout
MC-Dropout
MC: Dropout + Entropy
MC-Dropout Entrapy
BAL-PM (ours)
(ours)
LRU
URU Entropy
LAU Entropy
-0.68
2000
8000
12000 16000 20000 24000 Acquired Data
Acquired Data
(a) Reddit TL:DR (Test)
(b) CNN/DM Dataset (OOD)
Figure 16:
The effect of incorporating the entropy objective in uncertainty baselines. This shows how our proposed objective can also boost the performance of different baselines.
Figure 15 reports performances for both test and OOD sets. In both cases, BAL-PM outperforms additional baselines. In the sequence, MC-Dropout works best as the baseline that targets the epistemic uncertainty of Bayesian model. Unsurprisingly, Entropy Minimizer and Low Perplexity work worse since they target points with lower entropy. LRU presented mixed results, suggesting that the latent reward may not represent well the preference model's uncertainty. More interestingly, while these models can represent different uncertainties to seek informative points, they naturally cannot provide in-batch diversity they suffer from the same challenges BALD. In this perspective, the BAL-PM objective can also improve upon those methods, as we show in Figure 16 we combined MC- Dropout and LRU with our entropy term provide in-batch diversity, which consistently improved both methods across the datasets.
26

---

K Batch Samples
In this Section, we present some selected samples of the first acquired batch from BALD (Table 3 and BAL-PM (Table for comparison. We sorted the points alphabetically to highlight duplicated prompts. BALD consistently acquires duplicated points, sometimes sampling more than ten times the same prompt. In contrast, BAL-PM samples semantically diverse points with no duplicates.
BALD - Acquired Batch (Truncated Prompts)
A bit of backstory: I've been in only 4 real long term relationships in my past....
A bit of backstory: I've been in only 4 real long term relationships in my past....
A bit of backstory: I' I've been in only 4 real long term relationships in my past...
A few weeks ago my wife admitted to me that my best friend, (let's call him Marc...
A week ago I called off my relationship with my partner for a number of reasons,..
About a month ago my (23 F) boyfriend (26 M) of three and half years and got...
After 8 months my girlfriend decided to break up with me. Shes very nice girl For starters, its been awhile loseit, and missed you! Things have been crazzzy... For starters, its been awhile loseit, and I missed you! Things have been crazzzy... For starters, its been awhile loseit, and missed you! Things have been crazzzy... For starters, its been awhile loseit, and I missed you! Things have been crazzzy... Hello all I need some help regarding a friend of mine and a dream she had, well ...
Hello everyone. I amm a student at boarding school which means am away from m...
Hi all. I am using throwaway. am 29f and my boyfriend is 32m. We have been d... Hi all. I am using throwaway. I am 29f and my boyfriend is 32m. We have been d... Hi all. I am using a throwaway. am 29f and my boyfriend is 32m. We have been d... Hi first time user, and am dyslexic so please forgive any spelling errors. T...
I am 31 years old and currently live in New York. have been a professional tre...
I was sitting
bus and the seat beside me was empty.. A young nun walked do...
I work inside of a bread depot, and the drivers are effectively brokers, or our
I work inside of bread depot, and the drivers are effectively brokers, or
I work inside of a bread depot, and the drivers are effectively brokers, or our
I work inside of bread depot, and the drivers are effectively brokers, or
I work inside of a bread depot, and the drivers are effectively brokers, or our
I work inside of bread depot, and the drivers are effectively brokers, or our
I've been married to my husband for 3 years, it's been wonderful, I couldn't ask... I've been married to my husband for 3 years, it's been wonderful, I couldn't ask... I've been married to my husband for 3 years, it's been wonderful, I couldn't 't ask...
I've been married to my husband for 3 years, it's been wonderful, I couldn't 't ask...
I've been married to my husband for 3 years, it's been wonderful, I couldn't 't ask...
I've been married to my husband for 3 years, it's been wonderful, I couldn't 't ask...
It was my school's 's annual 5K, so the runners are students, faculty, and then ran...
Ive worked with this girl once week for almost year. When we met we were bot...
Ive worked with this girl once week for almost year. When we met we were bot...
Ive worked with this girl once week for almost year. When we met we were bot...
Ive worked with this girl once week for almost year. When we met we were bot...
Ive worked with this girl once a week for almost a year. When we met we were bot...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
My girlfriend and have been going out for about a year and have decided to mov...
Table 3: First Acquired Batch by BALD (baseline).
27

---

BAL-PM- Acquired Batch (Truncated Prompts)
*Qiick Background**: As the title states, we've been together for 7 years datin...
"hee texts:** Him: at least my mom thinks I'm cute me: I think you' 'r cute :)... ** Warningg Avoid this film if you only broke up very recently! I advise this fil...
**i(2mm) have been dating her(26f) on and off for 5 years, ** I have come to the...
So we broke up as in words she had severe depression and it wasn't fair m...
A little while back, my sister asked me why some men were homophobic. I answere...
small background. I live in in Puerto Rico, where haven't had to good an ex...
About a month ago we started having problems with our cable. The picture would g... Background info: Little background. started medical school a few years back. I...
Backstory: I'm 17 year old student in the U.K. currently in sixth-form. Back i...
Be sure to explain in detail with line breaks. Hey my name is Matt and honest...
Because live in very conservative Catholic neighborhood, I cannot come out a... Hello all, Story: played around with some stocks a few years back buying... ... Hello reddit My LDR girlfriend of six months told me yesterday that she wasn't... Hello! Last group of friends I had was back in 10th Grade. Since then my depre...
Hello! I'm 23 y/ F dating 30 y/o male. This is by far the best relationship... Hello, I'm kind of new to this sub reddit but figured I'd get an opinion from
Hey Reddit. I spent at least 20 mins looking for the correct sub-reddit for men'...
Hey all. My classmates and at the SUNY Purchase Film Conservatory are in the p... Hey everyone I'm about 3 months in of my 6 month regimen before I get gastric... Hey fellow revenge-lovers, here's s a quick one, that happened about an hour ago.... Hey guys this is strange to begin with, but I'll introduce the situation: I'm... I am a 24 y/o male and have been dating a girl who is 22 years old for about 1...
I am 18 year old college student and have no attachments to my local area.
I am aware that this has been proposed before. I personally believe that it woul...
I am dating complete dime like I get compliments all the time about her from s... I can't 't focus. I can't 't become and remain motivated. When I've learned something...
I know we are young but bear with me, I didn't know where else to go for this ty... I live in California and am the co-owner of a car, with the names on the title b...
I made a previous post here but it sounded kind of stupid with the way phrased...
I met my current girlfriend in highschool. She's 's the only woman I've ever been I should start with saying neither of us have had chance to travel anywhere ex...
I want to start off by saying I love my SO and I'm looking for suggestions befor...
I was in a pretty serious car accident this week, and my car was easily totaled....
I will try to make this as short as possible. a long time ago met this girl,
I'll keep it short :3 I' I'm 18, he's 's 18. Dating for years. When we walk togethe...
I'll keep this as succinct possible. I moved in Sept. 1. I used to live here... I've been with my boyfriend for 4 years, it hasn't 't been the best relationship, b... I've been with my gf for almost 7 years. Lived together for about 5 years. A few... I've been working with this girl for 2 months. it started at work where was he... If you want to understand the scam, here's what's 's happening: Okay, so I found a... Im 27. Single. am a productive member of society. work full time i pay my ow... It was New Year's Eve and my family It wasn't 't that long term relationships but we lived together for 6 months so we
driving off to my grandparents' house. H
Just got the new Kobo touch and they provided me with $10 gift card for their... ...
My friend's little brother is really suffering in his dorm. He's lost 15-20 poun...
My girlfriend an I have been dating for three years. Its been the best time of m...
My girlfriend and I met through family friends a year and a half ago. We've ve been...
Table 4: First Acquired Batch by BAL-PM.
28

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Sections and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 0
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Yes]
Justification: See the Reproducibility Statement (Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer:[Yor Yes]
Justification:
See the Reproducibility Statement (Appendix B).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/uuiii wwwpc//ggue////eee CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer to Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See details in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
31

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See details in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttps://neuriie httppp/ss.....b..ttttttcccclllea
Answer:
[Yes]
Justification:
We reviewed the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the Impact Statement (Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
32

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly cited the frameworks and dataset owners used this work. See
B
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
33

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All code is available and documented in the link provided. See Appendix B Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "EDK1AA6AIC", "reviewer": "Reviewer_x99G", "summary": "This paper presents BAL-PM, a Bayesian active learning framework for the training preference model. The authors propose an acquisition policy that seeks examples that have high epistemic uncertainty and can maximize training data’s entropy. Specifically, the epistemic uncertainty is estimated by the training preference model and entropy is estimated by the base LLM. Experiments on two preference datasets show that the proposed method requires fewer training data compared to previous methods (33% and 68% reduction on each dataset accordingly). The author also shows that their method is scalable to larger LLMs.", "questions": "- How do the results compare to models trained on the whole dataset?\n- Do the trained models lead to better LMs?", "limitations": "The authors have properly adequately the limitations of their work in the paper.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The problem is clearly formulated. The method is well derived and justified.\n- The results are strong, showing significant improvement in sample efficiency.\n- The authors test their method on models of various sizes, demonstrating its scalability.\n- The paper is clearly written and easy to follow.", "weaknesses": "- The evaluation metric is limited. The authors only show the log-likelihood of the test sets. This single metric has no guarantee that preference models trained by this method have a more accurate judgment (shown by pair preference accuracy) and lead to better fine-tuned LMs (shown by tuning with the trained preference model)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work strengths and for raising concerns and questions. We hope to clarify them in this response. Please see the answers below:\n\n**W1** \"The evaluation metric is limited and gives no guarantee of more accurate judgment or better fine-tuned LMs.\"\n> **R1** We kindly refer the reviewer to **Q1** under the global rebuttal, where we clarify the Log Likelihood metric and shows that it provides more information about preference strength for ranking models than prediction accuracy. We also refer to **Q5** where we argue about the relationship between Average Log-Likelihood as a performance measure for preference models and the quality of the fine-tuned policy.\n\n**Q4** How do the results compare to models trained on the whole dataset?\n> **A4** In Figure 12 of the Rebuttal PDF, we show (in purple) the performance of a trained model in the full dataset (over five seeds). BAL-PM achieves on-par performance only requiring ~24000 points (the full dataset contains 92,858 points). This result is another interesting evidence of the sample efficiency of our method.\n\n**Q5** Do the models evaluated by Average Log Likelihood (LL) lead to better finetuned policies?\n\n> **A5** Despite our work strictly focusing on Preference Modeling (which has several applications in psychology [1], economy [2], and sociology [3] that goes beyond RLHF), we agree that fine-tuning LM policies is a very relevant downstream task. Our Preference Modeling optimization objective and model selection protocol follow exactly the prior influential work on the topic [4, 5], which provides evidence that better preference models (in terms of validation loss) lead to improved downstream policies. Thus, we expect our models to behave similarly under the same conditions.\n>\n> As additional evidence, we empirically illustrate the relationship between log-likelihood and policy performance on a simplified setup (see Figure 13 on Rebuttal PDF). Here, prompts $x$ and answers $y$ are real numbers in [0, 1]. The ground-truth reward function is given by a Gaussian density function $r(x, y) = \\mathcal{N}(x + y \\mid \\mu = 1.0, \\sigma = 0.4)$, and true preferences follow the Bradley-Terry model. In this setup, we progressively increase the training set size (the x-axis in Figure 13a) in which we train the preference models. This process generates different models with increasing levels of test-set average log-likelihood. Then, similar to [6], we optimize the base policy via a Best-of-N optimizer by leveraging each of these learned preference models. Finally, we report the rate in which the fine-tuned policy answer is preferable over the base policy answer according to the ground truth reward model (\"win rate\"). Although simple, this setting allows us to bypass several optimization and distributional challenges and solely focus on evaluating the relationship between average log-likelihood and the performance of the fine-tuned policy. Figure 13 (left) reports the log-likelihood (red) and the win rate against the base policy (blue). Figure 13 (right) directly plots both measures and fits a regression line. We observe a strong correlation, which aligns with our point: **a higher test-set average log-likelihood means that the preference model is better at predicting the ground truth preferences, assigning higher rewards for better answers, and, therefore, improving fine-tuned policies that maximize such reward scores.**\n\n\nWe hope we clarified your questions and concerns. Please let us know if there any any further points to be clarified - we genuinely appreciate your time reviewing our work.\n\n**References**\n\n[1] Kahneman et. al. Judgement under Uncertainty — Heuristics and Biases, 1981.\n\n[2] Armstrong, W.E. A note on the theory of consumer’s behavior. Oxford Economics, 1950.\n\n[3] Sen, A.K. Social choice theory. Handbook of Mathematical Economics, 1986.\n\n[4] Ziegler et. al. Fine-tuning language models from human preferences, 2019.\n\n[5] Stiennon et. al. Learning to summarize with human feedback. NeurIPS, 2021.\n\n[6] Gao et. al. Scaling laws for reward model overoptimization. ICML, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pIYm7BTk2j", "reviewer": "Reviewer_aVyo", "summary": "This paper proposed a novel framework to select the most informative preference data for training based on Bayesian active learning. \nTo collect the prompt-response pair (x,y), it firstly selects the prompt based on Bayesian Active Learning by Disagreement by maxmizing the information gain. Then, the selection of response considers both the preference model epistemic uncertainty estimation and entropy estimation for the acquired prompt distribution, which addresses the challenges of selecting diverse samples. When evaluated on Reddit and CNN/DM preference datasets, it saves 33% and 68% training samples by comparing to random sampling.", "questions": "See the details in Weakness.\n1. How are the performances of existing other baselines.\n2. What are the computation cost in estimate the utility score, diversity and entropy.", "limitations": "Yes, the authors mentioned the intrinsic limitation of the transformer's embedding, i.e., noise-TV. The impropriate selected samples can be toxic, unfair or not well-represented.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research problem is important to data-efficient training. \n2.  The presented emprical results are impressive in saving utilised training data over listed baselines.\n3.  The empirical studies is comprehensive and verify the effectiveness across different settings.", "weaknesses": "1. Lack for diversity-based and uncertainty-based sample selection methods, such as [1,2,3,4], which makes the technical contributions less reliable,\n2. As the method is based on feature extraction from existing LLMs, the computation cost in estimating the diversity and uncertainty is not clear, which inhits its pratical application.\n\n\nReferences:\n[1] What Makes Good In-Context Examples for GPT-3?\n[2] An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels\n[3] Demystifying Prompts in Language Models via Perplexity Estimation\n[4] Diverse Demonstrations Improve In-context Compositional Generalization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the strengths of our work and for bringing up your concerns and questions. We aim to address them in this response.\n\n**Q6** How does BAL-PM perform in comparison with other sampling methods [1, 2, 3, 4]:\n\n> **A6** Thank you for the references. We highlight that these works focus on data sampling for In-Context Learning (ICL), while ours investigate Active Learning for Preference Modeling in LLMs. While both topics aim to improve sample efficiency, they operate in different setups: **ICL is a test-time procedure** that incorporates training points to condition the prompt in order to improve predictions of a test point), while **Active Learning is a training-time procedure** that selects training points for improving the model. Fundamentally, this may lead to different conclusions as to what is required for sample efficiency. Nonetheless, we adapted most of the suggested methods to our scenario to evaluate them and provide more evidence of our contribution. We also added other baselines from the Active Learning literature:\n>  - Entropy Minimizer: Inspired by [1], we consider an objective that, in addition to selecting points with high epistemic uncertainty, also selects points that are semantically similar to the current training points. This is equivalent to selecting points that increase the entropy of the prompt distribution **the least**, thus the name \"Entropy Minimizer\". It serves as a check for our central hypothesis that entropy maximization leads to better batch active learning.\n>  - Perplexity: Inspired by [3], we consider an objective that selects points based on the perplexity of the base LLM. We consider two versions: one that chooses points with lower perplexity (Low Perplexity), and another with higher perplexity (High Perplexity). This is an interesting baseline since perplexity is equivalent to the predictive entropy of the token distribution. Therefore, it helps to analyze how much the base LLM \"knows what it does not know\" in terms of preference modeling.\n>  - MC Dropout [5]: This method performs approximate Bayesian inference via executing dropout at test time to generate different parameter hypotheses. Therefore, it can express epistemic uncertainty, which is used to select the most informative points.\n>  - Latent Reward Uncertainty (LRU): This method computes a reward distribution over the data points by leveraging the latent reward model learned via the Bradley-Terry model. Then, it selects extreme points (too high or too low rewards) as a proxy for the uncertainty of the model.\n>  \n> Figure 16 (Rebuttal PDF) reports performances for both test and OOD sets. In both cases, BAL-PM outperforms additional baselines. In the sequence, MC-Dropout works best, as the baseline that targets the epistemic uncertainty of a Bayesian model. Expectedly, Entropy Minimizer and Low Perplexity work worse since they target points with lower entropy. LRU presented mixed results, suggesting that the latent reward may not represent well the preference model's uncertainty. More interestingly, while these models can represent different uncertainties to seek informative points, they naturally cannot provide in-batch diversity - they suffer from the same challenges as BALD. In this perspective, the BAL-PM objective can also improve upon those methods, as we show in Figure 17: we combined MC-Dropout and LRU with our entropy term to provide in-batch diversity, which consistently improved both methods across the datasets.\n> Lastly, we highlight that the suggested method in [2] is akin to BALD, as it focuses on the mutual information objective (described in Equation 3 in our paper). [4] focus on semantic parsing approaches to enable compositional generalization, which implicitly assumes key properties of the considered input: For instance, there is a mapping between the natural language sentences and formal queries. In preference modeling for LLMs, we do not assume the existence of such formal queries. Given this assumption, [4] builds diversity sampling on top of syntax trees, which is out of the scope of our work on preference modeling.\n\n\n**Q7** What are the computation costs involved in BAL-PM?\n> **A7** We kindly refer the reviewer to **Q2** in our global response, where we discuss in detail the costs of our method, contextualizing with the Active Learning domain.\n\n**W2** Limitation: the impropriate selected samples can be toxic, unfair or not well-represented\n> **R2** We provide some context around this potential limitation. In terms of toxicity, we highlight that selecting toxic prompts for getting preferences is actually a crucial step for detoxifying fine-tuned language models. If the reward model is not trained on such prompts, they may behave as out-of-distribution samples, leading to incorrect predictions that could reinforce toxic answers. Therefore, providing preference labels that consider safe answers over toxic ones is necessary for preference modeling. A good example is the work of Bai et. al. [5], where the employed dataset contains content that may be offensive or upsetting, to make LMs less harmful.\n> Regarding representativity, we believe that having an entropy term that seeks diversity in the prompts will actually reinforce representativity over different prompt classes.\n\nWe hope that we addressed your questions and concerns. Let us know if there are any further points to be clarified - we genuinely appreciate your review.\n\n[1] Liu et. al. What Makes Good In-Context Examples for GPT-3?. ACL DeeLIO, 2022\n\n[2] Sorensen et. al. An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels. ACL, 2022 \n\n[3] Gonen et. al. Demystifying Prompts in Language Models via Perplexity Estimation. EMNLP Findings, 2023\n\n[4] Levy et. al. Diverse Demonstrations Improve In-context Compositional Generalization. ACL, 2023.\n\n[5] Bai et.al. (Anthropic Team). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nt3JvzhFcM", "reviewer": "Reviewer_vibJ", "summary": "&nbsp;\n\nThe authors present a method, Bayesian Active Learning for Preference Modeling (BAL-PM) that seeks to learn a preference model in a sample-efficient manner within an active learning setting. A key insight by the authors is the consideration of task-dependent and task-agnostic uncertainty to encourage diversity in the sampled prompt-completion pairs. My main concern with the approach is the link between the pool-based active learning setting and active learning over an open-ended and continuous prompt space which I will describe below. As such, my recommendation is borderline pending the authors rebuttal on this point. \n\n&nbsp;", "questions": "&nbsp;\n\n1. In Section 4, the authors mention that their Bayesian preference model is constructed using an ensemble of adapters. What was the motivation for this choice over Laplace-LoRA [2] for example?\n\n2. In Figure 7, random sampling outperforms BAL-PM for the 140b parameter model early in the active learning trace. Do the authors have an explanation for why this might be the case?\n\n3. It would be interesting to see the full details of the validation procedure for choosing the hyperparameters, most notably the entropy term \\Beta. How sensitive was performance to the choice of \\Beta hyperparameter? What is the link to Section F of the appendix?\n\n4. In terms of the results presented in Figure 9, how do the authors compute the aleatoric uncertainty for the datasets? Are these uncertainty estimates provided in the datasets?\n\n\n&nbsp;", "limitations": "&nbsp;\n\nThe biggest limitation I foresee with the current work is the potential for the performance of BAL-PM to carry over between the pool-based active learning setting and the non pool-based setting as described above.\n\n&nbsp;", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "&nbsp;\n\nThe paper is well-written and the idea behind BAL-PM is well-motivated. The empirical results appear compelling in the pool-based setting and furthermore, the authors provide the code to reproduce their results. The analysis of the diversity of the acquired prompts is a useful diagnostic to validate the authors' hypothesis that BAL-PM attains performance improvements by encouraging diversity in the sampled prompts.\n\n&nbsp;", "weaknesses": "&nbsp;\n\n__MAJOR POINTS__\n\n&nbsp;\n\n1. My main concern with the current work is the link to the real-world problem of constructing a  preference model in a sample-efficient fashion in the absence of a pool of labelled prompt-completion pairs. Specifically, as I understand, one of the main motivations for being sample efficient in constructing the preference model is the cost of human labelling. In the experiments considered by the authors, the datasets already contain labels. As such, I would ask  a) what is the motivation for being sample-efficient in the pool-based active learning setting? b) assuming the authors feel performance on the pool based setting is representative of an open-ended active learning problem, how would they go about verifying that BAL-PM performs well in the non pool-based setting?\n\n&nbsp;\n\n__MINOR POINTS__\n\n&nbsp;\n\n1. It would be great if the references appeared in numbered order.\n\n2. There are some missing capitalizations in the references e.g. \"Bayesian\".\n\n3. The arXiv identifier is missing for reference 2.\n\n4. When referencing PyTorch, [1] should be used rather than the earlier workshop paper version.\n\n5. It would be great if the authors could provide instructions for reproducing the experimental results in the README of their GitHub repository.\n\n6. Reference 7 was accepted at TMLR.\n\n7. Line 38, I would not expect the statistic that agreement rates among human labelers is typically 60-75% to be robust across all problem settings. It would be worth qualifying this statement.\n\n8. The arXiv identifier for reference 18 is missing.\n\n9. \"Prompt-completion pairs\" would potentially be a more appropriate terminology in place of \"prompt-answer\" pairs.\n\n10. On line 105, S is not defined, this should presumably be \\Chi.\n\n11. Line 134 typo, \"an unsupervised\".\n\n12. In Figure 4, it would be great to include in the caption the number of random trials for which the errorbars are computed.\n\n13. In the abstract the authors do not mention that the figures of 33% and 68% apply to a random sampling baseline.\n\n&nbsp;\n\n__REFERENCES__\n\n&nbsp;\n\n[1] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L. and Desmaison, A., 2019. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32.\n\n[2] Yang, A.X., Robeyns, M., Coste, T., Wang, J., Bou-Ammar, H. and Aitchison, L., 2024. Bayesian reward models for LLM alignment. arXiv preprint arXiv:2402.13210.\n\n&nbsp;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for sharing your concerns and questions. We address them below:\n\n**Q8** What is the motivation for being sample-efficient in the pool-based active learning setting?\n\n> **A8** We clarify that the goal of a pool-based setup is to mimic an open-ended data selection setup. Although we use logged data in the pool, **the experiment only reveals the labels when the active learning policy selects the data points**. Hence, for the Preference Model, the pool data is completely unlabeled. It is a standard experimentation protocol in Active Learning literature [1,2] and allows the use of real human feedback. For instance, Gleave [3] also adopts the same experimental setup.\n> A non-pool-based setup would require either collecting human feedback for every batch in each experiment (impractical) or relying on a preference simulator that tries to mimic human behavior. This is also unrealistic, as accurately modeling the human behavior and the aleatoric uncertainty involved in the preference elicitation process is really challenging. Therefore, using data that leverages real human preferences is more realistic for Preference Modeling and enables purely offline experiments.\n\n\n**Q9** How would BAL-PM perform in a non-pool-based setup?\n> **A9**  Given answer **A8**, the key thing here is to realize that the difference between a pool-based and a non-pool-based one is that the latter permits generating different answers for the prompts, while in the former the answers are fixed. As pointed out by previous work [3], this means that our setup may actually **underestimate** the benefit of Active Learners (BAL-PM) since we limit the stochastic policy to prompt selection. Therefore, in a non-pool setup, we believe that BAL-PM should work as well as, or even better, given another degree of freedom for data selection. Crucially, we designed BAL-PM to work in both settings without changes.\n\n\n**W3** Minor Points\n> **R3** We appreciate the detailed list. We added all points to our current draft, which should be reflected in the camera-ready version. For point 7, we qualified the statement by clarifying that this is the result observed in past preference modeling settings for LLM finetuning (as in the references).\n\n**Q10** What is the motivation of using adapters over other methods such Laplace-LoRA?\n> **A10** This is a great question that lies at the core of our method. The choice of adapters is computational tractability: they rely on LLMs solely for feature extraction. Hence, BAL-PM does not require training or finetuning LLMs during the active learning loops, considerably reducing the computational costs and allowing scaling for very large LMs. A method like Laplace-LoRA (or any LoRA method) requires finetuning the base LLM. Even if we assume that the finetuning is cheap (which is not for 70B+ models), updating the LLM requires generating new features for all the prompt-answer pairs in the pool at each training loop. This considerably increases the computational costs of performing active learning loops. For adapters, we can generate features once *before* the active learning experiment in inference-optimized settings (with quantization, for instance), and go beyond 100B models in a single A100 GPU.\n\n**Q11**  In Fig 7, random sampling outperforms BAL-PM for the 140b parameter model early in the active learning trace. Do you have an explanation for why this might be the case?\n> **A11** We hypothesize that this may be due to 4-bit quantization affecting prompt representations for entropy estimates, which is crucial in the early stages of training for accelerating the Bayesian Preference Model learning. The quality of the features is one of the limitations of BAL-PM. Also, as the Bayesian Preference Model contains more parameters for larger base LMs, it may require more initial data to fit well and provide accurate epistemic uncertainty estimates.\n\n**Q12** What is the validation procedure for hyperparameter selection? How sensitive BAL-PM is for the choice of $\\beta$? What is the link with the Appendix F?\n> **A12** The procedure was quite standard: given the search space in Table 2, we evaluated the candidates in the held-out validation set and selected the best-performing hyperparameter. We tuned the presented hyperparameters in isolation (a grid search would be too expensive).\n> **We perform a $\\beta$ robustness analysis in Fig. 15 (Rebuttal PDF)**, considering the values in the search space. The impact of the choice is more noticeable to values 100x greater/lower than the optimal choice. Values around 10x greater/lower still perform well, suggesting good room for choosing this hyperparameter. Furthermore, we employed the same value of $\\beta$ across the different datasets and LLMs, suggesting robustness across different relevant dimensions.\n> Crucially, $\\beta$ trades off the contribution of two different terms. As such, it provides a spectrum of objectives and may recover the two extremes presented in the ablation of Fig. 8. Naturally, different choices of $\\beta$ will change the uncertainty score ratio presented in Fig. 11 on Appendix F (i.e., the contribution of each term after convergence). Nevertheless, and most importantly, the behavior of the curves – the entropy contribution progressively reducing and converging and the relevance of epistemic uncertainty estimates increasing – should remain.\n\n**Q13** How do you compute the aleatoric uncertainty? Is this provided in the datasets?\n> **A13** We compute the predictive uncertainty as the entropy of the posterior predictive distribution (the first entropy term in the Equation 3). Similarly, we compute the epistemic uncertainty via Equation 3. By the Law of Total Variance, the Total (Predictive) Uncertainty in a Bayesian Model is the sum of the Epistemic Uncertainty and Aleatoric Uncertainty. Thus, we can estimate the Aleatoric Uncertainty based on that. These are only **estimates** from our model. Thus, they are not values provided in the dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QdTY7pWlP4", "reviewer": "Reviewer_tUWV", "summary": "This work proposes BAL-PM, a stochastic acquisition policy aiming to select informative data samples (i.e., the prompt and its corresponding paired responses) that need to collect human feedback for LLMs' preference learning.\nSpecifically, BAL-PM mainly addresses the issue of sampling redundant data points in previous methods that rely solely on epistemic uncertainty, such as BALD. On top of the previous BALD acquisition method, BAL-PM employs a task-agnostic entropy estimator for the prompt distribution to enhance the diversity of acquired data samples.\nExperiments on the Reddit and CNN/DM datasets under a pool-based setting demonstrate the effectiveness of BAL-PM over random sampling and the BALD baseline.", "questions": "- What is the advantage of adopting the (static) feature space of the base LLM for estimating the entropy of the prompt distribution? Is there any evidence to demonstrate that it is superior to other approaches, such as using smaller proxy models (e.g., BERTScore) or simpler fashions?\n- I am curious about whether different adapters can make diverse predictive distributions with limited expressive power. Is there any empirical evidence for this?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow.\n- The method is well-motivated by the observation of the tendency of sampling redundant data points in previous methods.\n- The effectiveness of key components of the proposed BAL-PM is well justified and supported by the experimental results and ablation.", "weaknesses": "- The experiment part is not compelling and clear enough to me. Regarding the reported log-likelihood metric, it is unclear to me how it is calculated (e.g., on preferred response?) and why it is the only metric used for evaluating preference modeling. It seems to me that it can not measure to what extent the chosen response is preferred over the rejected one. Regarding the datasets evaluated, the task is limited to text summarization. The experiment part can be improved by adopting more metrics on more general-purposed datasets/tasks to illustrate the effectiveness of the proposed method.\n- While encouraging data diversity is plausible and effective in LLM's preference learning according to the experiments, the necessity of using the feature of the base LLM to measure the entropy of the prompt distribution is unclear to me. Although it is mentioned as a common approach in lines 97-99, the motivation for focusing on feature space is still somewhat vague to me and may introduce extremely high computational overhead in the context of LLM's preference learning.\n- The proposed BAL-PM, which requires maintaining/updating an ensemble of adapters for collecting each batch, may be difficult to practice in large scale.\n- The discussion of prior works is inadequate. Besides the difference in configuration, the technical parts of other related active preference optimization of LLMs should be discussed more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for sharing your concerns and questions. We address them below:\n\n**Q14** Why is the Log-Likelihood metric used for evaluating Preference Modeling?\n> We refer to **Q1** (global rebuttal).\n\n**W3** The task is limited to text summarization.\n> **R3**  We understand the importance of different NLP tasks to enrich evaluation. Still, we focus on validating the BAL-PM as an Active Learner in Preference Modeling. We argue that the considered datasets already bring the key challenges for this task to provide empirical evidence to support our findings:\n> - They present real human feedback, reflecting the true human behavior and aleatoric uncertainty involved in the preference elicitation process;\n> - Prompts are human-generated and go beyond simple instructions. They incorporate human subjectivity and diversity in terms of culture and topics, requiring a high level of language understanding. We refer to Tables 25-28 in [1] for samples. It is a challenging setup.\n> - The CNN/DM News dataset presents OOD prompts (from News, far from Reddit posts) while following the same label-generation process. It allows the evaluation of robustness/generalization of learned preference models.\n\n**W4** The experiment part can be improved by adopting more metrics on more general-purposed datasets/tasks to illustrate the effectiveness of the method.\n> **R4** We highlight that our work already explores several dimensions to validate BAL-PM, going beyond performance claims. For instance, we:\n> - Provide evidence in an OOD setting (Figs 4b, 5b) to validate robustness/generalization;\n> - Present measures of diversity to analyze the acquisition of redundant samples (Fig 6);\n> - Evaluate the scalability of our method for larger LMs with 70b+ parameters (Fig 7);\n> - Evaluate crucial design choices related to the policy objective and its implementation (Figs 8-10);\n> - Analyze how the stochastic policy balances the influence of each epistemic uncertainty source, a key property of BAL-PM (Fig 11).\n> \n> All raised experiments validate crucial points of BAL-PM and Active Preference Modeling, providing evidence to support our method.\n\n**Q15** What is the advantage of adopting the feature space of the LLM for entropy estimation of the prompt distribution? Is it superior to using smaller proxy models (e.g., BERTScore) or simpler fashions?\n> **A15** Estimating the entropy of the prompts requires representations that encode semantic features. The underlying idea follows the core of BERTScore: leveraging the distances on this semantic vector space to represent similarities among natural language sentences. We expand this perspective to use not as a metric but as a density/entropy estimator of the prompt distribution.\n> The key point is that **this process requires a feature space that can perform well in encoding the extent of the natural language distribution**. Simpler feature extractors, such as tf-idf or CBoW, struggle to encode contextual semantic information. While traditional deep learning models like BERT and ELMo have promising architectures and learning objectives, they are currently limited by their model and training set sizes compared to LLMs trained on meticulously curated datasets that encompass the entire web. This inherent relative limitation translates to performance: for evidence, we refer to the SuperGLUE benchmark [2], comprising several language understanding tasks and online leaderboard. Currently, the best ranked models present 10b+ parameters (the scale of our experiments). BERT-based baselines and classic techniques place in 26th position (see “SuperGLUE baselines”). This result strongly suggests that LLMs provide better representations than models with smaller scales, which is especially crucial for our method, particularly for the challenging prompts in the considered datasets.\n\n**W5** Focus on the LLM feature space to measure entropy may introduce computational overhead in the context of LLM's preference learning.\n> **R5** We refer to **Q2** (global response) for computational cost clarifications. There is no additional cost as we can extract these \"entropy features\" and \"preference model features\" simultaneously.\n\n**W6** BAL-PM requires maintaining/updating an ensemble of adapters for collecting each batch, which may be difficult to practice in large scale.\n> **R6** We again refer to **Q2**. Our adapters are simple MLPs with 2 hidden layers, whose cost is reasonably cheap for LLM research.\n\n**W7** The discussion of prior works is inadequate and should describe the technical parts of Active Preference Optimization in LLMs. \n> **R7** We extended the \"Active Preference Modeling\" section in Related Work, detailing the draft references [11, 13, 25, 35]. The actual text is too long to fit here, so we summarize: [11, 13, 25] theoretically studies active query generation and propose different methods based on estimating confidence bands to generate high-uncertainty triples. [35] generate answers using double Thompson Sampling, representing model uncertainty with an Epistemic Neural Network. We also describe that [18] uses a fully fine-tuned deep ensemble for model uncertainty estimation. Let us know if this is clear. Otherwise, we may paste the exact text as a comment.\n\n\n**Q16** Can adapters make diverse predictive distributions with limited expressive power?\n> **A16** Adapters can be seen as a special case of Epistemic Neural Networks [3], where the prior distribution is a mixture of delta functions over indices. EpiNets provide principled theoretical justification for adapters. There is empirical evidence of adapters in the context of exploration in LLMs [4], which is similar to our method (on capturing epistemic uncertainty) but focused on answer generation, not data selection. We also incorporated MC-Dropout [5] as baseline (see **Q6**), a well-known method for Bayesian approximation. Adapters outperform MC-Dropout in our setup, suggesting effectiveness on approximating posterior distributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for raising concerns and providing feedback to improve our work. We appreciate the acknowledgement that:
- **The paper is clear and well-written** (x99G, vibJ, tUWV);
- **The proposed method is principled and well-motivated** (all reviewers); 
- **Empirical results are strong/impressive/compelling/effective** (all reviewers);
- **The method demonstrated scalability (x99G) and effectiveness across different settings** (aVyo).

We highlight the new empirical evidence in the Rebuttal PDF to clarify concerns and incorporate feedback. In detail:

- **The performance of preference models in the full dataset** (Fig. 12, **Q4**, x99G);
- **The relationship between a preference model test loglikelihood and the corresponding finetuned policy performance** (Fig. 13, **Q5**, x99G)
- **The objective's $\beta$ hyperparameter robustness analysis** (Fig. 14, **Q12**, vibJ)
- **Several additional baselines** (Figs. 15/16, **Q6**, aVyo)

We now clarify questions raised by different reviewers:

**Q1** Is Average Log Likelihood (LL) a proper performance measure for Active Preference Modeling? **(Reviewers x99G, tUWV)**

 > **A1** We first clarify how the Average LL is computed. Given the test set $\mathcal{D}\_{test} = \\{(x, y_1, y_2, {y_1 \succ y_2})\\}^{N}$ and the learned preference model $p_{\boldsymbol{\theta}}(y_{1} \succ y_{2} \mid x, y_{1}, y_{2})$, the average Log Likelihood is given by $LL(\mathcal{D}\_{test}, \boldsymbol{\theta}) = \mathbb{E}\_{(x, y_1, y_2, y_1 \succ y_2) \sim \mathcal{D}\_{test}} [\log(p_{\boldsymbol{\theta}}(y_{1} \succ y_{2} \mid x, y_{1}, y_{2}))]$. It is exactly the objective maximized in standard binary classification (or, equally, the minimization of the negative log-likelihood loss) but computed over the test data. In other words, this is the negative "test loss". 
 >
 >  Average LL is a typical metric in the Active Learning and Uncertainty Quantification literature [1, 2, 3]. For Preference Modeling, it is very relevant as **LL directly accounts for the *preference strength* to rank models**: given a triple $(x, y_{1}, y_{2})$ where all raters agree that $y_{1}$ is preferable over $y_{2}$, LL allows us to measure that a model A predicting $p_{A} = 0.9$ ($LL = -0.1$) is better (in that data point) than another model B predicting $p_{B} = 0.6$ ($LL = -0.5$). Accuracy would provide an equal score for both models since it only accounts for the binarized prediction. LL provides a more "fine-grained" measure. 
 >
 >  Another crucial point is that **LL factors in the aleatoric uncertainty in the label-generating process**. For instance, in a scenario where only 70% of the raters agree that $y_{1}$ is preferable, LL better ranks models whose predictions are closer to p = 0.7, respecting the ground truth preference strength, which is not possible with accuracy. 
 >  We also empirically illustrate in Figure 13 (Rebuttal PDF) in a simple problem setting that preference models with higher average LL lead to finetuned policies with higher win rates over the base policy (see **Q5**).

**Q2** What is the computational cost of BAL-PM? **(aVyo, tUWV)**
> **A2** We clarify this point since we argue that computational tractability is one of the main contributions of our method. First, some context: our work focuses on *(Bayesian) Active Learning*, which is naturally more computationally demanding than simply training predictive models. This is because **we require models that expresses epistemic uncertainty** to acquire informative labels for efficient training. This also **requires models to constantly update their uncertainties given the new data, via re-training**. The key is that **Active Learning reduces the number of labels required to train a better model, which considerably overcomes the additional computational cost**. The labeling process is considerably more expensive and laborious.
>
> As described in the Introduction, Preference Modeling in LLMs requires batch acquisition - it is impossible to request the label of a single point, re-train the model, and repeat this process. Still, tractable methods rely on these single-point acquisition objectives. Thus, **what BAL-PM does computationally is to replace $B - 1$ model re-trainings per acquired batch with computing entropy estimates** (considerably cheaper, as explained below). $B$ is the batch size, and $B = 320$ in our experiments.
>
> **BAL-PM does not require training or inference on LLMs during the active learning loops**. This considerably reduces the computational cost and allows us to scale up to 140b models in a single A100 GPU. To put it in perspective, fully fine-tuning a 7b model currently requires at least 4 A100s. LoRA methods also require new LLM inferences for every model update, while BAL-PM only requires once.
>
> The computation of BAL-PM has three pieces: offline processing (LLM inference and kNN computation), adapters update, and entropy estimation. LLM inference is done only once, prior to Active Learning. It is the bare minimum for LLM adoption. Furthermore, **we can compute the features used for the preference model and for entropy estimation in the same forward pass**: every prompt-answer input concatenates prompt/answer texts; thus, we can extract prompt features as the last layer embedding right after the last prompt token, and the prompt-answer features right after the answer's last token. Hence, there is no extra cost to extract features for entropy estimation.
> The cost of updating adapters is minimal, as they are MLPs with 2 hidden layers, reasonably cheap for LLM research. The entropy estimation only requires computing the di-gamma function (Equation 11) in the pool.
>
> Ultimately, our experiments show that BAL-PM can handle a challenging real-world preference dataset (used in the precursor of ChatGPT) with LLMs with up to 140b parameters, demonstrating its scalability (as highlighted by Reviewer x99G). We hope this clarifies its applicability in practical settings.

Meta-review: This paper proposes to use active learning techniques to select examples for querying preference ratings for use in preference modeling algorithms. The main contention of the author is that paying attention only to epistemic uncertainty can result in collecting redundant batches of data and thus propose to also pay attention to the diversity of the data as characterized by "maximiz[ing] the entropy of the acquired prompt distribution in the feature space". Reviewers found the results to be strong/impressive/comprehensive, to be shown to scale, the paper to be clearly written. They also raised some concerns about the metric which doesn't actually show that the resulting data here led to more effective training of LLMs and the "link to the real-world problem of constructing a preference model in a sample-efficient fashion in the absence of a pool of labelled prompt-completion pairs". Following rebuttals, the mean score of the paper is a 6 and given the presence of champions and lack of a strong detractor I am recommending acceptance.
Decision: Accept (poster)