Deep Bayesian Active Learning for Preference Modeling in Large Language Models

Authors: luckeciano c melo
panagiotis  tigas
alessandro  abate
yarin  gal
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:00:34

Abstract: Leveraging human preferences for steering the behavior of Large Language Models (LLMs) has demonstrated notable success in recent years. Nonetheless, data selection and labeling are still a bottleneck for these systems, particularly at large scale. Hence, selecting the most informative points for acquiring human feedback may considerably reduce the cost of preference labeling and unleash the further development of LLMs. Bayesian Active Learning provides a principled framework for addressing this challenge and has demonstrated remarkable success in diverse settings. However, previous attempts to employ it for Preference Modeling did not meet such expectations. In this work, we identify that naive epistemic uncertainty estimation leads to the acquisition of redundant samples. We address this by proposing the Bayesian Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition policy that not only targets points of high epistemic uncertainty according to the preference model but also seeks to maximize the entropy of the acquired prompt distribution in the feature space spanned by the employed LLM. Notably, our experiments demonstrate that BAL-PM requires 33\% to 68\% fewer preference labels in two popular human preference datasets and exceeds previous stochastic Bayesian acquisition policies.

Main paper: 
 related work: 
 Bayesian Active Learning is an established form of active learning that leverages the uncertainty in model parameters to select the most informative points [14, 27, 16] , demonstrating relevant impact in several applications [19, 18, 17, 28, 29] . In this work, we apply this technique for Preference Modeling [30, 31] in LLMs. Given the requirements of such a problem setting, we focus on batch acquisition [14, 13] , particularly in the design of stochastic acquisition policies, similarly to Kirsch et al. [22] . However, our work fundamentally differs from theirs in the strategy of incorporating stochasticity. The policies introduced by Kirsch et al. [22] directly sample from the distribution determined by the single-point, task-dependent epistemic uncertainty scores. In contrast, our method maximizes the entropy of the acquired data distribution, which allows leveraging a task-agnostic source of epistemic uncertainty, alleviating the effect of biased task-dependent uncertainty scores.Active Preference Modeling leverages active learning techniques to reduce the feedback needed for Preference Modeling [27] . There has been a recent surge of interest in the area [32-35, 20, 36] given the impact of Preference Optimization in fine-tuning LLMs [2, 5, 10, 6] . A portion of this line of work focuses on query generation to directly optimize preferences. Mehta et al. [32] theoretically formalizes the problem and proposes a method that generates one completion to maximize the uncertainty of the triple in a kernelized setting. Das et al. [35] proposes a method based on confidence bands, accounting for both completions in the triple and relaxing linearity assumptions on the reward function. Ji et al. [33] constructs an optimistic estimator for the reward gap between completions and selects those with the least gap, using an uncertainty estimator to reduce query complexity. Lastly, Dwaracherla et al. [36] generate completions using double Thompson Sampling, representing epistemic uncertainty with an Epistemic Neural Network [37] , similar to our Bayesian model. Overall, while having the shared goal of reducing the volume of human feedback, query generation is orthogonal to our problem setting. Instead, we focus on the pool-based setting, as in Gleave and Irving [20] , which allows us to leverage real human feedback in experiments rather than relying on synthetic preference simulators.Gleave and Irving [20] was the first attempt of Bayesian Active Learning in this setting, and it directly applies BALD acquisition [23] for Preference Modeling, using a fully fine-tuned deep ensemble for epistemic uncertainty estimation. In contrast, our work proposes a new objective that extends BALD acquisition to account for the entropy of the acquired prompt distribution to encourage the acquisition of more diversified samples, and formulates the Bayesian model as an ensemble of adapters.Task-Agnostic Uncertainty Estimation refers to a set of techniques that quantifies uncertainties based on density estimation of the input in a learned latent feature space [38, 39] . In this context, distant points from the training set offer more information about the input space, which is useful for out-of-distribution detection [39] and unsupervised active learning [40] . Similarly, leveraging information from the feature space via entropy maximization is a common approach in Reinforcement Learning for state exploration [41] [42] [43] [44] . While our method relies on the same principles -acquiring more information about the feature space -our problem setting and methodology differ substantially, as we focus on Active Preference Modeling in the context of LLMs.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Problem Statement. We formulate our setup as an active inverse variant of the contextual dueling bandit problem [45, 46] . We assume a prompt space X , an action space Y, and a language policy τ : X × Y → [0, ∞). Given x ∼ X , this language policy τ (e.g., an LLM) selects actions y 1 , y 2 ∼ τ (• | x) (also referred to as completions), generating a dataset of tuples D pool = {x i , y i 1 , y i 2 } N . Crucially, x is sampled with replacement, i.e., we may generate multiple completions for the same prompt. Then, we define a policy π : X × Y × Y → [0, ∞), which select tuples (x, y 1 , y 2 ) ∈ D pool to query for human binary preference over completions y 1 ≻ y 2 , forming a preference datasetD train = {x i , y i 1 , y i 2 , y 1 ≻ y 2 i } B .Finally, D train is used to learn a preference model p θ (y 1 ≻ y 2 | x, y 1 , y 2 ), parameterized by θ, which aims to recover the human preference function. The goal is to find π that minimizes the amount of samples B required to learn p θ (y 1 ≻ y 2 | x, y 1 , y 2 ).Preference Modeling. In this work, we assume that the preferences y 1 ≻ y 2 are generated by an unknown latent reward model r(x, y). We model y 1 ≻ y 2 following the Bradley-Terry (BT) model [47] :EQUATIONThe BT model is often implemented by learning a parameterized latent reward model r θ(x,y) and optimizing θ via maximum likelihood estimation. This means minimizing the negative Log-Likelihood with respect to the human preference labels.Bayesian Active Learning. We adopt a Bayesian Model, which assumes a probability distribution over the parameters θ, such that, given a classification setting with inputs x ∼ X and labels y ∼ Y the predictive preference distribution is given by:EQUATIONFor active learning, we follow the methodology introduced by Lindley [23] , namely BALD (Bayesian Active Learning by Disagreement), which proposes that the utility of a data point x ∼ X is given by the expected information gain about the parameters θ with respect to the predictive distribution, a proxy of epistemic uncertainty:EQUATIONKozachenko-Leonenko Entropy. The KL entropy estimator [48] is a non-parametric, particle-based estimator that leverages the k-nearest neighbors distance. Given a random variable X and a set of N i.i.d particles {x i } N , x i ∼ X, the KL entropy estimation for X is defined as:EQUATIONwhere d X is the dimension of X, v d X is the volume of the d X -dimensional unit ball, ψ is the digamma function, and D x (i) is twice the distance between the particle x i to its k-nearest neighbor.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: All code is available and documented in the link provided. See Appendix B. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review."task" refers to Preference Modeling. A task-agnostic estimation is independent of preference labels.We release our code at https://github.com/luckeciano/BAL-PM. 
bayesian active learner for preference modeling: 
 We now introduce our method for Active Preference Modeling, BAL-PM, illustrated in Figure 2 .Our desiderata is to design an acquisition policy that addresses the shortcomings of naive epistemic uncertainty estimation -such as the acquisition of redundant samples -by leveraging an unsupervised source of epistemic uncertainty that encourages diversity in the acquired training distribution.Objective. Based on the above, we propose the following objective:EQUATIONwhere Û (x, y 1 , y 2 ) is the preference model epistemic uncertainty estimate for the tuple (x, y 1 , y 2 ) and Ĥ(X tr ∪ {x}) is the entropy estimate for the acquired prompt distribution, assuming the policy selects x. X tr is a slight abuse of the notation that refers to the set of prompts in the previously acquired training set. Lastly, β is a hyperparameter to balance the contribution of each term. Crucially, the first term represents a task-dependent source of epistemic uncertainty, since it refers to the learned preference Model. In contrast, the second term represents a task-agnostic source, as it solely relies on the information available in the feature space spanned by the base LLM.Preference Model Epistemic Uncertainty Estimation. We first describe our Bayesian Preference Model. Assuming a prior distribution over parameters p(θ), the posterior predictive distribution over preferences after observing D train is given by:EQUATIONwhere the likelihood term p(y 1 ≻ y 2 | x, y 1 , y 2 , θ) follows the BT model in Equation 1. Considering deep models, solving this inference problem is intractable, given the large parameter space. Nonetheless, we may assume a simple yet effective posterior approximation via deep ensembles [49] [50] [51] :EQUATIONEquation 8 approximates the posterior distribution over parameters p(θ | D train ) as a mixture of delta functions, where K is the number of ensemble models and θk is the MAP estimate of model k.The posterior predictive distribution is then computed via the following approximation:EQUATION)Equations 7 and 8 allow approximate inference by training multiple preference models separately. However, this is challenging in the context of LLMs, as fine-tuning billions of parameters several times is computationally expensive and impractical in many settings. Alternatively, we employ an ensemble of adapters [52, 53] , which consists of multiple lightweight networks (with a few million parameters each) on top of the frozen LLM that works as a feature extractor. This allows us to generate the LLM features offline and use them as a dataset, considerably reducing the resources required for training and Bayesian inference. This also enables using very large base models, with dozens or hundreds of billions of parameters in a single GPU setting. Finally, based on the previous modeling assumptions, we can estimate the epistemic uncertainty term employing Equation 3:EQUATION)Figure 3 : Illustration of entropy estimators. The green point maximizes the entropy estimation of the prompt distribution (according to the employed estimator). Dashed lines represent its k-NN distance. In (a), the KL estimator (Equation 4) does not account for the available prompts in the pool (in red) and underestimates the density in regions not covered by the acquired set (in blue). In (b), the KSG estimator (Equation 10) uses all data points, leading to better estimation and effectively selecting the point that maximizes the true entropy.Feature Space Entropy Estimation. Equation 5 requires estimating the entropy of the acquired prompt distribution, H(X train ). For this matter, we employ a kNN-based entropy estimator. We represent each prompt in the pool as the lastlayer embedding vector generated by the base LLM, leveraging the semantic representations learned during unsupervised pre-training.However, naively applying the KL estimator from Equation 4 has a major drawback: the training set D train initially contains very few data points and does not provide support to represent the probability density, introducing bias to the estimates and affecting the data selection.For illustration, we show the scenario of Figure 3a . In this case, we estimate the entropy using Equation 4, with k = 3. Since it does not account for the available points in the pool, it underestimates the density around the top cluster and ends up selecting the green point as the one that maximizes the entropy of the feature space, while the point that does so is in the bottom cluster. In an extreme case where all the points in the top cluster are the same, this bias leads to acquiring duplicated points. In Appendix E we formally derive the KL entropy estimator and show how the low-data regime challenges its main assumptions.Alternatively, we may use the available unlabeled pool, often much larger than the acquired set. Following the argument introduced by Kraskov et al. [54] , the key insight is to notice that Equation 4holds for any value of k and it does not require a fixed k over different particles for entropy estimation (we provide more details in Appendix E). Therefore, we can find the distance to the k-th nearest neighbor in the joint space spanned by the pool and the acquired set and map it to the corresponding neighbor (denoted as n Xtr ) in X train to estimate the marginal entropy. This results in the KSG marginal entropy estimator [54] , but repurposed to our setting:EQUATIONwhere D x (i) is now computed in the joint space and n Xtr (i) is the number of points in D train whose distance to x i is less than D x (i)/2. Figure 3 (b) illustrates the data selection by following this alternative estimation, leading to more diversity in the feature space.Implementation. Firstly, we simplify the entropy term by dropping the constant terms with respect to x:EQUATIONEquation 11 acquire points by computing D(x) (based on the kNN distance) and the counter n Xtr related to prompt x only. Furthermore, as D(x) accounts for the full dataset in the KSG estimator, it does not change over training. Hence, we may compute it offline once, and potentially scale to very large datasets [55] . Lastly, BAL-PM acquisition scheme builds a batch of data by successively selecting points following Equation 5. Crucially, while BAL-PM keeps the preference model uncertainty estimates the same over the batch, it updates the entropy term after in-batch iteration. This operation boils down to updating the counter n Xtr , a lightweight operation. In Algorithm 1, we present the pseudocode for BAL-PM. Appendix H further describes its computational cost. 
experiments and discussion: 
 In this Section, we aim to evaluate how BAL-PM performs in Active Preference Modeling. Our central hypothesis is that leveraging the information available on the feature space spanned by the base LLM -a task-agnostic source of epistemic uncertainty -addresses the problem of acquiring redundant samples, a natural pathology of relying on task-dependent epistemic uncertainty estimators designed for single-point acquisition schemes. BAL-PM, our proposed stochastic acquisition policy, promotes this diversity by maximizing the entropy of the acquired prompt distribution, besides selecting points for which the preference model presents high epistemic uncertainty.Experimental Setup. We consider a pool-based active learning setup. Each experiment consists of several acquisition cycles, where each iteration performs a batch acquisition in the currently available pool. The training set starts with the size of one acquired batch and leaves the remaining data for the pool set. Following previous works [13, 14] , we reinitialize the model after each acquisition to decorrelate subsequent acquisitions. We train the ensemble of adapters on previously acquired data and employ early stopping based on the Log-Likelihood of a held-out validation set. We evaluate the preference model after each acquisition loop and report the average Log-Likelihood of the test sets. Appendix G discusses why the test average Log-Likelihood is a proper metric for Preference Modeling. Finally, Appendix C details hyperparameters and tuning methodology used in this work 4 .Model Architecture. As described in Section 4, we employ an ensemble of adapters on top of a base LLM. Each adapter is a multi-layer perceptron with non-linear activations. In most experiments, the base LLM is a 7-billion parameter model, although we also employed 70-billion and 140-billion parameter ones when analyzing the effect of scaling the base LLM. All considered models are only unsupervised pre-trained and have not undergone any preference fine-tuning.Datasets. Following previous work [20] , we considered prompts from the Reddit TL;DR dataset of Reddit posts [25] and the CNN/DM News dataset [26] . We leverage the generated completions and human feedback collected by Stiennon et al. [1] . The Reddit dataset contains train/eval/test splits, and we adopt the train split (92,858 points) for the pool and training sets, the eval split (33,083 points) for validation, and report results in the test set (50,719 points). The CNN/DM dataset contains a single split (2,284 points), and we use it for the Out-Of-Distribution (OOD) evaluation.Comparison Methods. We considered Random Sampling and BALD [23] as baselines. BALD selects points based on the utility function of Equation 3 and is equivalent to the acquisition function used by Gleave and Irving [20] . We also compared BAL-PM with other stochastic acquisition policies [22] , namely SoftmaxBALD, SoftRankBALD, and PowerBALD. We refer to Kirsch et al. [22] for a detailed description of these methods.We highlight and analyze the following questions to evaluate our hypothesis and proposed method. Does BAL-PM reduce the volume of feedback required for Preference Modeling? We start evaluating how BAL-PM performs against standard random acquisition and BALD, as presented in Figure 4 . BAL-PM considerably reduces the volume of data required to learn the preference model. Particularly compared with random sampling, it reduces the number of required samples in 33% for the Reddit TL;DR dataset and 68% for the out-of-distribution setting of CNN/DM News dataset, representing a substantial reduction in the human feedback needed. BALD does not present any benefits over random sampling in the TL;DR dataset, which aligns with previous work [20] . Interestingly, BALD also presents an interesting improvement over random sampling in the OOD setting, but BAL-PM consistently outperforms BALD with more data.How does BAL-PM compare with other stochastic acquisition policies? Next, we analyze BAL-PM in comparison with other Bayesian stochastic acquisition policies. These policies address the acquisition of redundant samples by relying on sampling points from the distribution determined by the task-dependent epistemic uncertainty scores. BAL-PM consistently surpasses all variations in both datasets, suggesting that leveraging the information available in the prompt feature space -as a task-agnostic source of epistemic uncertainty -is more effective in encouraging diversity for batch acquisition in the considered setting.Does BAL-PM encourage diversity and prevent the acquisition of redundant samples? We evaluate the exploration approach of the considered methods by analyzing the statistics of the acquired prompt distribution, particularly the number of unique prompts over the course of training.Figure 6 presents three different perspectives on the acquired distribution. On the left, it presents the number of unique acquired prompts over learning, which indicates diversity in the training set. BAL-PM selects new prompts at a much faster rate than random sampling and BALD. Naturally, this rate saturates when the selection exhausts the number of distinct prompts available in the pool (approximately 14,000). The rate is also not equivalent to the data acquisition rate since BAL-PM does not simply select different prompts but also prioritizes points with high epistemic uncertainty.The middle plot shows the ratio of unique prompts in each active learning loop, and BAL-PM acquires batches with all distinct prompts during almost the whole training. BALD only maintains a rate of 70%, which means a substantial number of duplicated prompts. In Appendix K, we present the first batch sampled by BALD and BAL-PM for a qualitative analysis. Lastly, the plot on the right shows the ratio of unique prompts across all training. While random sampling presents a high unique prompt ratio in each separate batch, it consistently samples duplicated prompts throughout learning.In contrast, BAL-PM maintains a high ratio of unique prompts during most of the training. Again, this rate progressively decays as BAL-PM exhausts the pool of different prompts and due to the influence of the epistemic uncertainty prioritizing particular prompt-completion pairs.How does BAL-PM scale to larger LLMs? As highlighted in Section 4 our design choices allow us to scale our experiment for very large base LLMs in a single GPU setting. We investigate the effect of scaling the base LLM in BAL-PM performance, considering 70-billion and 140-billion parameter models in their 4-bit quantized versions. Naturally, the preference model performance improves substantially against the 7-billion parameter model. More interestingly, BAL-PM presents similar gains across all scales, with around 31%-33% reduction of required samples compared to random sampling. In contrast, BALD still does not present benefits over random sampling, suggesting that the scale of the base LLM is not the prevailing factor for its negative result.Ablations and Further Analysis. We conduct ablation studies in the key components of the proposed method in Appendix D. More concretely, we ablate the components of the objective to show that both preference model epistemic uncertainty and entropy scores play a relevant role in BAL-PM. We also ablate the type of uncertainty and the employed entropy estimator. Furthermore, we conduct further empirical analysis in Appendix F to investigate how each component of Equation 5 contributes to the data selection, and conduct a robustness analysis for the β hyperparameter in Appendix I. Lastly, we provide comparison with additional data selection baselines in Appendix J. 
closing remarks: 
 In this work, we present BAL-PM, a Bayesian Active Learning method for Preference Modeling in Language Models. BAL-PM is a stochastic acquisition policy that selects points for which the preference model presents high epistemic uncertainty and also maximizes the entropy of the acquired prompt distribution. We show that leveraging the information available on the feature space spanned by the base LLM via this entropy term has a crucial role in preventing the acquisition of redundant samples. BAL-PM substantially reduces the volume of feedback required for Preference Modeling and outperforms existing Bayesian stochastic acquisition policies. It also scales for very large LLMs and effectively balances the contribution of both considered sources of uncertainty.Limitations. Despite its encouraging results, BAL-PM presents some limitations. For instance, it heavily relies on the quality of the feature representations provided by the base LLM. Particularly, it might be subject to the Noisy-TV problem [56] and provide high-entropy scores to nonsensical prompts if those are spread in the representation space rather than collapsed into a single region. Fortunately, we expect this limitation to be progressively addressed by better LLMs.Future Work may evaluate BAL-PM in larger preference datasets with millions or billions of data points. Another direction analyzes how the learned models perform in the Preference Optimization setting. Lastly, future work may extend BAL-PM to consider recent prediction-oriented methods of epistemic uncertainty estimation [57] in contrast to parameter-based methods such as BALD.Preference fine-tuning has become a crucial step in aligning LLMs toward human preferences and demonstrated a real-world impact in many open-source and production systems [2, 10] . Nonetheless, collecting human feedback is very expensive and time-consuming, posing a substantial bottleneck for further development of LLMs. In this work, we approach the problem of Active Preference Modeling, which aims to reduce the volume of feedback required for learning preferences. We show that our proposed method, BAL-PM, requires 33% to 68% fewer labels in the human preference datasets considered. We believe that these results point out to a strong impact in the process of acquiring labels, and estimate an economy of hundreds of thousands of dollars and months of labeling work in the current scale of LLMs. This scenario represents faster cycles of preference optimization, potentially leading to better-aligned and safer models. Therefore, we believe our work poses a relevant positive societal impact for the upcoming years.Code Release. To ensure the reproducibility of our research findings, we release our code at https://github.com/luckeciano/BAL-PM. Our implementation is based on PyTorch [58] and HuggingFace [59] . All baselines are available in the released code.Experiments Reproducibility. We detail our methodology in Section 4 and our experimental setup in Section 5. We provide all hyperparameters used in this work as well as the selection strategy in Appendix C. We plan to release all the raw experiment logs and feature datasets generated in this work. For all experiments in this paper, we report the results over five seeds with standard errors. For better visualization, we applied smoothing for the curves considering two past observations.Datasets. All preference datasets are open-source and available online for academic use [1] .Compute Resources. We execute all active learning experiments in a single A100 GPU, and each experiment takes approximately one day. For the base LLM feature generation, we also use a single A100 GPU, taking a few hours for the 7-billion parameter model and approximately four days for the 70-billion and 140-billion parameter models.In Table 2 , we share all hyperparameters used in this work. We specifically performed a hyperparameter search on the entropy term parameters and baselines. The search strategy was a simple linear search on the options in Table 1 , considering each parameter in isolation. The selection followed the final performance on a held-out validation set. For baselines, we mostly considered the values presented in prior work [22] . For the proposed method, we also considered d X as a hyperparameter and found that smaller values often work better than using the dimensionality of the base LLM embeddings. This Section presents and discusses the results of the ablation studies. We focused on three different aspects: the components in the objective of Equation 5; the nature of the uncertainty considered; and the entropy estimator.Objective Components. We considered three different versions for ablating components: BAL-PM (ours), which follows Equation 5 exactly; a version with No Uncertainty Score in the objective; and another version with No Entropy Score. Figure 8 shows the findings. In the datasets considered, both terms of the objective play a crucial role in the performance of BAL-PM. Disregarding the entropy score fundamentally means solely following BALD, which acquires several redundant samples. On the other side, disregarding the uncertainty score prevents the learner from acquiring points where the model lacks information. Type of Uncertainty. In Machine Learning, we identify two different sources of uncertainty: epistemic and aleatoric. Epistemic Uncertainty refers to the uncertainty in the parameters of the model, often due to the lack of information from some regions of the parameter space. In contrast, aleatoric uncertainty refers to the uncertainty in the data, originating from the inherent noise of the data generation process. We reduce epistemic uncertainty by acquiring new data, while aleatoric is irreducible.A common practice in Active Learning is to select points based on high Predictive Uncertainty, which is often referred to as "Uncertainty Sampling" [60] . This type represents the total uncertainty, i.e., it accounts for both epistemic and aleatoric sources. Therefore, we expect that following Predictive Uncertainty underperforms in datasets with high label noise, as the objective may favor points with high aleatoric uncertainty and low epistemic uncertainty.Figure 9 compares using Predictive and Epistemic uncertainties in the objective of Equation 5.Selecting points based on epistemic uncertainty strongly outperforms the other variant, which aligns with the fact that preference datasets contain high levels of label noise -as mentioned in Section 1, the agreement rate among labelers is low, typically between 60% and 75%. This ablation also highlights the importance of a Bayesian preference model for epistemic uncertainty estimation.Entropy Estimator. In Section 5, we argue for using the KSG entropy estimator (rather than the KL estimator) since it leverages the full dataset and better estimates the probability density in the feature space, leading to less biased entropy estimates. In this ablation, we compare both estimators to measure the impact of this design choice.Figure 10 presents the results of this ablation. In the Reddit TL;DR dataset, the KSG estimator consistently outperforms the KL estimator, requiring approximately 20% fewer samples. In the OOD setting, both estimators performed equally. This is expected once that the available pool and training set does not provide support in the regions of the feature space with out-of-distribution samples. In this Section, we review the derivation of the KL entropy estimator and highlight the main assumptions and how they impacted the design choices for BAL-PM. We mostly follow the derivation from Kraskov et al. [54] .We start defining X as a continuous random variable in a vector space where the Euclidean norm ∥x -x * ∥ is well-defined (x and x * are two realizations of X). Let µ(x) represent the density of X over this vector space. The Shannon entropy is defined as:H(X) := -E µ(x) [log µ(x)] = -µ(x) log µ(x)dx.To build an estimator, we can approximate Equation 12via Monte-Carlo sampling:EQUATION)where N is the number of samples for approximation and μ(x) is an estimate of the density of X.The goal of kNN-based entropy estimators is primarily to provide a good approximation for the density. For this matter, we first define a probability distribution P k (ϵ) for the distance between any realization x i and its k-nearest Neighbor. We start highlighting the first assumption: Assumption 1. The probability P k (ϵ)dϵ is equal to the chance of existing one point such that ∥x -x * ∥ < ϵ/2, k -1 other points with smaller distances, and N -k -1 points with larger distances.Following this assumption we can obtain the following trinomial distribution:EQUATIONwhere p i (ϵ) is the probability mass of the ϵ-ball centered in x i . The expectation of log p i (ϵ) is:EQUATIONwhere ψ denotes the digamma function. We now highlight the second assumption: Assumption 2. The density µ(x) is constant in the ϵ-ball.Assumption 2 allows us to approximate p i (ϵ) ≈ c d ϵ d µ(x i ), where d is the dimension of x and c d is the volume of the d-dimensional unit ball. Using Equation 15 in this approximation and rearranging terms, we have:EQUATIONFinally, using this estimator in Equation 13, we obtain the KL entropy estimator in Equation 4.Remarks. Now, we analyze how this derivation and assumptions impact our entropy estimator. First, Assumption 1 models the probability based on the choice of k. For the low-data regime (i.e., N is small), this could lead to considerably large ϵ-balls where the Assumption 2 does not hold, and, therefore, it is not a good approximation. Thus, naively applying the KL estimator in the acquired training set may lead to strongly biased entropy estimates.Secondly, in Section 4, we raise the key insight that Equation 4 holds for any value of k, and it does not require a fixed k over different particles for entropy estimation. Indeed, the density estimation μ(x i ) is estimated for each particle x i in isolation (Equation 16). Therefore, we may choose a different k for each particle to ensure that Assumptions 1 and 2 are valid.Balancing Task-Dependent and Task-Agnostic Epistemic Uncertainty for Active Learning. Since considering the information in the feature space is crucial for Active Preference Modeling, a relevant question arises: how should an algorithm balance the contributions between the Bayesian preference model epistemic uncertainty and the prompt feature space uncertainty? Excessive reliance on the task-dependent term leads to acquiring redundant points. Similarly, the exacerbated contribution of the task-agnostic term prevents the acquisition of the points that reduce the uncertainty in the preference model. Thus, we investigate how BAL-PM balances these two terms over active learning.In Figure 11 , we show the ratio of the entropy and preference model epistemic uncertainty scores in the first selected point of each acquired batch. Interestingly, BAL-PM automatically adjusts the contribution of each term. It progressively decays and converges the influence of the entropy score (task-agnostic source) as the novelty in the prompt feature space reduces due to the acquisition of new points. Similarly, it increases the relevance of the preference model uncertainty estimates (taskdependent source). A positive downstream effect is that BAL-PM switches to a more task-dependent selection as it improves the Bayesian model and, consequently, its epistemic uncertainty estimates. G Is Log-Likelihood a proper performance measure for Preference Modeling?In this Section, we argue why the Average Log-Likelihood on the test set is a good performance measure for Preference Modeling. Given a test set D test = {(x, y 1 , y 2 , y 1 ≻ y 2 )} N and the learned preference model p θ (y 1 ≻ y 2 | x, y 1 , y 2 ), the average Log-Likelihood is given by:EQUATION)Equation 17 is exactly the objective maximized in standard binary classification (or, equally, the minimization of the negative Log-Likelihood loss) but computed over the test data. In other words, this is the negative "test loss".Average LL is a typical metric in the Active Learning and Uncertainty Quantification literature [61] [62] [63] . For Preference Modeling, it is very relevant as LL directly accounts for the preference strength to rank models: given a triple (x, y 1 , y 2 ) where all raters agree that y 1 is preferable over y 2 , LL allows us to measure that a model A predicting p A (y 1 ≻ y 2 | x, y 1 , y 2 ) = 0.9, (LL ≈ -0.1) is better (in that data point) than another model B predicting p B (y 1 ≻ y 2 | x, y 1 , y 2 ) = 0.6 (LL ≈ -0.5).Accuracy would provide an equal score for both models since it only accounts for the binarized prediction. LL provides a more "fine-grained" measure.Another crucial point is that LL factors in the aleatoric uncertainty in the label-generating process. For instance, in a scenario where only 70% of the raters agree that y 1 is preferable, LL better ranks models whose predictions are closer to p = 0.7, respecting the ground truth preference strength, which is not possible with accuracy.G.1 Do the models better ranked by Average Log Likelihood (LL) lead to better fine-tuned policies?In the context of Preference Modeling, fine-tuning LM policies is currently a very relevant downstream task. The Preference Modeling optimization objective and model selection protocol adopted in this work follow exactly the prior influential work on the topic [6, 1] , which provides evidence that better preference models (in terms of validation loss) lead to improved downstream policies. Thus, we expect our models to behave similarly under the same conditions. Figure 12 : The Relationship between the Test Log-Likelihood of a Preference Model and the Performance of the corresponding fine-tuned Policy. We show that, under simple conditions, there is a strong correlation between these two performance measures.As additional evidence, we empirically illustrate the relationship between Log-Likelihood and policy performance on a simplified setup (Figure 12 ). Here, prompts x and completions y are real numbers in [0, 1]. The ground-truth reward function is given by a Gaussian density function r(x, y) = N (x + y | µ = 1.0, σ = 0.4), and true preferences follow the Bradley-Terry model. In this setup, we progressively increase the training set size (the x-axis in Figure 12a ) at which we train the preference models. This process generates different models with increasing levels of test-set average Log-Likelihood. Then, similar to [64] , we optimize the base policy via a Best-of-N optimizer by leveraging each of these learned preference models. Finally, we report the rate at which the fine-tuned policy completion is preferable over the base policy completion according to the ground truth reward model ("win rate"). Although simple, this setting allows us to bypass several optimization and distributional challenges and solely focus on evaluating the relationship between average Log-Likelihood and the performance of the fine-tuned policy. Figure 12a reports the Log-Likelihood (red) and the win rate against the base policy (blue). Figure 12b directly plots both measures and fits a regression line. We observe a strong correlation, which aligns with our point: a higher test-set average Log-Likelihood means that the preference model is better at predicting the ground truth preferences, assigning higher rewards for better completions, and, therefore, improving fine-tuned policies that maximize such reward scores.In this section, we describe the computational cost of executing BAL-PM. We argue that computational tractability is one of the main contributions of our method. We start by providing some context: our work focuses on (Bayesian) Active Learning, which is naturally more computationally demanding than simply training predictive models. This is because we require models that express epistemic uncertainty to acquire informative labels for efficient training. This also requires models to constantly update their uncertainties based on the new data via re-training. The key is that Active Learning reduces the number of labels needed to train a better model, which considerably overcomes the extra computational cost. Labeling is significantly more expensive and laborious.As described in Section 1, Preference Modeling in LLMs requires batch acquisition -it is impossible to request the label of a single point, re-train the model, and repeat this process. Still, tractable methods rely on these single-point acquisition objectives. Thus, what BAL-PM does computationally is to replace B model re-trainings per acquired batch with computing entropy estimates (considerably cheaper, as explained below). B is the batch size, and we set B = 320 in our experiments.BAL-PM does not require training or inference on LLMs during the active learning loops. This considerably reduces the computational cost and allows us to scale up to 140b models in a single A100 GPU. Comparatively, even fully fine-tuning a 7b-parameter model currently requires at least several A100 GPUs. LoRA methods [65] also require new LLM inferences for every model update, while BAL-PM only requires a single time.The computation of BAL-PM has three pieces: offline processing (LLM inference and kNN computation), updating adapters, and entropy estimation. LLM inference is done only once before Active Learning, which is the minimum for LLM adoption. Furthermore, we may compute the features used for the preference model and sentropy estimation in the same forward pass: every promptcompletion input concatenates prompt/completion texts. Thus, we can extract prompt features as the last layer embedding right after the last prompt token and the prompt-completion features right after the completion's last token. Hence, there is no extra cost to extract features for entropy estimation. The cost of updating adapters is minimal: it boils down to updating MLPs with two hidden layers, which is reasonably cheap for LLM research. Lastly, the entropy estimation only requires computing the di-gamma function (Equation 11) in the pool.In this Section, we introduce a robustness analysis for the β hyperparameter (Figure 8 ) considering the values in the search space, described in Table 2 . As presented in Equation 5, this hyperparameter balances the effect of the epistemic uncertainty and entropy scores. Figure 13 : β Robustness Analysis. We considered different scales of the β hyperparameter in the BAL-PM objective (Equation 5).The impact of the choice is more noticeable when values are 100x greater/lower than the optimal choice. Values around 10x greater/lower still perform well, suggesting good room for choosing this hyperparameter. Furthermore, we employed the same value of β across the different datasets and LLMs, suggesting robustness across different relevant dimensions. Crucially, β trades off the contribution of two different terms. As such, it provides a spectrum of objectives and may recover the two extremes presented in the ablation of Figure 13 . Naturally, different choices of β will change the uncertainty score ratio in Figure 11 on Appendix F (i.e., the contribution of each term after convergence). Nevertheless, and most importantly, the behavior of the curves -the entropy contribution progressively reducing and converging and the relevance of epistemic uncertainty estimates increasing -should remain. In this section, we provide additional baselines for further comparison.We start by evaluating the performance of a preference model trained in the full dataset. Figure 14 presents this result in purple, with the shaded area representing the standard error computed across five seeds. BAL-PM achieves onpar performance to this baseline, although it only requires 24000 data points (the full dataset contains 92,858 points). This result is another strong evidence of the sample efficiency of our method. We considered the following additional baselines:• Entropy Minimizer: Entropy Minimizer: Inspired by Liu et al. [66] , we consider an objective that, in addition to selecting points with high epistemic uncertainty, also selects points that are semantically similar to the current training points. This is equivalent to selecting points that increase the entropy of the prompt distribution the least, thus the name "Entropy Minimizer". It serves as a check for our central hypothesis that entropy maximization leads to better batch active learning. • Perplexity: Inspired by Gonen et al. [67] , we consider an objective that selects points based on the perplexity of the base LLM. We consider two versions: one that chooses points with lower perplexity (Low Perplexity) and another with higher perplexity (High Perplexity). This is an interesting baseline since perplexity is equivalent to the predictive entropy of the token distribution. Therefore, it helps to analyze how much the base LLM "knows what it does not know" in terms of preference modeling. Figure 16 : The effect of incorporating the entropy objective in uncertainty baselines. This shows how our proposed objective can also boost the performance of different baselines.Figure 15 reports performances for both test and OOD sets. In both cases, BAL-PM outperforms additional baselines. In the sequence, MC-Dropout works best as the baseline that targets the epistemic uncertainty of a Bayesian model. Unsurprisingly, Entropy Minimizer and Low Perplexity work worse since they target points with lower entropy. LRU presented mixed results, suggesting that the latent reward may not represent well the preference model's uncertainty. More interestingly, while these models can represent different uncertainties to seek informative points, they naturally cannot provide in-batch diversity -they suffer from the same challenges as BALD. In this perspective, the BAL-PM objective can also improve upon those methods, as we show in Figure 16 : we combined MC-Dropout and LRU with our entropy term to provide in-batch diversity, which consistently improved both methods across the datasets.In this Section, we present some selected samples of the first acquired batch from BALD (Table 3 ) and BAL-PM (Table 4 ) for comparison. We sorted the points alphabetically to highlight duplicated prompts. BALD consistently acquires duplicated points, sometimes sampling more than ten times the same prompt. In contrast, BAL-PM samples semantically diverse points with no duplicates. Justification: See Sections 4 and 5.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
log likelihood: 
 Random Sampling BALD BAL-PM (ours)Figure 1 : Log-Likelihood of learned preference models in the Reddit TL;DR dataset [1] . Our method, BAL-PM, reduces the volume of required human feedback by 33% over random acquisition.Preference Modeling is a key component to aligning unsupervised pre-trained Large Language Models (LLMs) towards human preferences [1] [2] [3] [4] . It is often performed by collecting human feedback for a set of prompt-completion pairs and then leveraging the data to steer the behavior of such models, either directly [5] or via reward models [6] . Nevertheless, human feedback generation is laborious [7] , especially when it requires specialized knowledge [8, 9] . Furthermore, the quality of the prompts has a crucial impact on the performance of fine-tuned models [10] . Hence, selecting the most informative points to gather feedback is essential to reduce costs and enable better LLMs.Despite its substantial impact, data selection for Preference Modeling poses a significant challenge. The prompt-completion pool is arbitrarily Figure 2 : An illustration of how BAL-PM works. For each tuple (x, y 1 , y 2 ) ∈ D pool , we obtain features for the prompt and prompt-completion pairs by computing the last layer embeddings of the base LLM. We leverage the prompt feature space to estimate the entropy score of the acquired prompt distribution, Ĥ(X train ∪ {x}). Similarly, we use the prompt-completion features as input for the Bayesian Preference Model, which is used to estimate task-dependent epistemic uncertainty scores, Û (x, y 1 , y 2 ). BAL-PM selects the tuple that maximizes the linear combination of both scores.large and semantically rich. Additionally, human feedback is inherently noisy, with low agreement rates among labelers, typically observed between 60% -75% for these settings [6, 1, 11, 12] . Lastly, the intrinsic scale of LLM development requires parallelized labeling and makes frequent model updates prohibitively expensive, limiting the applicability of many active learning schemes that rely on single-point acquisition [13] .Bayesian Active Learning provides a principled approach to data selection [14] [15] [16] , which has demonstrated remarkable success across different fields [17] [18] [19] . However, its application in Active Preference Modeling is not straightforward. Past attempts of employing the framework in this setting reported no benefits over random selection [20] , arguably due to poor uncertainty estimation in the context of LLMs, which is indeed an open challenge and active area of research [21] .We identify two reasons for this phenomenon. First, the inherent bias of approximate Bayesian inference in deep learning models, particularly for LLMs. Second, and more nuanced, the current intractability of epistemic uncertainty estimation methods in Preference Modeling for LLMs, a context that intrinsically requires batch acquisition. Proper estimators for this setting present combinatorial complexity, and even greedy approximations are still computationally demanding and impractical [13, 22] . This limitation leads to relying on simpler single-point acquisition schemes such as BALD [23] (as in Gleave and Irving [20] ), designed to acquire individual points followed by model updates. However, these assumptions are far from realistic for the scale of Preference Modeling in LLMs, and naively applying such methods for batch acquisition leads to the selection of redundant samples.In this work, we argue that leveraging the information available from the feature space spanned by the LLM -a task-agnostic 3 source of epistemic uncertainty -alleviates these problems. We propose Bayesian Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition policy that not only targets points of high epistemic uncertainty according to the preference model but also seeks to maximize the entropy of the acquired prompt distribution in the feature space. This entropy score encourages the active learner to select prompts from low-density regions, effectively reducing the feature space epistemic uncertainty [24] . As a result, it promotes diversity in the acquired training set, preventing the selection of redundant samples and also helping in learning a better Bayesian preference model and its task-dependent epistemic uncertainty estimates for subsequent acquisitions. Figure 2 illustrates how BAL-PM works.We conduct active learning experiments in the Reddit and CNN/DM preference datasets [25, 26, 1] to validate our method. BAL-PM demonstrates strong gains over random sampling, reducing by approximately 33% (as shown in Figure 1 ) and 68% the volume of feedback required to learn the preference model in the considered datasets. It also consistently surpasses other strong stochastic Bayesian acquisition policies [22] . Finally, we further analyze the acquired prompt distribution to show that BAE-PM prevents redundant exploration and effectively balances the contribution of the two sources of epistemic uncertainty. 
 
 Appendix:  


Decision: This paper proposes to use active learning techniques to select examples for querying preference ratings for use in preference modeling algorithms. The main contention of the author is that paying attention only to epistemic uncertainty can result in collecting redundant batches of data and thus propose to also pay attention to the diversity of the data as characterized by "maximiz[ing] the entropy of the acquired prompt distribution in the feature space". Reviewers found the results to be strong/impressive/comprehensive, to be shown to scale, the paper to be clearly written. They also raised some concerns about the metric which doesn't actually show that the resulting data here led to more effective training of LLMs and the "link to the real-world problem of constructing a preference model in a sample-efficient fashion in the absence of a pool of labelled prompt-completion pairs". Following rebuttals, the mean score of the paper is a 6 and given the presence of champions and lack of a strong detractor I am recommending acceptance.

Reviews: review: paper_summary: paper_summary: This paper presents BAL-PM, a Bayesian active learning framework for the training preference model. The authors propose an acquisition policy that seeks examples that have high epistemic uncertainty and can maximize training data’s entropy. Specifically, the epistemic uncertainty is estimated by the training preference model and entropy is estimated by the base LLM. Experiments on two preference datasets show that the proposed method requires fewer training data compared to previous methods (33% and 68% reduction on each dataset accordingly). The author also shows that their method is scalable to larger LLMs.
strength_weakness: strength_weakness: - The evaluation metric is limited. The authors only show the log-likelihood of the test sets. This single metric has no guarantee that preference models trained by this method have a more accurate judgment (shown by pair preference accuracy) and lead to better fine-tuned LMs (shown by tuning with the trained preference model)
questions: questions: - How do the results compare to models trained on the whole dataset?
- Do the trained models lead to better LMs?
limitations: limitations: The authors have properly adequately the limitations of their work in the paper.
paper_summary: paper_summary: This paper proposed a novel framework to select the most informative preference data for training based on Bayesian active learning. 
To collect the prompt-response pair (x,y), it firstly selects the prompt based on Bayesian Active Learning by Disagreement by maxmizing the information gain. Then, the selection of response considers both the preference model epistemic uncertainty estimation and entropy estimation for the acquired prompt distribution, which addresses the challenges of selecting diverse samples. When evaluated on Reddit and CNN/DM preference datasets, it saves 33% and 68% training samples by comparing to random sampling.
strength_weakness: strength_weakness: 1. Lack for diversity-based and uncertainty-based sample selection methods, such as [1,2,3,4], which makes the technical contributions less reliable,
2. As the method is based on feature extraction from existing LLMs, the computation cost in estimating the diversity and uncertainty is not clear, which inhits its pratical application.


References:
[1] What Makes Good In-Context Examples for GPT-3?
[2] An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
[3] Demystifying Prompts in Language Models via Perplexity Estimation
[4] Diverse Demonstrations Improve In-context Compositional Generalization
questions: questions: See the details in Weakness.
1. How are the performances of existing other baselines.
2. What are the computation cost in estimate the utility score, diversity and entropy.
limitations: limitations: Yes, the authors mentioned the intrinsic limitation of the transformer's embedding, i.e., noise-TV. The impropriate selected samples can be toxic, unfair or not well-represented.
paper_summary: paper_summary: &nbsp;

The authors present a method, Bayesian Active Learning for Preference Modeling (BAL-PM) that seeks to learn a preference model in a sample-efficient manner within an active learning setting. A key insight by the authors is the consideration of task-dependent and task-agnostic uncertainty to encourage diversity in the sampled prompt-completion pairs. My main concern with the approach is the link between the pool-based active learning setting and active learning over an open-ended and continuous prompt space which I will describe below. As such, my recommendation is borderline pending the authors rebuttal on this point. 

&nbsp;
strength_weakness: strength_weakness: &nbsp;

__MAJOR POINTS__

&nbsp;

1. My main concern with the current work is the link to the real-world problem of constructing a  preference model in a sample-efficient fashion in the absence of a pool of labelled prompt-completion pairs. Specifically, as I understand, one of the main motivations for being sample efficient in constructing the preference model is the cost of human labelling. In the experiments considered by the authors, the datasets already contain labels. As such, I would ask  a) what is the motivation for being sample-efficient in the pool-based active learning setting? b) assuming the authors feel performance on the pool based setting is representative of an open-ended active learning problem, how would they go about verifying that BAL-PM performs well in the non pool-based setting?

&nbsp;

__MINOR POINTS__

&nbsp;

1. It would be great if the references appeared in numbered order.

2. There are some missing capitalizations in the references e.g. "Bayesian".

3. The arXiv identifier is missing for reference 2.

4. When referencing PyTorch, [1] should be used rather than the earlier workshop paper version.

5. It would be great if the authors could provide instructions for reproducing the experimental results in the README of their GitHub repository.

6. Reference 7 was accepted at TMLR.

7. Line 38, I would not expect the statistic that agreement rates among human labelers is typically 60-75% to be robust across all problem settings. It would be worth qualifying this statement.

8. The arXiv identifier for reference 18 is missing.

9. "Prompt-completion pairs" would potentially be a more appropriate terminology in place of "prompt-answer" pairs.

10. On line 105, S is not defined, this should presumably be \Chi.

11. Line 134 typo, "an unsupervised".

12. In Figure 4, it would be great to include in the caption the number of random trials for which the errorbars are computed.

13. In the abstract the authors do not mention that the figures of 33% and 68% apply to a random sampling baseline.

&nbsp;

__REFERENCES__

&nbsp;

[1] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L. and Desmaison, A., 2019. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32.

[2] Yang, A.X., Robeyns, M., Coste, T., Wang, J., Bou-Ammar, H. and Aitchison, L., 2024. Bayesian reward models for LLM alignment. arXiv preprint arXiv:2402.13210.

&nbsp;
questions: questions: &nbsp;

1. In Section 4, the authors mention that their Bayesian preference model is constructed using an ensemble of adapters. What was the motivation for this choice over Laplace-LoRA [2] for example?

2. In Figure 7, random sampling outperforms BAL-PM for the 140b parameter model early in the active learning trace. Do the authors have an explanation for why this might be the case?

3. It would be interesting to see the full details of the validation procedure for choosing the hyperparameters, most notably the entropy term \Beta. How sensitive was performance to the choice of \Beta hyperparameter? What is the link to Section F of the appendix?

4. In terms of the results presented in Figure 9, how do the authors compute the aleatoric uncertainty for the datasets? Are these uncertainty estimates provided in the datasets?


&nbsp;
limitations: limitations: &nbsp;

The biggest limitation I foresee with the current work is the potential for the performance of BAL-PM to carry over between the pool-based active learning setting and the non pool-based setting as described above.

&nbsp;
paper_summary: paper_summary: This work proposes BAL-PM, a stochastic acquisition policy aiming to select informative data samples (i.e., the prompt and its corresponding paired responses) that need to collect human feedback for LLMs' preference learning.
Specifically, BAL-PM mainly addresses the issue of sampling redundant data points in previous methods that rely solely on epistemic uncertainty, such as BALD. On top of the previous BALD acquisition method, BAL-PM employs a task-agnostic entropy estimator for the prompt distribution to enhance the diversity of acquired data samples.
Experiments on the Reddit and CNN/DM datasets under a pool-based setting demonstrate the effectiveness of BAL-PM over random sampling and the BALD baseline.
strength_weakness: strength_weakness: - The experiment part is not compelling and clear enough to me. Regarding the reported log-likelihood metric, it is unclear to me how it is calculated (e.g., on preferred response?) and why it is the only metric used for evaluating preference modeling. It seems to me that it can not measure to what extent the chosen response is preferred over the rejected one. Regarding the datasets evaluated, the task is limited to text summarization. The experiment part can be improved by adopting more metrics on more general-purposed datasets/tasks to illustrate the effectiveness of the proposed method.
- While encouraging data diversity is plausible and effective in LLM's preference learning according to the experiments, the necessity of using the feature of the base LLM to measure the entropy of the prompt distribution is unclear to me. Although it is mentioned as a common approach in lines 97-99, the motivation for focusing on feature space is still somewhat vague to me and may introduce extremely high computational overhead in the context of LLM's preference learning.
- The proposed BAL-PM, which requires maintaining/updating an ensemble of adapters for collecting each batch, may be difficult to practice in large scale.
- The discussion of prior works is inadequate. Besides the difference in configuration, the technical parts of other related active preference optimization of LLMs should be discussed more.
questions: questions: - What is the advantage of adopting the (static) feature space of the base LLM for estimating the entropy of the prompt distribution? Is there any evidence to demonstrate that it is superior to other approaches, such as using smaller proxy models (e.g., BERTScore) or simpler fashions?
- I am curious about whether different adapters can make diverse predictive distributions with limited expressive power. Is there any empirical evidence for this?
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.

Comments: title: Many Thanks to the Authors for their Rebuttal
Reply to the Rebuttal
References
Global Rebuttal References
Rebuttal References
comment: Thank you for your response and the additional results. My concerns are addressed. I'll maintain my overall rating as I've already given an acceptance score. But I will increase my Soundness rating.
&nbsp;

Many thanks to the authors for their rebuttal.

1. In terms of the pool-based vs. non-pool-based setting, I think it would be beneficial to include a preference simulator for the non-pool-based setting which would a) emulate an open-ended active learning loop b) allow some simulated notion of ground truth aleatoric uncertainty to be present. I would encourage the authors to consider this for the camera-ready version of the paper.

2. Many thanks for the clarification on the use of adapters in place of Bayesian LoRA. The intuition regarding finetuning the base LLM makes a lot of sense and I can see why this would be an important factor to stabilize the surrogate in an active learning loop.

3. The intuition for the performance of BAL-PM relative to random sampling early in the trace makes a lot of sense.

4. Many thanks for the sensitivity analysis on the \Beta parameter.

Given the above, I am happy to raise my score but I would strongly encourage the authors to think about including an additional experiment emulating an open-ended active learning loop with a preference simulator. I think this would "complete" the paper in the sense that both a) the pool-based setting with realistic human preferences and b) the non-pool-based setting with synthetic preferences are considered.

&nbsp;
The authors did a good job of addressing most of my concerns and making clarifications regarding their method and evaluation. However, I strongly encourage the authors to validate the effectiveness of BAL-PM on a broader range of tasks/datasets, especially given the notable computational efficiency of the proposed method and the moderate scale of common academic preference datasets. In my humble opinion, the current paper is a nice technical contribution with fair empirical significance to the community.
Thanks for the detailed responses, which largely address my concern. I will raise the score accordingly.
[1] Smith et. al. Prediction-Oriented Bayesian Active Learning. AISTATS, 2023.

[2] Imberg et. al. Optimal sampling in unbiased active learning. AISTATS, 2020.

[3] Gleave et. al. Uncertainty Estimation for Language Reward Models, 2022.
[1] Kossen et. al. Active Testing: Sample–Efficient Model Evaluation. ICML, 2021.

[2] Lakshminarayanan et. al. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS, 2017.

[3] Gal et. al. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML, 2016.
[1] Stiennon et. al. Learning to summarize with human feedback. NeurIPS, 2021.

[2] Wang et. al. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. NeurIPS, 2019.

[3] Osband et. al. Epistemic Neural Networks. NeurIPS, 2023.

[4] Dwaracherla et. al. Efficient Exploration for LLMs. ICML, 2024.

[5] Gal et. al. Dropout as a Bayesian Approximation:
Representing Model Uncertainty in Deep Learning. ICML, 2016.