Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Authors: Yunyi Shen
Hao Sun
Jean-Francois Ton
Venue: ICML 2025 poster (2025)
Keywords: Reward modeling
active learning
LLM alignment
Primary area: deep_learning->large_language_models
TL;DR: We propose a last-layer Fisher information-based method to choose comparisons for reward modeling.

Abstract: Building neural reward models from human preferences is a pivotal component in reinforcement learning from human feedback (RLHF) and large language model alignment research. Given the scarcity and high cost of human annotation, how to select the most informative pairs to annotate is an essential yet challenging open problem. In this work, we highlight the insight that an ideal comparison dataset for reward modeling should balance *exploration of the representation space* and make *informative comparisons* between pairs with moderate reward differences. Technically, challenges arise in quantifying the two objectives and efficiently prioritizing the comparisons to be annotated. To address this, we propose the Fisher information-based selection strategies, adapt theories from the *classical experimental design* literature, and apply them to the final linear layer of the deep neural network-based reward modeling tasks. Empirically, our method demonstrates remarkable performance, high computational efficiency, and stability compared to other selection methods from deep learning and classical statistical literature across multiple open-source LLMs and datasets. Further ablation studies reveal that incorporating cross-prompt comparisons in active reward modeling significantly enhances labeling efficiency, shedding light on the potential for improved annotation strategies in RLHF. Code and embeddings to reproduce all results of this paper are available at https://github.com/YunyiShen/ARM-FI/.

Active Reward Modeling:
Adaptive Preference Labeling for Large Language Model Alignment
Yunyi Shen
Hao Sun
Jean-François Ton
Abstract
1. Introduction
The safe and successful deployment of Large Language Models (LLMs) across various application domains requires alignment with human values. Current research working on LLM alignment mainly focuses on reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Sti- ennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a), which rely on preference-based annotations provided by human annotators (Bai et al., 2022b). However, obtaining human feedback can be expensive, and the noisy and binary nature of such data often limits its information density, pos- ing challenge for effective reward modeling (Wang et al., 2024a; Liu et al., 2024).
Building neural reward models from human pref- erences a pivotal component in reinforcement learning from human feedback (RLHF) and large language model alignment research. Given the scarcity and high cost of human annotation, how to select the most informative pairs to annotate is an essential yet challenging open problem. In this work, we highlight the insight that an ideal comparison dataset for reward modeling should balance exploration of the representation space and make informative comparisons between pairs with moderate reward differences. Technically, challenges arise in quantifying the two objec- tives and efficiently prioritizing the comparisons to be annotated. To address this, we propose the Fisher information-based selection strategies, adapt theories from the classical experimental design literature, and apply them to the final lin- car layer of the deep neural network-based re- method ward modeling tasks. Empirically, demonstrates remarkable performance, high com- putational efficiency, and stability compared to other selection methods from deep learning and classical statistical literature across multiple open- source L.Ms and datasets. Further ablation stud- ies reveal that incorporating cross-prompt com- parisons in active reward modeling significantly enhances labeling efficiency, shedding light on the potential for improved annotation strategies in RLHF, Code and embeddings to reproduce all results of this paper are available at https: / //githuts com www//oom//i//eeeeeeeeeeeaeeaaw Swwnnnnnnnnn//aaieeieee FAReee//R//F///
Active learning, where the model queries most informative labels based on its current state, offers potential solution. typically involves three key components: an initial model, a query strategy often in the form of maximizing scoring function over unlabeled data, and pool of unlabeled data. The model selects subset of data for labeling and retrains iteratively until a stopping criterion is met.
In this work, we study the problem of active data acquisi- tion in reward modeling. Technically, we introduce various scoring rules inspired by both classical experimental de- sign (Chaloner & Verdinelli, 1995) and recent deep learning- based advancements (Sener & Savarese, 2017; Houlsby et al., 201 I; Kirsch et al., 2019). We adapt those methods to the learning of Bradley-Terr (BT) reward models (Bradley & Terry, 1952), which have been successfully applied in large-scale alignment practices (Ouyang et al., 2022; Tou- vron al., 2023) and proven be theoretically sound (Sun et al., 2025a).
We benchmark 8 scoring algorithms using 2 datasets and 3 LLMs, ranging in size from 2B to 8B, and evaluate wide range of active learning setups. Our results show that two classical experimental design methods - applied to the final linear feature layer of deep models - achieve state- of-the-art performance and strong stability across different setups, model architectures, and datasets.
'Equal contribution 'ECSS, MIT, Cambrdige, MA, USA Universityy of Cambridge, Cambrdige, UK ByteDance Re- search, London, UK. Correspondence to: Yunyi Shen <yshen99@mit.cdu>>> Hao Sun <hs789@cam cau.aa.uk. Jean- François Ton <jeanfrancois catefrancooisbyyyyyeecc...ommm bytedance.com>>
Our main contributions can be summarized as follows:
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Formally, we characterize the problem of optimal prefer-

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
ence label annotation using embedding space BT regres- sion framework and establish connections between active learning and classical experimental design literature under the BT context.
tations onto an embedding & ( 11.1). This is mild assumption since these embeddings can be, for example, concatenation of word embeddings, the output of tokenizers, or the output embedding of an LLM.
Methodologically, we introduce a set of algorithms in- spired by classical experimental design literature, adapt them for deep BT regression models, and develop an efficient gradient approximation for the associated combi- natorial optimization challenge in large-scale alignment problems.
When there is no risk of confusion, we denote the embed- dings of pair i as 1,, E RP and 22, E RD, with a reward function : RD - R. The goal is to learn this function from annotations. In the BT model, we assume that
h2 ~ Bernoulli (a(r(W;.)) r( r(
(I)
Empirically, we evaluate different methods for preference label annotation across diverse setups, datasets, and base models. Our results suggest that applying classical ex- perimental design techniques to the final layer of deep neural network yields strong performance and stability.
with a being the sigmoid function.
Active learning. In a typical active learning setting, we have labeled dataset of size, D, (Z Vi,1, 22 Ur ha) with sample size I, at step and a ttpicaaly large pool of unlabeled data, P. = (I 1 15, (,,,,,,,,,,5.). V3:2), with sample size J2 at step s. The goal is to select a small subset C, C Pa. subject to certain constraints, for labeling. Once labeled (denoted as C.), this subset is added to the labeled dataset to train the next iteration of the model.
2. Background and Setup
Reward modeling in alignment. Reinforcement learning is a key technique for aligning LLMs to ensure their safe and effective deployment (Christiano et al., 2017; Ouyang et al., 2022; Stiennon et al., 2020). The most prevailing approach, RLHF, relies on reward models as fundamental mechanism for quantifying content quality and scaling the reinforcement learning (Lambert et al., 2024; Wang et al., 2024a). During fine-tuning and deployment, reward models serve as proxies for human evaluators (Dubey et al., 2024; Dong et al., 2024; Wang et al., 2024b), assessing how well LLM outputs align with human intent. Despite significant progress, reward modeling remains challenging due to the searcity and inaccuracy of annotations (Lambert et al., 2024; Wang et al., 2024a; Gao et al., 2023). Prior research has attempted to mitigate these challenges through different aspects when learning from fixed set of annotations (Wang et al., 2024b; Winata et al., 2024; Liu et al., 2024; Lou et al., 2024; Coste et al., 2023; Zhang et al., 2024). While Xiong et al. (2023); Dong et al. (2024) demonstrate that online annotations are more efficient in RLHF, the topic of online annotation prioritization strategy remain under- explored except for heuristic designs (Muldrew et al., 2024).
We also consider this process in the embedding space, where the labeled and unlabeled sets are given by D, { 2,,,, (M,,,hh))<r h= and P. = { V ical model-based active learning procedure is outlined in Algorithm 1. In this work, we focus on identifying the best perforerfonin scoring rules.
3 A typ-
Algorithm Model-based active learning
Require: initial labeled dataset Do, pool set Pa. model Mg. a scoring rule S, budget constrain c, and number of rounds
1: RETURN Last trained Mn
2: for s x 1 to n do
3: generate pool P.
4: C, +
5: get labeled dataset C,
6:
D. + C. D
7: train model M, using D.
8: end for
9: return Mn
Bradlcy-Terry model for reward modeling. A canon- ical model used for reward modeling from binary pref- erence data is the Bradley-Terry (BT) model (Bradley & Terry, 1952), or more precisely, its regression variant (Sun et al.. 2025a). In the most general setting, which allows for cross-prompt comparisons, a human annotator is pre- sented with two pairs of prompts and responses, (zv,1, 1 and (Ii.2, ):. The annotator then provides a preference, h4 = 1 (( 1 (2)))) indicating whether the first pair is preferred over the second.
In literature, Muldrew et al. (2024) considered active learn- ing and proposed a strategy that combines entropy with model certainty (which is equivalent to the maxdiff strat- egy in our notation). For non-binary data, Mukherjee et al. (2024) suggested maximizing the determinant of the fea- ture matrix. BatchBALD (Kirsch et al., 2019) is a general- purpose active learning algorithm that requires a Bayesian model. The scoring in this method aims to maximize the expected entropy reduction by selecting the most informa- tive data points. Experimental design for generalized linear models has been extensively studied in the classical sta- tistical literature, with logistic regression serving as a key
Often, both responses correspond to the same prompt, i.e., J 2i however, Bradley-Terry regression can operate without this assumption. The model regresses these anno-
2

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
example (see e.g., Chaloner & Verdinelli, 1995; Sener & Savarese, 2017). Under the assumption of a linear reward function, the Bradley-Terry (BT) model simplifies to logistic regression. We acknowledge concurrent work Feng et al. (2025) also used Fisher information (FI) as one of their tools in sample selection for DPO. Feng et al. (2025) used Fl of all parameters in their neural net to establish theory on their sampling based algorithm but did not directly as optimiza- tion target in sample selection. While this paper compared many different strategies empirically and suggests determi- nant of Fl of last layer in BT as the optimization target- we believe this last layer choice acts as regularization to avoid over optimization. We focused on reward modeling instead of an entire DPO workflow to better separate source of gain and for flexibility of downstream RL workflow. Importantly Feng et al. (2025) only choose response conditioning on prompts while we also target on choosing best prompts via joint prompt-response selection.
The Fisher information matrix can be interpreted the met- ric tensor in Riemannian manifold of distributions, where the distance between them is given by the symmetrized KL divergence (Costa et al., 2015). FI quantifies the amount of information in the dataset for estimating the parameters B.1. From a Bayesian perspective, the Bernstein-von Mises theo- rem (Van der Vaart, 2000, Ch. 10.2, Thm 10.1) states that I is also the asymptotic covariance matrix of the poste- rior distribution of B.,, assuming mild regularity conditions the prior.
The FI can be viewed as a sum over all independent data points' contribution. For each data point, there are two terms multiplied together: the empirical covariance of embedding differences ( +b:2) (,,, - .2,2) and p1 - pc), the variance of the comparison results. Sun al. (2025a) suggested that improving the variance of comparisons can be interpreted as improving annotation quality which can also be seen from FI.
3. Designing of Comparisons
To make the FI large Equation (3) an ideal comparison should exhibit both a large variance in the embedding differ- ence (thus (...1 $1,2)
3.1. Linear BT Regression.
)2,2) T having large eigen- values) and a high variance in the comparison outcomes (thus p1 p.) large). This implics that the embedding space should be diverse, such that . $ captures a wide range of differences, and each comparison should be informative-not too close to 0 or 1. The former encourages exploration within the embedding space, leading to a better regression model, while the latter ensures that comparisons are not trivial, improving sample efficiency. An everyday analogy for comparing non-obvious pairs would be that comparing world champion to newbie in chess offers little insight into the abilities of either player.
Consider a simplified case where the true reward fune- tion is linear with respect to some intermediate embedding, r( 3., for weight vector B. We use & instead of Vi because the reward may not be linear with respect to the original embedding T used in reward modeling, and we wish to avoid confusion. The subscript -1 in B.1 reflects how we will apply these results in practice: c represents the output before the final linear layer, and 3.1 corresponds to the weight of this last layer. For now, we assume that this linear feature & is known to us. Note that there is bias term because linear BT is identified only up to translation.
The FI plays a crucial role in the classical theory of experi- mental design, both in frequentist and Bayesian frameworks, as highlighted by the Bernstein-von Mises theorem. This leads a family of design strategies known as alphabetical designs (Chaloner & Verdinelli, 1995; Pukelsheim, 2006).
Under this simplified setting the preference generating pro- cess of ith pair hi can be simplified to
h ~ Bernoulligo (( 2.2)) 1.1l
(2)
can be observed that this corresponds to a logistic regres- sion, where the covariates are the difference . P.2-
(Bayesian) D-optimality (Chaloner & Verdinelli, 1995). The alphabetical designs focus on the (co)variance of either estimating weights B.1 or making predictions under new embeddings, typically summarized through the covariance matrix. For example, the D-optimal design minimizes the determinant of the (asymptotic) covariance matrix of the last layer weights. B., Since |I-1| 1/|I], this is equivalent to maximizing the determinant of the FI.
By applying the theory from generalized linear models, we know that the maximum likelihood estimate B.1 is asymptot- ically Gaussian distributed, with mean B.1 and covariance matrix I-1, where I denotes the Fisher information (FI) matrix (see e.g., Shao, 2008, Ch. 4.5.2). For the linear Merlyy-Terry model, the Fl is
The Bayesian variant of D-optimal involves having prior contribution, such as maximizing II + I/02), where is the identity matrix, to avoid a determinant of zero. This corresponds to the inverse covariance matrix of the Laplace approximation of the posterior of (.,, assuming normal prior with variance o?.
I Σ 1=1
I
P.I
T ( 2T22) p1(1 - -p))
(3)
Where p. = o ( - T (.],in it can be observed that (1(1- - p.) represents the variance of a Bernoulli random variable.
3

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
A plug-in estimator of Pt. Ps. using the current best model, can be used to estimate the Fl (Chaloner & Verdinelli, 1995; Pukelsheim, 2006). In this approach, the scoring rule is the determinant of the Fisher Information matrix.
samples that the current reward model predicts to be the best and the worst, corresponding probabilities close to 0 1. This approach was used by Muldrew et al. (2024) to measure model certainties. The scoring rule to be maximized can thus be interpreted as difference in estimated reward [Ai,1 fi
(cop((C) = Σ
1 1, 1.11-1,2)) 2 ( ..,1
2122) ' (111-.)) (4)
iEC
Smaxdin (C)
Σ IEC
18,1 f1.2
(6)
In experiments, we refer to this strategy as D-opt. Other forms of optimality also exist, cach targeting different sum- maries of the Fisher Information (FI), such as A-optimality, which focuses on minimizing the trace of I- When the prediction of new, known embedding is the primary con- cern, G-optimality aims to minimize the variance of predic- tions on new embeddings.
This strategy encourages exploration in the reward space rather than the embedding space. It is sometimes used in ac- tive learning when the goal is to identify positive examples rather than the best classification (Settles, 2009). This justi- fies its use in reward modeling, where the goal is obtain responses that yield better rewards in downstream tasks. In our experiments, we refer to this method as Maxdiff. ff.
In this work, we suggest using D-optimality because it avoids the need to invert the FI, as required in A-optimality, and doesn't require specifying which samples to predict, as in G-optimality. For readers interested in further details, we refer to Pukelsheim (2006) (Ch. 9).
Optimizing Design Matrix (Mukherjee et al., 2024). This strategy focuses on finding the best collection of embed- dings, or the design matrix in statistics terms P. 2.2. without looking at model predictions. A common objec- tive is optimize the covariance matrix of the designs, ( T One approach is
The D-optimality strategy can be made a past-aware version by incorporating previously collected data. The asymp- totic covariance of the full data-conditioned posterior is then (Ipaat + -1, where Ipat is computed using prior data and Equation (3). This approach relates to Bayesian methods like Bayesian active learning by disagreement (BALD) (Houlsby et al., 2011), which minimizes poste- rior entropy. Since Gaussian entropy is proportional to the fog-determinant of its covariance. In our experiments, we refer to this variant as PA D D-opt.
Σ = E/= to maximize the determinant of E, [Σ], which encourages exploration over a large space of embedding differences. In fact, if we assume a linear regression model with additive Gaussian noise instead of logistic regression, this covari- ance matrix corresponds to the Fisher Information matrix of the regression coefficients, and this strategy aligns with the D-optimal design. The scoring rule is
Sxx(()= = -(O P D (P.) $1.2
Next, we review some other strategies that can be applied to BT models.
iEC
Mukherjee al. (2024) used a similar strategy for different type of preference data that is not purely binary. In our experiments, we refer to this method as det (XtX), for the determinant of XTX
Entropy Sampling (Settles, 2009; Muldrew et al., 2024). This strategy aims to select samples about which the current model is most uncertain (Settles, 2009). In the context of binary preference modeling, this corresponds to choosing data whose predictions P. are closest to 0.5, effectively exploring the level set of the reward. This is similar to a binary classification problem where the goal is to explore the decision boundary. This approach was also proposed by Muldrew et al. (2024) maximizing predictive entropy. The scoring rule is then.
Coreset (Huggins et al., 2016; Munteanu et al., 2018). Instead of minimizing uncertainty in parameter estimations, the Coreset strategy aims to find small subset of samples such that the trained model closely approximates the one trained on the full dataset, effectively transforming the prob- lem into a sparse approximation task on weighting data points. The Coreset method for logistic regression has been studied recently by Munteanu et al. (2018) and Huggins et al. (2016) in both frequentist and Bayesian settings. In our cupeeixpemmen experiment, we adopted the method of Huggins et al.
Sentropy (C) = E1- log - (1 1 1o)] - pe)] (5)
IEC
Since the entropy of a Bernoulli distribution reaches its maximum when p 0.5, this approach is equivalent to selecting the top pairs where the predicted probability is closest to 0.5. In our experiments, we refer to this method as Entro
(2016). The scoring rule does not have a simple closed-form solution, so refer interested readers to Huggins al. (2016) and denote it as S.oeeet. In our experiments, we refer to this method Coreset.
BALD and batchBALD (Houlsby et al., 2011; Kirsch et al., 2019). When transitioning from frequentist to Bayesian framework, BALD (Houlsby et al., 2011) and
Maximum Difference (Muldrew et al., 2024). Contrasting with entropy sampling, this strategy focuses on comparing
4

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
BatchBALD (Kirsch al., 2019) select data with high mutual information between the candidate batch's predic- tion and model parameters, making the data more infor- mative. Houlsby et al. (2011) showed that this approach maximizes expected posterior entropy reduction. This strat- egy applies to preference learning (Houlsby et al., 201 but requires a Bayesian model. We denote the corresponding scoring rule as SDBALD. In our experiments, we refer to this method as BatchBald. We used implementation in catchbaldddede (Kirsch et al., 2019).
3.3. Handling nonlinear model using last layer features.
For nonlinear reward models Equation (2), the dependen- cies embeddings become more complex. Strategies like maximum difference and entropy sampling, which depend only on model predictions, remain unaffected by the archi- tecture, while batchBALD is designed for (Bayesian) deep models. Feature-based methods like coreset or D-optimal need adaptation. A heuristic from the Bayesian last layer (Tran et al., 2019) and computer vision literature Sener Savarese (2017) suggests using the last layer before the linear output feature, applying linear strategies to it.
This strategy relates to Bayesian D-optimality; when poste- rior entropy is tractable, it can be minimized directly instead of relying on approximations from Houlsby et al. (2011). the posterior is Gaussian, entropy proportional to the log-determinant of its covariance, leading to D-optimality.
( Fo (V T
(11)
For some nonlinear function Fo parameterized by 0, e.g., an MLP and 4 :- ( We apply methods in linear settings with features ( ) We then train and B.1 together once data are labeled. In particular, in Sener & Savarese (2017) the nonlinear function Fg a CNN and they took a coreset approach. Here we apply this strategy to the coreset, optimal design matrix and D-optimal setting.
3.2. Gradient Approximation for Combinatorial Optimization.
In some strategies, we select data subset to maximize information criteria like the determinant of FI or the de- sign matrix. These often lead to intractable combinato- rial optimization problems. To address this, we use the sensitivity approach from the coreset and robustness liter- ature (Huggins et al., 2016; Campbell & Broderick, 2018: 2019). When the information criteria are expressed as a nonlinear function over sum of data point contributions, i.c.,
4. Illustrative Examples in Dimension Two
Experiment Setups this experiment, we provide a two- dimensional example of the comparisons made by each strategy. The ground truth reward was defined as the log probability of mixture of two Gaussians, centered -22.5 22.) and 2.55, with a variance of 0.25. Pref- erence data was simulated using the BT model, and we attempted to learn the reward function with a 3-layer MLP with 16 hidden units. For each round, 1000 points were sampled from a standard normal distribution, and 200 com- parisons were selected using different strategies. 4 rounds are shown in Figure 1, with zoomed-in version in Figure 6 and performance measured by 1-Spearman correlation in Figure 7. D-optimal method outperform alternatives includ- ing full Bayesian method like BatchBALD.
S f( Gi), where each data point contributes C;, we introduce weights wy, allowing the score to be rewritten S(w) f f(E; waa)). For instance, the D-optimal score expresses the determinant of FI of subset C as a weighted sum.
Stopi (w) = 1 ".w((... 2..2) (P.. 21.2) T (1)1 (8)
Each candidate pair is assigned weight w; liee- Select- ing subset C to maximize Samp is equivalent to finding sparse 0-1 weight vector w that maximizes (w))
Ground Truth Dio
Fandom
Entracy
Coresot
detixtX
Maxdiff
BatchBALD
To approximate the optimization, we treat w as continuous and perform Taylor expansion around w = 1, the all vector, i.c., all data points are included.
tounco
Buans
S(w) 8(1) (1 -1-w0 (
TV VVn((w))mm
tound neun sc
The approximated optimization problem becomes
argmax,, S(w) x argmax w V VwS(w))w= (10)
Figure 1: Comparisons drawn by different strategies learn a 2D bimodal reward function. The heat map showed the estimated functions. Red dots connected by lines are selected pairs and gray dots on the first column candidate points to choose from.
A sparse 0-1 valued vector w that optimizes the right-hand side of Equation (9) can be obtained by selecting the data points with the largest gradient, S(w) w= A proba- bilistic approach, when all gradients are positive, involves sampling according to the weights given by V Sww)( m=1"
What were compared in dimension two? We observed that D-optimal selects diverse samples with many anchor- ing points, often comparing multiple points to single one,
5

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
(ARM iter
Iter 1-
LLM generate
Dataset
Her 1-1 Reward Model
40k prompts
sample
500 prompts
10 responses per prompt (5000 prompt-response pairs)
22500 potential
ARM methods
In- In-Prompt comparisons
(ARM iter !)
sample
12.5 pairs to annotate
+
20000 pairs
255M potential
annotator
Cross- Pross- comparisons
Iter Preference Daraset
Preference Datasct
Reward Model
ARM Active Reward Modeling.
train
desunes conzalenation of dataoots.
illustative ecamples of potental hypee--prmmeters. Senctivvty analysis experineon.
iter Reward
Model
(ARM iter
Figure 2:
Workflow active reward modeling and experimental setups. each round, we start with randomly sampling prompts,
generating responses and candidate comparisons, active labeling, and model retraining.
spreading out the level set in the original space. Entropy sampling, similar to random sampling, focuses on points near reward values, effectively traversing the reward func- tion's 's level set. Coreset also selects diverse comparisons, though always among points with similar reward val- ues. The best design matrix method behaves similarly to coreset, emphasizing diversity in comparisons. In contrast, the max difference method tends to compare extreme val- ues with many others, promoting exploration but potentially yielding less informative comparisons. BatchBALD also selects diverse comparisons, though without clear pattern. These observations suggest that most methods encourage exploration, entropy sampling prioritizes informative com- parisons, and D-optimal seeks balance between the two.
correlations as a test set metric.
Best-of-N Reward: we evaluate the best-of-N (N=500) reward on test prompts (Gao et al., 2023; Gui et al., 2024).
A method is considered superior if achieves smaller Spearman's 's correlation, larger Best-of-N reward, the same performance with fewer annotations.
Base Models, Annotations, and Golden Reward Models. We conducted experiments using three open-source LLMs: Gemma2b, Gemma7b, and LLaMA3-8b (Team et al., 2024;
Meta, 2024). To ensure affordability, we followed methods from Gao al. (2023); Liu al. (2023): Tran & Chris Glaze (2024); Dong al. (2024); Sun et al. (2025a)
open- source golden reward models as annotators. We used the Anthropic Harmless and Helpful datasets (Bai et al., 2022a) that has been widely studied in reward modeling, and golden reward models are available (Yang et al., 2024; Dong et al., 2023; 2024). The dataset includes 40k prompts with 10 responses each for training, and 2k prompts with 500 generations each for testing.
5. Experiments with LLMs
5.1. Overall Setup
In this section, we test different design strategies in LLM ap- plications. We start with discussions of general experiment setups and the evaluation metrics we used in experiments.
Reward Modeling. To separate representation learning from reward modeling, we train our reward model using joint embeddings of prompts and responses. An MLP with three hidden layers and BT loss was used. Since the BT model is not identified up to a translation, we exclude bias in the final linear layer. Our ablation studies show that the size of hidden units does not significantly affect the results. For more details, Appendix A.5.
Objective and Evaluation Metrics. We assess the data efficiency of various comparison selection methods. The main metrics are - Spearman's 's rank correlation and best- of-N test-time reward, as reward modeling aims to order responses correctly and select the best one during test time. The golden reward models from Dong et al. (2024) serve as the surrogate for ground truth. Specifically, we consider
Online Annotation Pipeline. We train our model sequen- tially, increasing the sample size at each step. At the be- ginning of each step, we randomly draw 500 prompts. For
measure the ranking correlation within each test prompt across 500 generations (Sun et al., 2025a), We took 1 Spearman's
Batched Spearman's Correlations:
6

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Harmless Gemma28
Harmless
Gemma78
Harmless
LLaMA3-88
2.05
00
0.8
5.40
0.71
0.85
2.00
2.55
2010
stoc
10100
12530
15i0e
17500
20020
2530
5000
7550
30000
12526
15000
17530
25000
2500
5000
7530
10000
12530
15000
17520
20003
1.6
1.6
3:35
1.5-
1.-
1.4
1.4
1.00
BoN
1.-
1.3
0.55
122
12-
11.
5.00
2506
scos
7500 1o600 12500 iscos 17500 25650 Number of samples
2506 5000 7506 30000 12696 15600 17520 25000
2506
5000
20000 12500 15600 17550 25600 Number of samples
7550
Number of samples
D.opt
D-opt
Random
Entropy
det(XtX)
BatchBALD
Figure 3: Comparing annotation efficiency of different methods. (Harmless Dataset, 3 Models, 8 Methods). First row: 1- Spearman's Correlation (lower is better); second row; Best-of-N reward (higher is better). Experiments are repeated with seeds.
D-opt Harmless
PA Harmless
D-opt Helpful
D-opt Helpful
1.0
1.2
1.00
1.20
Annot. Batch Size 1000
1-Speaman Corr
Annot. Batch Size 500
0.-
0.8
0.92
0.20
Annot. Batch Size 125
0.7
0.5
15000
0.5
13000
20000
0.75
0.75
10200
Diopt Harmless
PA D opt Harmless less
D-opt Helpful
PA D-opt Helpful
1.6-
1.4-
BoN Reward
25-
1.5
7.5
7.5
7.0
7.0
1.4
1.4-
8.5
E5
Annot. Batch Size 500
13.
1.3
5.0
5.0
Annot. Batch Size 250
5.5
Annot. Batch 125
12
5.0
5000
12000
15200
20300
5000
10000
15000
20002
5000
10200
15000
20030
5050
19260
15200
20020
Number of samples
Number of samples
Number of samples
Number of samples
Figure 4: Investigating how annotation batch size choices affect learning performance of methods. Model: Gemma 2B, The first two columns present results on the Harmless datasct, and the second two columns present results on the He Helpful datasct. First row: 1- Spearman's 's Correlation (lower is better); second row: Best-of-N reward (higher is better). The results presented are from 5 runs with different seeds.
each of the 500 prompts, we randomly select 2 out of 10
dictions use the reward model from the previous itera- tion. We test different annotation batch sizes, an important hyperparameter to tune, ranging from 125, 250, 500, 1000 to understand performance across vari- ous settings. After annotation, we retrain the entire model and evaluate it after each re-training.
responses for in-prompt comparisons, yiclding 500 x (10) = 22, 500 potential comparisons. Note that since these 22, 500 contains repeated use of prompt-response pairs, therefore needs only 500 10 5, 000 generations rather than 22, 500 generations. For cross pross-prompl annotations, there approximately 25 million potential comparisons. We randomly sample a fix-sized subset 20000 out of those po- tential comparisons for different algorithms to choose from, see Figure 2.
5.2. Comparing Annotation Efficiency
Figure presents results the Harmless dataset (see Appendix A.2, Figure 8 for He Helpful results). D-opt and Past-Aware D-opt outperform other methods, demonstrat-
At each online iteration, strategies that require model pre-

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Harmless Gemma28
Harmless Gemma 7B
Harmless LLaMA3-88
1.00
15peamann Corr
0.90
0.8
577
0.73
0.6-
6.6
2502
5200
7506
10000
12500
LI500
20000
2500
5000
7500 10000
12100
15000
175000 20000
5000
1000
12500
15000
17500 20000
2.6
3.5
BoN Reward
3.5-
1.1
1.0
2.3
1.2
3.
0
2505
5oc0
1690
po00o 12505 1500o 1i500 20000
6
2505
5000
10000 12505 15000 17505 20100 Number of samples
7500
2500 5006 7505
10005 12500 1503s 1750s 20000 Number of samples
Number of samples
D.opt PA D-opt
Random
Entropy
det(XtX)
BatchBALD
Figure 5:
Comparing annotation efficiency of different methods under the Cross-Prompt annotation sctups. (Harmless Dataset, Models, 8 Methods). First row: 1- Spearman's 's Correlation (lower is better); second row: Best-of-N reward (higher is better). Experiments are repeated with seeds.
ing both superior performance and greater stability. In con- trast, alternative approaches exhibit training instability and significantly higher variance during online learning.
setups and compare different approaches.
Figure 5 shows the results under cross prosmpt annotation. D-opt and Past-Aware D-opt achieve significantly better performance in both annotation efficiency and alignment across tasks and base models.
5.3. Comparing Annotation Batch Sizes
In this section, evaluate different methods under varying annotation batch size setups, ranging from 125 to 1000. Notably, our proposed methods are computationally efficient: since the reward model operates on embeddings, re-training a 3-layer MLP with 10k annotations takes only a few minutes on a GPU server-while human annotation is significantly more time-consuming.
Comparing Figure with Figure 3, we observe efficiency gains across all methods, with the entropy-basee approach exhibiting the most substantial improvement. Appendix A.4 provides direct comparison between in-prompt and cross- prompt annotations for interested readers.
5.5. Hyper-Paraee pprr-mar Sensitivity Analysis
Figure 4 presents results for the Gemma2B model (results for the other two base models are in Appendix A.3 due to space constraints). Overall, D-opt and Past-Aware D-opt consistently outperform other methods across different an- notation batch sizes. Additionally, we observe performance improvements when using smaller batch sizes, correspond- ing to a more online setup. Given the low computational cost, this suggests practical strategy: using small anno- tation batches with frequent model retraining to enhance annotation efficiency in reward model development.
To examine the sensitivity of different methods to hyper- parameter choices and provide insights for real-world ap- plications, we varied two key factors: Candidate Number and Hidden Dimension of Reward Model MLPs across different active reward modeling designs.
Our sensitivity analysis reveals that all algorithms remain robust and largely insensitive specific pyper--arameter choices in embeddings anspugss------usss are provided in Appendix A.5.
inputs setup. Detailed results
5.4. Results with Cross-Prompt Annotations
6. Discussion
Cross prosspprompt annotation has been proposed alterna- tive to in-prompt comparison, demonstrating superior per- formance in LLM alignment tasks (Yin et al., 2024; Sun et al., 2025a, see also Appendix B). To assess the generality of our methods, we extend our experiments to cross- prosmpt
Designing Comparisons. Our experiments show that ap- plying the classic method to the last-layer features yields strong performance and stability. The D-opt method is also highly efficient, as its information criteria and optimization procedure are largely analytical, enabling real-time pair se-
8

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
lection. This is valuable when collecting user preferences in user interface without introducing significant latency. Additionally, this approach might be adapted to other model architectures, including e.g., vision-language models.
In addition, if we train embedding based reward models (e.g., BT with pretrained embeddings Sun al., 2025a;b), it is computationally light. For reward model with d- dimensional input. n comparisons. and E' training epochs, total computational cost is (nd) FLOPP. This is negligi- ble compared to LLM response generation - e.g., general- ing two 1000-token responses per prompt using a 7B LLM may require 10 FLOPs and drastically cheaper than human labeling.
An Empirical Bayes View and Stability of Last-Layer Design. The connection between the last-layer D-optimal method and BALD can be seen by considering previous layers as a transformation of the Gaussian prior for the last layer's weights. These previous layer weights act as hyper- parameters of the prior, which are fitted using maximum likelihood, akin to an empirical Bayes procedure (Deely & Lindley, 1981). By minimizing posterior entropy, we perform D-optimal design followed by Gaussian approxima- tion after the transformation. Empirical Bayes helps reduce the subjectivity and uncertainty in selecting priors, poten- tially explaining the robustness of our method compared to full Bayesian approaches like batchBALD, which involve hyper-priors on these hyperparameters.
Limitations and Future Directions. Our proposed active learning scheme relies on a well-trained embedding model. The effectiveness of selecting comparisons based on last- layer features depends on these features being informative, which might in turn requires signal-rich embeddings with low noise as input of the reward model (which is MLP in our experiment). An interesting question is whether embeddings that better capture human values (and thus improve reward modeling) differ fundamentally from those optimized for generation. A related consideration is whether reward mod- cling in LLMs should start from embedding or earlier. Our method relies on BT model as the main statistical model to learn human preference. This is not necessarily, e.g., one can consider quantile regression (Dorka, 2024) and other type of models beyond binary preference data. The main idea of using last layer feature, treating the statisti- cal model as (generalized) linear model and apply classic thcory on experimental design literature can be used as a general recipe for active reward modeling.E.. E.g., for quantile regression, one can use influence function to calculate the asymptotic variance (Koenker, 2005) the metric of infor- mativeness of pair of comparison. This calculation leads to method similar to optimizing design matrix (Mukherjee et al., 2024).
Classic Experimental Design Methods in the Founda- tion Model Era. We conjecture that the success of using the last layer in classical experimental design stems from the fact that the embeddings are already close linear fea- tures. Given the extensive study of experimental design in generalized linear models (see e.g., Atkinson et al., 2007: Pukelsheim, 2006), we believe it is general strategy to ap- ply these methods to the last layer of deep learning models, particularly when leveraging learned representations from foundation models.
Comparison of Reward Model Training Cost, Annota- tion Cost, and Response Generation Cost In this section, we highlight the relationship among the cost of the reward model training, annotation, and LLM (multiple) response generation. In most cases
Impact Statement
costamnotttin coStgeneration
Our work advances the efficiency of aligning LLMs with hu- man values by optimizing preference annotation strategies. Since human feedback is costly and time-consuming, our approach can potentially reduce wasted effort on uninforma- tive comparisons, maximizing the value of each annotation. By improving the efficiency of learning from human pref- erences, this research has the potential to accelerate the development of safer and more helpful AI systems.
Specifically, while our method requires generating more responses per prompt, the cost is modest. For 500 prompts, generating 10 responses per prompt instead of 2 increases the generation cost by approximately $2.5 (from $0.7 to $3.2 in total, using DeepSeek API at off-peak rates). How- ever, the number of possible pairwise comparisons grows quadratically with the number of responses - for example, from 1 to (10) = 45 comparisons per prompt - leading to up to 45x x more training signal per prompt.
Software and Data
This enables more efficient use of the annotation budget. Assuming& a fast annotation speed of 125 comparisons per hour, annotating 20,000 randomly selected comparisons would take 160 hours, costing over $1,000 at the U.S. federal minimum wage ($7.22 25/horr). Our method achieves similar reward model performance using only approximately 1,000 informative comparisons (see Fig. 3), costing less than $50
Code and embeddings to (efficiently) reproduce all re- sults, including all baselines, can be found through https: / //github. com com/Yuny for reward modeling and efficient Best-of-N testing are available at /tttps://githb. htmm/hoarissse. embedding-bbaed--1 1m- al anieett...
FI. Embeddings
a 20x reduction in annotation cost.
9

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Acknowledgements
Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked fine- tuning for generative foundation model alignment. arXiv preprint arXiv23304.0676. 2023.
YS acknowledge the MIT SuperCloud and Lincoln Labora- tory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper. YS and HS acknowledge Yaqi Duan (NYU) for discussions in the early stage of this work.
Dong, H., Xiong, W., Pang. B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang. T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXi:22405.0786. 2024.
References
Dorka, N. Quantile regression for distributional reward models rlhf. arXiv preprint arXiv:2409. 10/64, 2024.
Atkinson, A., Donev, A., and Tobias, R. Optimum exper- imental designs, with SAS, volume 34. OUP Oxford, 2007.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05880 05862, 2022a.
Feng, Y., Kwiatkowski, A., Zheng, K., Kempe, J., and Duan, Y. Pilaf: Optimal human preference sampling for reward modeling, arXiv preprint arXiv:2502.042700 2025.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b..
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10885.10866.... PMLR, 2023.
2022b.
Gui, L.., Gârbacea, C., and Veitch, V. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.00832 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M.
Campbell, T. and Broderick, T. Bayesian coreset construc- tion via greedy iterative geodesic ascent. In International Conference on Machine Learning, pp. 698-706. PMLR, 2018.
Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 11 2011.
Huggins, J., Campbell, T., and Broderick, T. Coresets for scalable bayesian logistic regression. Advances in neural information processing systems, 29, 2016
Campbell, T. and Broderick, T. Automated scalable bayesian inference via hilbert coresets. Journal of Ma- chine Learning Research, 20(15):1-38, 2019.
Kirsch, A.,
Van Amersfoort, J., and Gal, Y. Batchbald: Ef- ficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information process- ing systems, 32, 2019.
Chaloner, K. and Verdinelli, I. Bayesian experimental de- sign: A review. Statistical science, pp. 273-304, 1995.
Koenker, R. Quantile regression, volume 38. Cambridge university press, 2005.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y.. Smith, N. A., and Hajishirzi, H. Rewardbench: Evalu- ating reward models for language modeling. 2024.
Costa, S. I., Santos, S. A., and Strapasson, E. Fisher information distance: A geometrical reading. Discrete Applied Mathematics, 197:59-69, 2015.
Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward Bag of tricks for reward modeling in llms. arXiv preprint 2024.
Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743: 2023.
arXiv:2410.188
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Salch, M., Liu, P. J., and Liu, J. Statistical rejection sam- pling improves preference optimization. arXiv preprint arXiv:2309.066. 066677677779.00 2023.
Deely, J. and Lindley, D. Bayes empirical bayes. Journal of the American Statistical Association, 76(376):833-88) 841, 1981.
10

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Lou, X., Yan, D., Shen, W., Yan, Y., Xie, J., and Zhang, J. Uncertainty-awer reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847, 2024.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Meta, A. Introducing meta Ilama 3: The most capable openly available Ilm to date. Mela AI, 2024.
Tran, D., Dusenberry, M., Van Der Wilk, M., and Hafner, D. Bayesian layers: A module for neural network uncertainty. Advances in neural information processing systems, 32, 2019.
Mukherjee, S., Lalitha, A., Kalantari, K., Desh- mukh, A., Liu, G., Ma, Y. and Kveton, B. Optimal design for human preference clicitation, 2024. Sasiga--o-huumm----rrrreeeeeee-d
Tran, H. and Chris Glaze, B. Shorkel-miistralppirmm-p-oo 2024.
Muldrew, W., Hayes, P. Zhang, M., and Barber, D. Active preference learning for large language models. arXiv preprint xxXv:2242..08114 2024.
Van der Vaart, A. Asymptotic statistics, volume Cam- bridge university press, 2000.
Wang. B., Zheng, R., Chen, L.,, Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06006 2024a.
Munteanu, A., Schwiegelshohn, C., Sohler, C., and Woodruff, D. On coresets for logistic regression. Ad- vances in Neural Information Processing Systems, 31, 2018.
Wang. H., Lin, Y., Xiong, W., Yang. R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objectio rewards. arXiv preprint arXh::2402.1857. 2024b.
Ouyang, L.., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2774: 2022.
Winata, G. I., Anugraha, D., Susanto, L.., Kuwanto, G., and Wijaya, D. T. Metametrics: Calibrating metries for generation tasks using human preferences. arXiv preprint arXiv:2410.03381, 2024.
Pukelsheim, F. Optimal design of experiments. STAM, 2006.
Sener, O. and Savarese, S. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 00489, 2017.
Xiong, W., Dong, H., Ye, C., Zhong, H., Jiang, N., and Zhang. T. Gibbs sampling from human feedback: A prov- able kl-constrained framework for rlhf. arXiv preprint arXiv:2312.114455 2023.
Settles, B. Active learning literature survey. 2009.
Shao, J. Mathematical statistics. Springer Science & Busi- ness Media, 2008
Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., and Chen, Rewards in context:: Multi-objective alignment of foundation models with dynamic preference adjust- ment. arXiv preprint arXiv:2402.1 10207, 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Yin, Y., Wang, Z., Gu, Y., Huang, H., Chen, W., and Zhou, M. Relative preference optimization: Enhancing I1m alignment through contrasting responses across identical and diverse prompts. arXiv preprint 2rXiv:2402.109555 2024.
Sun, H., Shen, Y., and Ton, J.-F. Rethinking reward model- ing in preference based large language model alignment. In The Thirteenth International Conference on Learning Representations, 2025a.
Sun, H., Shen, Y. Ton, J.-F. and van der Schaar, M. Reusing embeddings: Reprodueible reward model research in large language model alignment without gpus. arXiv preprint arXiv:2502.04335 2025b.
Zhang, X., Ton, J.-F,, Shen, W., Wang, H., and Liu, Y. Overcoming reward overoptimization via adversarial pol- icy optimization with lightweight uncertainty estimation. arXiv preprint arXiv:2403.05177 2024.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.00 08295, 2024.

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
A. Additional Experiment Results
A.1. Two dimensional example
In this section we provide an enlarged version of figure Figure and the performance measured by Spearman's 's rank correlation on an 2D grid.
Ground Truth D-opt
Random
Entropy
Coreset
det(XtX)
Maxdiff
BatchBALD
Round O
Round 5
Round 10
Round 15
Figure 6: Comparisons drawn by different strategies to a 2D bimodal reward function. heat map showed current estimated function. Red crosses connected by black lines are compared pairs and gray dots on the left-most penal are candidate points to be compared.
In Figure we showed the performance measured by 1-Spearman correlation on the entire grid. Last layer D-optimal method outperform other baselines including full Bayesian approach like BatchBALD.
0.7
1-Sparmma conaationn
0.5
0.4
0.2
200
400
600
800
1000
1200
1400
1600
Number of samples
D-opt
Entropy Coreset
det(XtX) Maxdiff
BatchBALD
Random
Figure 7: Performance measured by 1-Spearman correlation (smaller the better) the 2D illustrative example.
12

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
A.2. Comparing Annotation Efficiency the Helpful Dataset
In- In-Prompt Annotation
efficiency is provided in Figure (as supplementary of Figure 3 in the main text).
Helpful Gemma2B
Helpful Gemma78
Helpful LLaMA3-88
1-Spearman Corr
0.50
8.85
0.55
3.80
3.80
0.75
15000 17500
2503
5003
7500
10000
12603
15003
17503
2010s
19630
8.0
5.5
BoN Reward
5.0
7.0
80-
6.5
5.5
65-
55
10000 Number of samples
2000a
2503 5263 7500
10000 12502
18630 1250e 17500 20000 Number of samples
Number of samples
D-opt
D-opt
Random
Entropy
det(XtX)
Figure 8: Comparing annotation efficiency of different methods. (Helpful Dataset, Models, 8 Methods), First row:
Spearman's 'ss Correlation (lower is better); second row: Best-of-N reward. Experiments repeated with 5 seeds.
Cross-Prompt Annotation
efficiency is provided in Figure 9 (as supplementary Figure the main text).
Helpful Gemma28
Helptul Gemma78
Helpful LLaMA3-88
6,550
0.65
3900
1-Spearan Corr
0.000
0.50
0.88
0.85
0,600
0.64
2.800
0.79
0.775
0725
20003
9.0
BoN Reward
7.5
8.5
8.0
7.0
75-
10003 12300 13300 17500 20003
20000
20003
2500
Number of samples
Number samples
Number of samples
Diopt
PA Diopt
Random
Entropy
Coreset
det(x
Maxdiff
BatchBALD
Figure 9: Comparing annotation efficiency of different methods under the Cross-Prompp annotation setups. (Helpful Dataset, Models, Methods). First row: Spearman's 's Correlation (lower better); second row: Best-of-N reward. Experiments are repeated with 5 seeds.
13

---

Active Reward Modeling:
Adaptive Preference Labeling for Large Language Model Alignment
A.3. Annotation Batch Size
Results on All Models Due the space limit of the main text, we deferred the experiment results with Gemma7B and the LLaMA3-8B model when studying the effect of different annotation batch sizes in the following Figures (Figure 10, Figure 11, Figure 12). To summarize the main takeaways
observe the same trend as we have observed with the Gemma2B model, the proposed methods achieve better performances in the small batch size setups (more online setups). The stability of small batch setups is in general higher than the large batch setups.
D-opt Harmless
PA D-opt Harmless
D-opt Holpful
PA D-opt Helpful
2.0
Areet: Barth Sam 1800
8.5
8
5.55
Artul Butch siz
siz
15pearman
3.5
84
8:00
0.00
0.5
1200
Harmleess
8.
ured
opt
thore
20000
077
D-opt Helpful
1999
calan
2000
6:75
1000
opt Helpful
18990
20000
Roward
70
Arnet: Sanch 3as 1000
Arnot Batch 2as 500
Kremt Barth See 250
55-
Arnot: Butch Saz 122
1900
Number samples
Number samples
Number samples
Figure 10: Investigating how annotation batch size choices affect learning performance of different methods. Model: Gemma 2B.
D-opt Harmless
D-opt Hammless
D-apt Helpful
PA
035
Armot Blach duc
Awnet Barch for 1000
iwnot Datch Sae 250
035
0.93
0.80
Anrol Mutch Sice 225
075
0.5
0,80
3.80
D ept Harmless
PA D-opt Hammless
D opt Helpful
PA. b-opt Helpfal
BoN Reward
65-
14-
/enot Barch Soe 3000
79
10-
Armitt Match diz
Armol Burch duc 200
Annct Batch Sbe :25
2008
3009
3500
X090
5099
10000
1580
20000
5300
23006
19000
3000
530
15506
2000
samples
Number samples
Number samples
Numbeee samples
Figure 11: Investigating how annotation batch size choices affect learning performance of different methods. Model: Gemma 7B.
Diopt
ogt
D opt Helpful
Helpful
Soe
0.93
0.90
Annot Batch Sox
Awnce Barch Sbe 125
020
2008
D-opt Harmless
13300
2003 Diapt Harmless
18000
D-opt Helpful
PA D-opt Helpful
2.5
S.5-
1.5
BuN Reward
1.30
138
1.
(wnst Barch Soe
Arrol Butch dizt 215
2008
10000
2500
20090
5003
18000
1900
20000
5296
2000
U0000
2000
5269
1500
2000
Number of samples
Number of samples
Number of samples
Numbeer stapples
Figure 12: Investigating how annotation batch size choices affect learning performance different methods. Model: LLaMA3-8B.
14

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Results with All Methods. In addition, we use the figures below (Figure 13, Figure 14, Figure 15) for a full analysis on the annotation batch size choices for all methods. For other methods, we do not observe clear trend on the effect of increasing or decreasing annotation batch sizes.
Diept tlemiees
C-epe Harmene
Random limmiiee
Frrmey Lamiles
Corport Hormleer
Hammbe
Houdtt Hormiese
BancnBMil Harmieo
1.paymm On
Arfic 2004
Arrict Ba 1a0
beeet limm ang
Bn C-epe darm ane
Ka Bendom firmmieee
Liee Enmmy Famiissr
Corpee Hammlede
2Ct Voust lirmieos
Lis Barceriii Harrmiaee
Jein Ra 2005 Jerac
PG D-
Rontom (oke71)
Crimpy Helphul
Coneaet Eiephy
Xox Helpll
Maud degph
2019 1956
50% 10:0 ub5n 12h0
ING 100 u3dc 2010
000 1290 10:00
260 1000
100 1860 180 2009
2059 1800 3500
Figure 13: Investigating how annotation batch size choices affect learning performance of different methods. Model: Gemma 2B.
Entrray
Dieer llmmees
C-epe 1amm
Ratdom limmiiee
Frmmayy Taamiine
Corper Hommaeer
Houd Hlammete
D-spt: "Ropful
2019 150g
300 10:2 1900 1000
Rundom
Ertropy
199 1008
300 380 1000
200
1900
Pexal Negeu
5023 18
BuEshBALL
Aerict Ba
Anrict Ba
D-spt
Nimberr of amm
Number of complee
100m Nimber CVTD mM
1299 Namber Sxroples
Mumber 07 Etrrppeee
N: Numberr oframpplee
3009
5000 2000 nunder owrpper
Numberia of oump ved
Figure 14: Investigating how annotation batch size choices affect learning performance of different methods. Model: Gemma
15

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Dicpe
Narmiess
Sandom Harrieus
Drinaay Heamlen
Car Thrmiees
TEX rox-taamiess
Sater BALD Farmlas
Jerect de +2
11:0
Diapt Hetril
Hardion Fee 201
Exerody Helpoy
Cormrct Fre g01
Novil Frichiile
1poorn Con
D-spt Horrd
Hondon
EIRr0dy Hedplu
1906 Hoxd Miidallll
MATCYSALD
19h Ji0c
pi0 Number compliss
100p Samples
humber camppees
samples
Number (ampptes
Mumber 5a5mmiles
Figure 15: Investigating how annotation batch size choices affect Icarning performance of diffcrent methods. Model: LLaMA3 8B.
A.4. Compare Cross-Prompt Comparisons and In- In-Prompt Comparisons
In this section, we provide direct comparisons of learning efficiency when using cross-prompe annotations and in-prompt annotations. In most cases, annotating comparisons using cross-promp comparison improves learning efficiency, and this can be observed across all methods. Specifically, with the entropy-based method, cross-prompt annotations bring noticeable boost to learning efficiency and reward model performance.
D-ept llarniss
Sandom Harriisse
Entrepy llarmiece
Coree lammeeee
larmiier
Maudft (iamlee
1.ppyrr Cot
2e 1996
200 siee niso 1h
U-u Nambiso
Gandem Numbioo
only c2l0
1016 Hilt 1009 Hamiice
2018 c9do paby 1100 Familoo
540 1200 R010 2010
Muadd 1990
2010 caloo
Fee Rewaord
Bostindald
1% piocl Rtye
Burdan tep
Brinss di ytu
100 Pniolsl
200
1850
1586 Samplis
3000
506
650
2000
1800
1999
2008
2009 1050 1996
585 1000
1:30
2000 Solie Lako
Figure 16: Cross-Prompt preference annotation improves overall annotation efficiency. Annotation batch size 500. Model: Gemma2B.
16

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
5:
Ro-oon Mand Harmica
3-spthi haaaooos
1% opt Narmics
Sandom Nermios
E-tropy Narmics
Corsast Farmisss
321 Harmisss
1805 pa0
1080
30c 1320
2010
200 201
Random
1200
200 109h 2013
2010
203 1300 1301 1100
Colegr lieppii
300 1200 B0B
2015
1120
Nixdi 1800
2013
Chass Paa
Promee
200 1200
samples
humber
D00
Namber aaammllcc
1000
Number
1990
2013
203 1250
1906
1000
300 1130
2000 2003 1a00
Figure 17: Cross- Pross-Prompt preference annotation improves overall annotation efficiency. Annotation batch size 500. Model: Gemma7B.
Dost
Diast
Dosk Harmhoo
PA Drspk Marm Harmios
Purrdom Parrics
Ertrppy Harrikos
Consaed Piarriis
XOL Marrnhoo
11%
1100
1000
1999
caice Lake
milm Sandom Fietrtul
comm limptull
calo 120m
302
10x
Mamber caramplee
Number of aarple
Kumber of complee
Namber of sumolee
Wlmoerr of exmp Me
Number Steanglee
Number of camppie
Figure 18: Cross-Prompt preference annotation improves overall annotation efficiency. Annotation batch size 500. Model: LLaMA3-8B.
17

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
A.5. Hyper-Parameter Sensitivity Analysis
Number of Candidate Numbers In the main text, our empirical pipeline starts by sampling 500 candidates (candidate number) from the training prompts, and then randomly generates 20000 pairs of comparisons using either in in-prompt comparison cross tross-prompi comparison. Then, select annotation batch size number of comparisons to annotate. In this section, we evaluate the performance difference by using a larger candidate number 1000.
In experiments, we find those setups do not significantly change the performance of different methods. The performance of D-opt and Past-Aware D-opt are especially robust to those pappme-e choices.
Hupptooriies
Aandom
Familes
Hormiees
Maadd
1.ppamm Cit
pales Coroaut Farmloas
D-sptharrsesss
opt
Harmitss
E-trapy
Harmiess
19, Diiee seg
201 Randim degii
1.5
Entrncy
190h 2019
1250 Coropet (elplll
1004 Irttens
1000
644
Nurboer of samplea
humber samppp 53
Namber atsurplis
Numbbr semolia
Number chsumppic
Number of samp ka
Nimber of samp
Figure 19: Preference annotation with different candidate number choices. Annotation batch size 500. Model: Gemma2B.
D-spehorntess
PA Dregt Harmlegs
Sandion Harmeess
Ertrpyy Harmless
Corezer Fammees
XC1 Hommiees
Mowidff Hammless
040
330
C20
420
LNC
6:70
810
420
4.00
841
2060
P5 Doteettto de pou
Randim degite
Entmay de giu
Consert Helpll)
Houd Hegple
D BostFidof
1999
2000
30hh 1399 665
200 2005 Bandam
1200
120 1996 2019
2012
209 1000
2010
1120 vola
car Miri to moper LA-
Numoor samplia
number aarp 13
Namber stmmcci
Number samola
Numrorr ohturpiis
humbar aarp
Namber of sumpip
Figure 20: Preference annotation with different candidate number choices. Annotation batch size 500. Model: Gemma7B.
18

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
D-azt Harmleas
D-opt Harmilss
Ransam Familon
Entropy
Caromec Ftamiis
XBX Harmiaus
Maxud Haam
card Seme, sumtte 1a:
vallo
Dropt
1.7%
card dere
Dost Herghul
Dooit (f.1
Kardom Hebiful
Crtmpyy Heppll
Coresit Helpi
He piuy
Maacdd Helgiu
eilm Random Timbrral
udo Samppes
1800 5amppes
2000 Sismplles
Camplis
Numbo Comples
camppes
Figure 21: Preference annotation with different candidate number choices. Annotation batch size 500. Model: LLaMA3-SB.
Number of Hidden Units in 3-Layer MLPs In all main text experiments, we use 3-layer ML.ss with 64 hidden units. In this section, we evaluate the performance difference by using larger hi hidden unit 128.
D-eet temmees
De D-epe Htrrmare
Bandom lormleee
Enmmay i Hamiior
Corvoet Hommleos
XBX lismboe
Voudt Hirmiloo
BaneeBNl Harmieee
SSpeeron Can
velo 2009
2009
PN D-sppt (011
Pontom settiil
Frimpy Heephu
Coreact Lepph
Xox Helpoi
Heggh
Tan Vilw
Vile Dunsom Pndatal
Holpru
Canoust
Mexdl"
Siaee BubchbeLL Inllanu
1200 Number bumplic
1290 Marroe aumpko
1a8a 2000
2153 humber aurppis
Numbor of semp
NATID2 pump K3
Mumber ataumpic
Numberr sumplic
Humber of sump
Figure 22: Experiments with different hidden unit choices. Annotation batch size 500. Model: Gemma 2B.
19

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Dispt farmi
1N.C C-spc. form 533
Bandom Naarricss
Enirogy Fammios
Coroae: Nemmicss
X00 Fammic
Pluud Herrioo
Bisch BOLD Herriiss
2012 1800 D-oph:1 "Ra
200 10: 1205 1000
200 120p
F-dbot)
100 1008
200 200 1306
2000
Z0t
(205 p8la
2000
Maxall Reger
2003 0R0 1000
BUESnDALD 1800
2000 201
Erimpy
Number
2003 19th 1508
W00 120 1290
200 K00
Namoe
1290 102
K00 20: 1991
200 1890 1209 3000
Number
2059 2000
Numbor
1100 2010
Figure 23: Experiments with different hidden unit choices. Annotation batch size 500. Model: Gemma 7B.
Ciept
Dasl Hemrisso
Sanshon Nernins
brlogy Hambiee
Nemboss
BAL-- Famblod
Dropt Marmiss
Kanddum Hurmioo
bioasy Harmoos
Car CAO. Hummioo
Mandel Pormiess
Sale BOLD Parmili
ain wicht
200 Jales 100m
cade nice Haet (11
Enerady
Nimber cample
Number carappee
Number of canplles
Number commpee
Mumber esmpllee
Number CX At
Marther of ovricle
Aumobr of camp
Figure 24: Experiments with different hidden unit choices. Annotation batch size 500. Model: LLaMA3-8B.
20

---

Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
B. Further Notes on Cross-Prompt Annotations.
Cross-Prompt annotations were explored as a way to increase annotation quality by Sun al. (2025a) and empirically studied Yin et al. (2024). natural question whether this is possible in practice. If one is willing to assume there exists scalar value reward functtin, and human comparisons are based on that function, then cross-prompt is possible because each prompt-response pairs are assigned real value that comparable to each other. A single-word change in the prompt without changing its meaning likely will not change what responses are helpful or harmful and make these pairs, even if cross-prompt, comparable. It is possible, however, that the reward function is very rough in changing prompts, making the reward function for one prompt not transferable to the other and making it hard to get better response for one prompt using a reward function learned from other prompts. Even though, if one is willing to believe that prompts live in some lower-dimensional manifold and the reward function acquires some regularity that space, Cross- Cross-Prompt annotations might help better learn these dependencies.
21


Reviews:
[{"review_id": "hVeaSmLgDB", "reviewer": "Reviewer_6Pdx", "summary": "This paper investigates strategies to leverage adaptive preference labeling for reward modeling in LLM alignment. The authors propose an Active Reward Modeling ARM framework that uses Fisher information to score and select informative preference comparisons to improve annotation efficiency; then, they benchmark several active learning and experimental design-based strategies across multiple models and datasets. They report gains in annotation efficiency and reward model performance compared to random sampling and other baseline methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank our reviewer for their careful and detailed review of our work. We appreciate that several valuable points of feedback were included, and we believe that the updated version of our work will be strengthened by reflecting on these points. We address concerns and questions below.**\n\n---\n### **Q1. Can using the last layer overlook complexity and lost robustness**\nWe thank our reviewer for raising the insightful question. An extension using uncertainty of more parameters via Fisher Information (FI) is possible. E.g. [2] used Bayesian uncertainty for all parameters. In our experiments we find our method to be better than [2]. An intermediate approach using uncertainty of more layers than the last could potentially further improve the performance yet is left for future research.\n\n### **Q2. Can actively selected samples lead to better generalization**\nWe agree with our reviewer that evaluating the generalization ability of RMs is necessary. In our work, we evaluated our RMs with holdout data, showing no signs of overfitting (in contrast, the entropy sampling method shows overoptimization). Relating to the previous point, using only the last layer features may be a regularization, preventing over-optimization. This could explain why we outperform [2]. \n\n### **Q3. Downstream performance**\n- We agree that a good reward model may not lead to improved alignment. Human preferences are complex, and no scalar reward function can fully capture it. While this is a valid concern, addressing it is beyond the scope of this paper. We follow the common RLHF assumption that a reward function serves as a sufficiently useful optimization objective. Ideally, we would test the entire alignment pipeline but such a test is beyond our computational resources. We believe our experimental setup provides sufficient evidence of the method's utility. \n- To be more concrete, our evaluation setup has\n   - **Operational definition of alignment** We define alignment as having high human reward value. Since the true human reward is not directly accessible, we use a \"golden\" reward model—trained on a large dataset—as a surrogate. \n   - **Metric of success** An active learning method is successful if best-of-N samples based on this reward model achieve higher rewards with fewer human annotations. We also reported the Spearman correlation of RMs and the golden reward models.\n\n\n### **Q4. FI Suitability**\n- Intuitively, this approach balances exploration in representation space, enabling the reward model to predict across a broader range of prompt-response pairs while exploiting uncertain comparisons between hard-to-distinguish pairs. This is preferable to purely diversity-based methods, which may focus on “easy” comparisons, and more effective than purely uncertainty-based methods, which often compare similar pairs, limiting generalization (c.f. Fig.1). \n- Empirically, our approach outperforms pure diversity methods, e.g. coreset, and pure uncertainty method e.g. entropy sampling [1]. Our method achieves both better sample efficiency and asymptotic performance.\n\n### **Q5. Does FI Align Better with Real Human Preferences?**\n- This is a valid concern, and we break it down into two parts as FI depends on model:\n   1) Is Bradley-Terry (BT) a good model for human preference?\n   2) If we accept BT as a good model can FI-based active learning handle noise and improve sample efficiency?\n- For 1), this is a valid concern. As George Box said, \"all models are wrong, but some are useful.\" BT is: a) widely adopted, and b) proven to be effective in large-scale applications. Thus, we consider it a useful model to improve sample efficiency. Testing alternative models to account for inconsistencies e.g. transitivity violations is beyond the scope of this paper.\n- For 2), assuming BT is useful, our experiments demonstrate that FI-based sampling design can yield a better-performing reward model with less data. The intuition behind is discussed in the response to why FI is a good metric.\n\nTo summarize: since BT is widely adopted and acquiring human preference data is expensive, our goal is to improve sample efficiency by carefully selecting comparisons.\n\n---\n**We thank the reviewer again for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them. Please kindly consider increasing the rating if their concerns are properly addressed.**\n\n---\n_**References**_\n\n[1] Muldrew et al. \"Active preference learning for large language models.\" ICML’24\n\n[2] Kirsch et al. Batchbald: Efficient and diverse batch acquisition for deep Bayesian active learning. NeurIPS’19", "claims_and_evidence": "The authors' claims regarding the efficiency of D-optimal and past-aware D-optimal methods are supported by experimental results, but the paper lacks theoretical analysis to explain why these methods outperform others in reward modeling and how they impact the process.", "methods_and_evaluation_criteria": "Generally appropriate.", "theoretical_claims": "For assumptions, I notice two potential issues:\n\ni. The paper assumes that Fisher information computed over the last layer of the reward model sufficiently captures the informativeness of preference comparisons. But this may be an overly idealized assumption. Actually, deep reward models exhibit complex, non-linear behaviors. If we focus only on the last layer, we may overlook critical interactions and uncertainties in earlier layers, and this could limit the robustness of the active selection strategy.\n\nii. The paper assumes that reward model improvement is directly aligned with selecting samples that maximize Fisher information. However, this does not guarantee that the selected comparisons will improve generalization to downstream tasks. In particular, maximizing Fisher information might bias the reward model toward \"hard\" comparisons that are not representative of typical user preferences. I worry that it may lead to a misaligned reward signal.", "experimental_designs_or_analyses": "The experimental design relies on Fisher information-based selection during the preference data sampling process, with the assumption that it consistently leads to better reward model performance. The paper does not address the potential biases introduced by focusing on \"informative\" but possibly unrepresentative comparisons. It would be valuable to discuss the trade-off between focusing on hard-to-predict samples and maintaining representativeness of human preferences, and the actual improvements in real-world alignment this strategy offers. Specifically, how does selecting for Fisher information impact the generalizability and alignment quality of the reward model?\nThe comparison of ARM's preference data selection and resulting reward model performance to baselines is not sufficiently explored. The experimental analyses focus on annotation efficiency and ranking accuracy but do not assess downstream alignment impacts or broader generalization. These are limited and do not offer enough evidence to assess ARM's real-world alignment capabilities. Given that reward models are often applied to diverse and complex downstream alignment tasks, and that preference data may not always map directly to better alignment, it would be beneficial to provide a broader set of downstream evaluations or human-in-the-loop assessments. This would offer a more comprehensive view of ARM's effectiveness in practical LLM alignment scenarios.", "relation_to_broader_scientific_literature": "This framework builds on prior work in active learning, experimental design, and preference modeling, specifically leveraging ideas from Fisher information-based experimental design and Bradley-Terry preference models to improve reward model data efficiency. And this paper compares its Active Reward Modeling framework against several baselines, including random sampling, max reward difference, max entropy, batchBALD, and coreset methods for preference data selection.", "essential_references_not_discussed": "The main limitation of this paper is the lack of discussion on the motivation and theoretical justification of the proposed method, rather than missing references.", "other_strengths_and_weaknesses": "The idea of actively selecting informative preference comparisons for reward modeling is novel. Most existing work focuses on random sampling or heuristic-based selection for preference data, but these approaches cannot efficiently optimize the annotation process to maximize learning efficiency, this could affect the scalability and effectiveness of reward models in LLM alignment. This paper introduces a method that leverages Fisher information-based selection strategies to identify the most informative preference comparisons, as well as significantly improves annotation efficiency and reward model performance with fewer labeled comparisons.", "other_comments_or_suggestions": "The authors do not clearly explain why Fisher information is the most suitable criterion for preference data selection in the context of reward modeling, compared to other uncertainty or diversity-based approaches.\n\nI wonder why selecting \"informative\" comparisons based on Fisher information would align better with real human preferences, especially when human preference data may be noisy or inconsistent. Could the authors clarify the motivation behind this assumption?", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Wfsrcs3Mx", "reviewer": "Reviewer_kGRg", "summary": "The paper presents an evaluation of different methods to determine which samples in a dataset of (prompt, response one, response two) triplets should receive a preference label and be used to train a LLM reward model using the Bradley-Terry model. The authors propose to use a modification of D-optimality on the embedding space of the reward model to determine which samples to label and train with. Six other strategies to select samples are compared against with a detailed description of each method to select the training samples. The difference between the methods is examined with a simple, 2-dimensional toy dataset and a the Helpful and Harmless data from Anthropic. For the Helpful and Harmless 3 different LLMs are compared to understand how the LLM impacts the data selected with each strategy. The experiments and results rely on 1 - Spearman correlation and Best of N reward. Performance is evaluated both within and across prompts, and an experiment is run to look at the how the number of samples impacts reward model performance. The learned reward models are compared to Gold Standard reward models for the HH Anthropic dataset. The experiments demonstrate the benefits of D-Optimality over the other methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank our reviewer for their encouraging feedback. To respond to the point raised by this reviewer, below, please find our answers to each of the questions.**\n\n---\n\n### **Q1. Translating performance of reward model not to alignment**\n\n- In our experiments, we evaluated the effectiveness of different reward models using Best-of-N (BoN) sampling following [1,2] and the Spearman Ranking correlation following [3]. This choice was driven by three key considerations:\n    1. **Performance**: Empirical studies show BoN achieves better performance than PPO [1,2,4,5]\n    2. **Stability and Reduced Engineering Overhead**: BoN requires no hyperparameter\ntuning and is more stable than PPO, leading to more consistent and interpretable results. [1,6,7]\n    3. **Computational Efficiency and Reproducibility**: BoN’s reusability across N generations during test time makes it more computationally efficient compared to policy gradient optimizations. In contrast, using PPO or DPO for our experimental setups (in total 3840 = 5 random seeds x 3 LLMs x 8 methods x 2 annotation strategies x 4 batch sizes x 2 network sizes x 2 candidate sizes) would be computationally prohibitive since each setup requires distinct LLM fine-tuning [8]. \n\n- It is indeed true that a good reward model is not sufficient for good performance. However, it is probably necessary for a good final policy performance and the best-of-n sampling we tested can be seen as a best-case scenario [1] that can be treated as an approximated upper bound of final performance after RL that does not involve costly DPO or PPO. \n\n\n### **Q2. Adding toy example performance figure**\nWe thank our reviewer for the great idea! We do have a similar performance figure and we will add it in the revision of the paper. \n\n\n### **Q3. More clearly highlight that D-Optimality**\nWe thank our reviewer for their suggestion. We agree highlighting more on the D-Optimality in the introduction, and method in our revision can further enhance its clarity.\n\n### **Q4. Conclusions from Figure 1**\nWe thank our reviewer for the great suggestion. We will add a table detailing the number of connections and sample diversity, e.g., the variance of pairs in the original space and the last layer of the neural net\n\n### **Q5. What is $I^s$ in the active learning section where the datasets are defined?**\nThe number of unannotated prompt-response pairs. We have clarified the notations accordingly in our revision. \n\n### **Q6. expand upon \"joint embeddings of prompts and responses\" on line 326**\nBy joint embeddings, we were referring to the embeddings of prompt-response combinations.\n\n### **Q7. For figures 3, 4, and 5, what are the error bars over?**\nIn our experiments, all error bars are generated with **5 repeated runs with different random seeds** to show the statistical significance of performance differences.\n\n---\n**We thank the reviewer again for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them.**\n\n---\n\n_**References**_\n\n[1] Gui, Lin, Cristina Gârbacea, and Victor Veitch. \"Bonbon alignment for large language models and the sweetness of best-of-n sampling.\" arXiv preprint arXiv:2406.00832 (2024).\n\n[2] Gao, Leo, John Schulman, and Jacob Hilton. \"Scaling laws for reward model overoptimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n[3] Sun, Hao, Yunyi Shen, and Jean-Francois Ton. \"Rethinking reward modeling in preference-based large language model alignment.\" The Thirteenth International Conference on Learning Representations. 2025.\n\n[4] Dong, Hanze, et al. \"Raft: Reward ranked finetuning for generative foundation model alignment.\" arXiv preprint arXiv:2304.06767 (2023).\n\n[5] Yuan, Zheng, et al. \"Rrhf: Rank responses to align language models with human feedback without tears.\" arXiv preprint arXiv:2304.05302 (2023).\n\n[6] Ivison, Hamish, et al. \"Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.\" Advances in neural information processing systems 37 (2024): 36602-36633.\n\n[7] Xu, Shusheng, et al. \"Is dpo superior to ppo for llm alignment? a comprehensive study.\" arXiv preprint arXiv:2404.10719 (2024).\n\n[8] Stiennon, Nisan, et al. \"Learning to summarize with human feedback.\" Advances in neural information processing systems 33 (2020): 3008-3021.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. However, the paper would be stronger if at least one other dataset (e.g. OpenAssistant) was evaluated as well.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem at hand.", "theoretical_claims": "There are not proofs to assess", "experimental_designs_or_analyses": "Yes, I checked the soundness/validity of the experimental design and analysis. I checked the set up of the 2-dimensional toy experiment and of the helpfulness and harmless experiments. Although many important details are provided to understand the experimental set up, there are some gaps. Please see the questions below. The impact of the number of samples is assessed as well as the impact of annotation size for the D-optimality sample selection strategies. \n\nAs there is evidence that reward modeling abilities does not always directly translate to final policy performance, an evaluation of the quality of policy the reward model is able to train is missing. This can be done either with DPO or PPO.\n\nIt would be great to supplement the comparison of samples selected by the different sample selection strategies for the 2-dimensional toy domain experiment with an evaluation of the quality of the learned distribution to draw the connection between the samples selected and what the reward model learns.", "relation_to_broader_scientific_literature": "The paper is well situated within the broader scientific literature on preference sample selection.", "essential_references_not_discussed": "The references discussed appears sufficient", "other_strengths_and_weaknesses": "The paper is well written and easy to follow with key take aways clearly stated along with key aspects of evaluation, such as success criteria clearly state.", "other_comments_or_suggestions": "1. Please more clearly highlight that D-Optimality is the method you are \"proposing\" in this paper. It takes a bit of re-reading to understand that this part is a main contribution.\n2. It is difficult to interpret and draw conclusions from Figure 1. Including a table in the appendix with numbers detailing the attributes mentioned in the discussion would be helpful. For example, the mean and standard deviation in number of connections between each point, the difference in rewards, and a measure of sample diversity.", "questions_for_authors": "1. What is I^s in the active learning section where the datasets are defined?\n2. What is meant by \"joint embeddings of prompts and responses\" on line 326? Please expand upon this.\n3. For figures 3, 4, and 5, what are the error bars over?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Saf7beDkL9", "reviewer": "Reviewer_M1YG", "summary": "The paper proposes active learning methods for reward modeling. These active learning methods work as follows:\n\n1. For a large set of prompts, use LMs to generate responses for comparison.\n2. Form these generated prompt-response pairs into tuples for either in-prompt comparisons (prompt, response 1, response 2) or cross-prompt comparisons (prompt 1, response 1, prompt 2, response 2).\n3. Select a subset of tuples to annotate, using the current reward model.\n4. Use the selected subset of tuples to update the reward model.\n5. Return to step 1 and iterate.\n\nThe key problem is step 3: how to select the subset of comparisons from a large candidate pool. For this selection, the authors consider a few choices of scoring functions (Section 3.1), which assign a score to each subset of comparison tuples. Empirically, the scoring rule based on D-optimality works best, according to the authors' evaluation.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for investing their time in reviewing our work, and providing insightful suggestions for improving our paper. We have carefully considered each point of feedback and provided our point-by-point responses below.**\n\n---\n### **P1. Response to the main concern: cost of oversampling**\n_We thank the reviewer for raising the question on sampling cost, and we are reminded there could be misunderstandings about the generation cost in the proposed procedure._\n\n**1. Sampling Cost**\n\nWhile our algorithm requires a **large number of comparisons** to choose from, these comparisons are generated **combinatorially from a small number of responses**, this is because we can reuse a single response in multiple comparisons. \nFor instance, generating $10$ responses rather than $2$ responses as in non-active random sampling will lead to $ 45 $ times more comparisons. Moreover, in the cross-prompt comparisons setup, $10$ responses per prompt on $500$ prompts lead to $12M$ potential comparisons. \n  - [**Spend 2.5 more USD in generation**] To compare the cost of generation and annotation. With our setup of 500 prompts, using the DeepSeek API at off-peak hours, generating 2 responses would cost 0.7 USD (2M input tokens and 2M output tokens); and generating 10 responses would cost 2.5 more USD.\n  - [**Saving 1000 USD with efficient annotation**] In our experiments, we find using 10 responses per prompt can significantly improve the annotation efficiency (by over 20x more efficiently as compared to only generating 2 responses). Annotating $20000$ randomly selected preference data will cost 160 labor hours (assuming 125 annotations per hour) and will cost more than 1000 USD according to the 7.5 USD minimum wage in the US. With our method, the same reward modeling performance can be achieved with $1000$ annotations, which cost $60$ USD (cf. Fig.3).\n\n    We will be more clear in the revision on this aspect. \n\n**2. Training Cost**\n\nAs for the cost of re-training, we would note the fact that training reward models can be much more computationally efficient than fine-tuning language models [1-4]. In our work, we follow the embedding-based reward modeling setups and the computational cost is negligible as compared to the cost of LLM generation (i.e., **cost of RM training << cost of generation << cost of annotation**)\n\nIn terms of FLOPs, training a BT reward model with $2048$-dim input on $1000$ samples and optimizing for $10$ epochs correspond to $1e11$ FLOPs. Generating 2 responses (1000-token-long) on those $1000$ samples using a 7B LLM needs $1e13$ FLOPs. The expense of finishing those computations on prevailing cloud service platforms costs less than **$0.01$ USD**\nOn the other hand, hiring annotators for annotation is much more expensive. Annotating preference over $1000$ samples will induce a much higher cost. \n\n- The number of candidate prompts and responses per prompt can be tuned to accommodate different budget constraints. With more candidates, it is more likely to get more informative comparisons (c.f. Fig. 18-19 in Appendix A.5) \n- We also want to highlight that in our experiments, Fisher Information-based active training can achieve better asymptomatic performance compared to other methods for training, especially the most widely adopted random sampling approach.\n\n- The reward models can be updated more frequently than the LLMs. This is why our work focuses on active reward modeling rather than a full active alignment workflow (i.e., including both reward model training and LLM fine-tuning). As we have analyzed above, the cost of training reward models is significantly lower than the cost of response generation and collecting preference annotations. On the other hand, from the practical perspective, active (and more frequently updated) reward models can quickly adapt to user preferences and improve user experiences (c.f., Fig.3).\n\n\n### **P2. Typos**\nWe thank the reviewer for spotting them. We have fixed them in our revision.\n\n---\n**Once again, we thank our reviewer for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them. Please kindly consider increasing the rating if their concerns are well addressed.**\n\n\n---\n_**References**_\n\n[1] Gao, Leo, John Schulman, and Jacob Hilton. \"Scaling laws for reward model overoptimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Go, Dongyoung, et al. \"Compositional preference models for aligning LMs.\" The Twelfth International Conference on Learning Representations. 2024.\n\n[3] Sun, Hao, Yunyi Shen, and Jean-Francois Ton. \"Rethinking reward modeling in preference-based large language model alignment.\" The Thirteenth International Conference on Learning Representations. 2025.\n\n[4] Barreto, André, et al. \"Capturing Individual Human Preferences with Reward Features.\" arXiv preprint arXiv:2503.17338 (2025).", "claims_and_evidence": "The authors' main argument for active learning of reward models is that it reduces training costs. However, I find the evidence lacking. \n\nThe reason I think so is that the current study focuses on reducing the cost of annotating examples, but at the expense of introducing significantly higher costs in the language model (LM) response sampling phase (Figure 2). Indeed, to reduce the overall training cost, a trade-off between **the cost of generating responses** and **the cost of annotating them** must be considered. While active learning methods may reduce the number of training examples (and thereby the annotations of these examples), it introduce the extra cost of sampling, too. Specifically, the active learning algorithm (Figure 2) requires sampling **a large amount of responses** from language models before discarding most of them and narrowing down to a smaller subset to train the reward model. Furthermore, this sampling occurs over many iterations, making this process even more expensive. \n\nThe repetitive sampling process, as well as the repetitive reward model training process, is unique to the proposed active learning approach and not present in standard training. It seems that the authors did not account for this part of the training cost. What exacerbates the cost situation for the proposed sampling-intensive active learning is that increasing the LM size could lead to higher sampling costs, even though the cost of preference annotation by human experts remains constant. For this reason, I believe that the cost advantage of the authors' sampling-based active learning approach over standard reward modeling will at best narrow---and more likely reverse---as we scale up the model size to a point where the over-sampling of responses becomes costly enough to negate the benefit of annotating fewer responses. \n\nBased on my reasoning above, I believe a comprehensive cost-performance trade-off comparison between various active learning methods and a standard reward modeling baseline (without active learning or sampling) is needed to validate the argument that active learning reduces training costs. Currently, such cost-performance trade-off experiment is lacking.", "methods_and_evaluation_criteria": "See \"Claims And Evidence\".", "theoretical_claims": "Not applicable, as the paper does not contain theoretical claims or proofs.", "experimental_designs_or_analyses": "See \"Claims And Evidence\".", "relation_to_broader_scientific_literature": "This paper aims to train a reward model using active learning. As the reward model is a binary classifier, the proposed approach is not specifically tailored to natural language processing. The literature on active learning for (binary) classification is most relevant, which is very broad.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "The main weakness, as detailed in \"Claims And Evidence\", is that it is unclear how much active learning can lower the overall training cost of reward model. Cost--performance tradeoff experiments are needed for the argument of reducing the training cost to be convincing.", "other_comments_or_suggestions": "A few typos in the paper:\n\nline 56: $r \\in \\mathbb{R}^D \\to \\mathbb{R}$ should be $r: \\mathbb{R}^D \\to \\mathbb{R}$.\n\nline 4 of Algorithm 1: If I am not mistaken, the argmax should be over $C \\subset \\mathcal{P}\\_{s}$ instead of  $C \\subset \\mathcal{P}\\_{s-1}$.", "questions_for_authors": "I am wondering whether the following approaches are valid for the cost-performance tradeoff comparison:\n\n1. Assign a constant value to represent the cost for each preference annotation.\n2. Measure the FLOPS required for training the active learning procedure (Figure 2), which should include both the reward model training cost and LLM sampling cost.\n3. Compare the overall cost (annotation cost and training FLOPS) of active learning with that of standard reward model training methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gmep2Gb7A6", "reviewer": "Reviewer_Hkj8", "summary": "This paper aims to enhance the reward model in RLHF. Drawing inspiration from active learning, the authors propose Fisher information-based selection strategies to construct an ideal comparison dataset.\nThe experiments show the effectiveness of the proposed method.\n\n## update after rebuttal\nThe authors' response have addressed my concerns, and I decide to raise my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank our reviewer for their time and effort devoted to improving our paper. We have carefully considered each point of feedback and will provide our point-by-point responses below.**\n\n---\n### **P1. Main Contributions**\nWe thank the reviewers for raising the question and reminding us to further highlight our contributions.\nWe would like to note a meaningful machine learning contribution typically involves (1) **improved empirical results** on a relevant task compared to strong baselines and (2) **insights into why the method works**. Our paper proposing active reward modeling meets both: \n1) Empirical success – We adopt a well-established approach to a novel setting, outperforming baselines including [1] (ICML’2024) and the SoTA active learning method from [2]. \n2) Insight – We highlight the trade-off between exploring representation space and comparing uncertain pairs.\n\nWhile active learning is well-studied, our focus is on **applying a classical method to prompt-response selection for reward modeling**. Classical approaches offer robustness, interpretability, and simplicity—our method leverages these strengths while achieving SoTA performance.\n\n### **P2. Applicability beyond the BT Model**\nWe acknowledge the reviewer’s concern that our method is designed around the BT model. The reference [1] in the reviewer’s original review was missing – if the reviewer can provide it, we’d be happy to discuss it further.\n\nWhile improving BT is worthwhile, it remains a widely used framework for modeling human preferences due to its simplicity. Given its role in RLHF workflows, improving sample efficiency within this framework is valuable and lays the foundation for further extensions.\n\nOur approach—leveraging last-layer features and optimizing Fisher information (FI) —is also compatible with other neural network-based statistical models, though evaluating broader applications is out of the scope of this paper.\n\n### **P3. Related Works in Sec 3.1**\nWe thank the reviewer and have moved them to the related work section in our revision.\n\n### **P4. Over-Optimization and Reward Hacking**\nWe understand the reviewer’s concern to be that active learning may lead to overfitting. Indeed, poorly designed active learning can cause overfitting— e.g., always selecting the least uncertain samples may degrade performance, sometimes performing worse than random sampling.\n\nOur FI-based approach mitigates this by encouraging **exploration in the embedding space**, ensuring selected samples remain diverse, and reducing the risk of overfitting to a narrow subset. Additionally, using last layer features—rather than all layers (as in [2])—acts as regularization, preventing over-optimization and improving robustness esp. in the early stage.\n\n### **P5. Hyperparameters**\nThe proposed method is not sensitive to its hyper-parameter choices, and we have extensive ablations studies on all hyper-parameters:\n- The reward model MLP architecture: ablation study in Appdx.A.4.\n- Batch size: ablation study in Appdx.A.2\n- Different base models: we test three of them in Sec 5.1\n\n### **P6. Preference data types**\nThe method is general --- the method can be applied as long as the FI can be calculated.\n\n### **P7. Datasets**\nResource constraints limited our experiments to the datasets we reported. While training reward models are cheap, benchmarking across 5 seeds x 32 hyperparameter settings x 3 LLMs x 8 methods led to a **3840× cost increase**, exceeding 2000 USD per dataset on cloud platforms. Despite this, our experiments provide strong empirical support for the method and we can draw statistically significant conclusions from those extensive empirical results.\n\nThat being said, should there be important insights we need to draw from new experiments, we are more than happy to add them. \n\n### **P8. Initial RM Performance**\n- The initial performance can be seen in e.g. Fig.3 at iteration 125 (for an initial model trained with 125 random samples). Using our strategy can lead to 5-20x higher sample efficiency as compared to the other baselines and more than 2x higher asymptotic performance.\n\n### **P9. Time consumption**\n- The optimization problem for selecting data based on the linear approximation of FI requires a single forward pass to obtain the last-layer embedding and a backward pass to compute the FI, which is fast. Working on the embedding space, the experiment shown in Fig.3 took less than 10 minutes to finish for our method and baselines other than [2] and 2 hours for [2] on a CPU machine.\n\n---\n**We thank the reviewer again for their effort in improving our work. If there should be any remaining concerns or questions, we are keen to do our utmost to address them. Please kindly consider increasing their rating if we address their concerns.**\n\n_**References**_\n\n[1] Active preference learning for large language models\n\n[2] Batchbald: Efficient and diverse batch acquisition for deep Bayesian active learning", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The proposed method is conceptually sound.\n\nHowever, the authors claim that their approach effectively balances the exploration of the representation space and facilitates informative comparisons between pairs. To substantiate these claims, it is essential to incorporate empirical metrics that quantify these aspects. For example, diversity metrics can be employed to measure the extent of exploration.", "theoretical_claims": "The paper does not contain proofs.", "experimental_designs_or_analyses": "No significant issues.", "relation_to_broader_scientific_literature": "The primary contributions of this paper pertain directly to the development and enhancement of reward models within the framework of RLHF.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Other Strengths\n\n1.\tThe authors conduct experiments using multiple LLMs of varying sizes, demonstrating the method's applicability across different model scales.\n\nOther Weaknesses\n\n1.\tAmbiguity in Main Contributions.\nThe primary contribution of this paper remains ambiguous, as the proposed method appears to be a straightforward application of active learning to the reward model without any significant modifications or innovations. \n\n2.\tLimited Applicability to the Bradley-Terry Model.\nThe proposed method is exclusively compatible with the Bradley-Terry model, which may restrict its applicability to other types of reward models[1]. This limitation poses concerns regarding the method's flexibility and its potential integration with alternative reward modeling frameworks. \n\n3.\tInappropriate Placement of Related Works in Section 3.1.\nSection 3.1 predominantly serves as a detailed introduction to various related works, which detracts from its placement within the methodology section. \n\n4.\tIncreased Risk of Over-Optimization and Reward Hacking.\nThe proposed method may inadvertently elevate the risk of over-optimization and reward hacking. By fine-tuning the reward model through active learning strategies, there is a potential for the model to excessively optimize for specific rewards, leading to behavior that exploits loopholes or unintended shortcuts rather than genuinely achieving desired outcomes.", "other_comments_or_suggestions": "None", "questions_for_authors": "1.\tWhat are the key hyperparameters used in this paper, and how were they chosen or tuned?\n\n2.\tHow does the proposed method adapt to scenarios where multiple types of preference data exist?\n\n3.\tWhat is the accuracy of the initial reward model on the dataset, and how does it compare to the final model?\n\n4.\tHave you experimented with datasets other than Anthropic? If so, what were the results?\n\n5.\tWhat is the time consumption of the proposed method, and how does it scale with larger datasets or more complex models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper applies ideas in active learning to improving data selection for training reward models for LLMs. Reviewers were broadly supportive of acceptance. The main objection is from reviewer M1YG, who is focused on the questions of whether the cost of getting labels is really the relevant problem. I agree this is an important consideration. However, there are certainly cases where the labels are the dominant cost (e.g., very difficult problems or applications in protein LLMs), so I find the active learning problem to be well motivated.
Decision: Accept (poster)