Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF
Authors: Syrine Belakaria
Joshua Kazdan
Charles Marx
Chris Cundy
Willie Neiswanger
Sanmi Koyejo
Barbara E Engelhardt
Stefano Ermon
Venue: COLM 2025 (2025)
Keywords: active learning
RLHF
llm
TL;DR: We provide a novel active learning method for RLHF based on the Sharpe Ratio.

Abstract: Reinforcement learning from human feedback (RLHF) has become a cornerstone of the training and alignment pipeline for large language models (LLMs). Recent advances, such as direct preference optimization (DPO), have simplified the preference learning step. However, collecting preference data remains a challenging and costly process, often requiring expert annotation. This cost can be mitigated by carefully selecting the data points presented for annotation. In this work, we propose an active learning approach to efficiently select prompt and preference pairs using a risk assessment strategy based on the Sharpe Ratio. 
To address the challenge of unknown preferences prior to annotation, our method evaluates the gradients of all potential preference annotations to assess their impact on model updates. These gradient-based evaluations enable risk assessment of data points regardless of the annotation outcome. By leveraging the DPO loss derivations, we derive a \emph{closed-form expression} for computing these Sharpe ratios on a per-tuple basis, ensuring our approach remains both \emph{tractable} and \emph{computationally efficient}.  We also introduce two variants of our method, each making different assumptions about prior information. Experimental results demonstrate that our method outperforms the baseline by up to 5\% in win rates against the chosen completion with limited human preference data across several language models and real-world datasets.

Published as a conference paper at COLM 2025
Sharpe Ratio-Guided Active Learning for Preference Opti- mization in RLHF
Syrine Belakaria", Joshua Kazdan', Charles Marx", Chris Cundy", Willie Neiswanger!, Sanmi Koyejo", Barbara E. Engelhardt" Stefano Ermon'
Stanford University* Gladstone Institutes +
University of Southern Californiat
(syrineb, jkazdan, ctmarx, cundy, sanmi, bengelhardt, ermon)@stanford.cdd edu {neiswang Jewaaanneesus..o @usc.. edu
Abstract
Reinforcement learning from human feedback (RLHF) has become a cor- nerstone of the training and alignment pipeline for large language models (LLMs). Recent advances, such as direct preference optimization (DPO), have simplified the preference learning step. However, collecting prefer- ence data remains a challenging and costly process, often requiring expert annotation. This cost can be mitigated by carefully selecting the data points presented for annotation. In this work, we propose an active learning approach to efficiently select prompt and preference pairs using risk as- sessment strategy based on the Sharpe Ratio. To address the challenge of unknown preferences prior to annotation, our method evaluates the gradi- ents of all potential preference annotations to assess their impact on model updates. These gradient-based evaluations enable risk assessment of data points regardless of the annotation outcome. By leveraging the DPO loss derivations, we derive closed fooed-form expression for computing these Sharpe ratios on per-tuple basis, ensuring approach remains both tractable and computationalyy efficient. We also introduce two variants of our method, each making different assumptions about prior information. Experimental results demonstrate that our method outperforms the baseline by up to 5% in win rates against the chosen completion with limited human preference data across several language models and real-world datasets.
1 Introduction
Reinforcement Learning from Human Feedback (RLHF) constitutes the final step of training for modern large language models (LLMs) (Christiano et al., 2017a). RLHF ensures that language models align with human preferences in many aspects, including response length (Singhal et al., 2024), helpfulness (Li et al., 2024), and lack of harmfulness, RLHF can be used to align models according to any criterion of choice from the user and has been extended beyond language to vision (Yang et al., 2024; Wallace et al., 2024) and scientific models (Gu et al., 2025). However, unlike pretraining data, which can be scraped in large quantities from sources such as books, archives, and the internet without requiring annotation, RLHF data costly to gather, as necessitates expert labeling depending on the specific domain (Bai et al., 2022c; Lee et al.).
RLHF data is generally structured as tuples consisting of a single prompt and multiple candidate responses. In an ideal setup, one response within each tuple is labeled preferred, while the remaining responses are marked as rejected. Due to the potentially large volume of such tuples, however, labeling them all is prohibitively expensive and impractical. As a result, only a limited subset of these data points can typically be presented to expert annotators. Established RLHF datasets, for instance, often include only few tens of

---

Published as a conference paper at COLM 2025
Prompt: How do cook polaloas
Response Frying polatoes IS...
Prompt: Whet's the best rouse from Sacrmmento Profered: Across the Oasland Broge...
Ressporse Baking polalies Bi..
CM
Prompt: What's
Respornse Acrossa the that you awold Responiee
Bridge
Bridgg
SHARP-DPO sulecta subout for libeingg based on the Sharpe Rario.
Human Hetwix selecttd
GBU
Prompt:
Yeee You rabulous... Respense 2:1 have some acvioe...
beard look good?
Figure Workflow for pool-based active learning in DPO. First, a user asks the LLM questions. The LLM generates two candidate answers to each question. A subset of the question-responss tuples are chosen for labeling by the user. Then, the model is updated using the collected human preferences.
SHARP: Sharpe Active Requasted Proferences
Prompt
(1) Answer
Loss of 1/
Loss of 8/2 3/1
Sharpe of G Gal
Argmax of Sharpe Ratio
Answer
Prompt
Gradient
Gradient
(1) Answer
G1
Answer
10.1
12,2
Figure illustration of the steps of active learning for RLHF using Sharpe Ratio selection criteria.
thousands of these expert-labeled preference pairs, despite the much larger volume of unlabeled data available (Bai et al., 2022a; Ethayarajh et al., 2022).
The high cost of producing RLHF fine-tuning data leads to investigating more efficient data collection strategies. Models generate millions of responses to human prompts each day; among these, which prompts-if labeled with preference pairs-would provide the greatest benefit during additional RLHF training? Identifying the prompt-responser triplets that yield the highest impact on training could substantially reduce both the time and monetary costs associated with human annotation. This question falls under the broader umbrella of Active Learning(AI) (AL), which aims to determine the most informative samples for model training. (Ren et al., 2021).
Active learning algorithms have demonstrated success for both general statistical models (Castro al., 2005; Tong & Koller, 2000) and deep learning models under supervised learning (Ren et al., 2021). However, relatively few approaches focus applying AL to RLHF for LLMs. In recent work, Muldrew et al. (2024) selected prompt-responses triplets by prioritizing higher reward gaps, while Mehta et al. (2023) used uncertainty metrics to target data where the model appeared less confident. Both of these methods implicitly rely on predicting which response will be preferred, incorporating that assumption directly into the selection process. contrast, our approach accounts for all potential preference outcomes, enabling the assessment of data points regardless of which response chosen by the expert.
More recently, direct preference optimization (DPO) was proposed as an alternative to the traditional RLHF pipeline that simplifies the process of learning from preference-labeled data (Rafailov et al., 2023a). In this work, we present a novel active learning technique that targets an effective selection of data for DPO. We propose to leverage information about the magnitude of gradient update as selection criterion. Before gathering human preferences about which of two responses favored, we note that the gradient update will assume one of two forms, depending on which response is set as chosen. As each of these responses equally likely to be preferred, the resulting gradient update can be seen as a random variable that will settle to one of two values. Rather than relying solely on the expectation or variance this random variable, we draw inspiration from statistical finance and adopt the Sharpe ratio to characterize and compare the potential uShates(Sharpe, 1998). The Sharpe ratio naturally balances the expected improvement (mean) against the uncertainty (standard deviation), making it well-suited to pinpoint samples that promise substantial gains while
2

---

Published as a conference paper at COLM 2025
managing risk. Accordingly, we select prompt-responser triplets that yield the highest Sharpe ratios, focusing on cases with the greatest potential for informativeness.
Importantly, we propose derivation that allows us to obtain closed-form expression for per-tuple Sharpe ratios, circumventing the need for the memory and computationally intensive multiple backpropagations and keeping our method tractable and efficient. We further introduce two variants of our approach. The first, SHARP (SHarpe Ratio-based Active Requested Preferences), assumes all possible annotations are equally likely. The second, W-SHARP, incorporates the implicit reward model as prior, producing a weighted version of SHARP that accounts for varying annotation likelihoods.
By applying our procedures, we achieve up to 5% improvement in win rate over the benchmark dataset's preferred completions, even with a highly constrained data budget, less than 18% of available training tuples in the HH (Bai et al., 2022b) and SHP (Ethayarajh et al., 2022) datasets. We demonstrate the effectiveness of our algorithm across different model ppales-specifically Llama-3-8B and Pythia-2.8B -using two state-of-the-art benchmarks: the Helpful-Harmless (HH) dataset (Bai et al., 2022b) and the Stanford Human Preferences (SHP) dataset (Ethayarajh et al., 2022).
To summarize our contributions:
Drawing inspiration from statistical finance, we introduce risk assessment ap- Our method uses the Sharpe ratio of
proach for active learning in gradient magnitudes determine which data points are most valuable for labeling.
We propose two instantiations of our proposed method. The first assumes that each response is equally likely to be chosen as preferred, while the second uses a prior derived from an implicit reward model to weigh the likelihood of each response.
Leveraging the DPO loss function, we derive fast and memory-efficient closed-form expressions of our acquisition functions.
We demonstrate improvements in win rates popular RLHF datasets using three different LLMs of varying sizes.
2 Background
In this section, we review the details of RLHF and direct preference optimization (DPO). Reinforcement Learning from Human Feedback (RLHF) has emerged as a key approach for aligning language models with human preferences. Originally popularized by works such as Christiano et al. (2017b) and Stiennon et al. (2020), the standard RLHF pipeline begins with a supervised fine-tuning (SFT) phase using high-quality data, followed by training reward model on preference-labeled examples. In the final phase, the policy is further refined through reinforcement learning, where the reward model, reflecting human feedback, serves learned utility function guiding the policy updates via algorithms like Proximal Policy Optimization (PPO) (Schulman et al., 2017; Shao et al., 2024).
A major drawback of traditional RLHF was the need to train a reward function, which increases the computational complexity of the alignment step due to the overhead of separate model. Additionally, reward models are often large, unstable, and might overfit to the preference data (Skalse et al., 2022; Yan et al., 2024; Chaudhari et al., 2024). To obviate the need to train a reward function, Rafailov et al. (2023b) developed direct preference optimization (DPO), an adaptation of the Bradley-Teryy model (Bradley & Terry, 1952) that converts the RLHF pipeline into preference classification problem and uses the language model and the reference model to form an implicit reward model. Specifically, let x be a prompt and y be a response to this prompt. Denoting the policy model by Pa and the reference model by Pret, The RLHF optimization problem is expressed as:
max Ex- (x, 5KL [pa(y]][]]])) xx) Prer(y x)] 9o
(1)
3

---

Published as a conference paper at COLM 2025
The optimal solution to the KL-constrained reward maximization objective leads to expression of the reward model as:
pe(y|x) r(x,y) = blog Pret(ylx)
+ P log Z(x).
(2)
In this equation, & is hyper-parameter that controls the deviation of the policy from the reference policy, and Z(x) is the partition function that depends only on x. Let r* be the ground-truth reward, and ip" be the optimal policy. Under the Bradley- Terry model (Bradley Terry, 1952), the probability that one response preferred over another is:
p" (11 y2/x) or" (x, (xly2))
Substituting in Equation equation 2, the preference probabilities under Bradley-Terry model can be expressed as a function of the optimal RLHF policy q= as follows:
(' > y2/x)
1 + exp (Blog & (yalx) (re(
g' (1(1x)) (1e
Blog
Since we can express the probability of human preference data in terms of the optimal policy rather than a separate reward model, we can construct a maximum likelihood objective for parameterized policy Φo in terms of the chosen yw and rejected ye rewards. This produces a preference classification loss function:
LD yw, = ---=-log-- log (Alog Py x)
Pres x) Bloo Polyi x) Prerly) x)
(5)
While training using the DPO objective, one simultaneously trains the language model and an implicit reward model. This saves substantial time and computation by removing the need to train a separate reward model. In this work, we develop an active learning method for RLHF. Although we experimentally focus on DPO due its lower computational overhead, our method also applies to RLHE.
Related Work
Some estimates suggest that over 80% of engineering efforts in machine learning concern data preparation and labeling (Fredriksson et al., 2020). Active learning (AL), also referred to as optimal experimental design (Olsson, 2009), aims to achieve strong model performance with fewer training samples (Alizadeh et al., 2021). The most common use case for active learning occurs when there is a large pool of unlabeled data, and the scientist training machine learning model must choose which of these data points should be labeled, subject to a labeling budget. In AL, an acquisition function applied to the unlabeled data points is used to perform this selection. AL techniques have been applied across various machine learning domains such as support vector machines (SVM) (Tong & Koller, 2001), image classification (Gal et al., 2017), and other areas (Settles, 2009). Recent efforts in deep active learning (DAL) have focused on text classification (Tuia et al., 2011), image analysis (Wang et al., 2023), and NLP (Hadian & Sameti, 2014). Many active learning methods are based on the principle of uncertainty (Tong & Koller, 2001), wherein the algorithm prioritizes labeling data points that the model is most uncertain about. Other active learning methods emphasize the importance of diversity and exploration when choosing different types of examples to label (Doucet et al., 2024).
Across domains, AL is notoriously difficult problem (Hanneke & Yang, 2010; Castro & Nowak, 2007). Active learning especially challenging for RLHF in large-scale models that lack convexity guarantees or bounded noise. Currently, few works tackle the design of acquisition functions in this context. Recently, Mehta et al. (2023) and Ji et al. (2024) formulated active learning for RLHF and DPO an offline contextual dueling bandit problem. Mehta et al. (2023) proposed an uncertainty- approach, measuring variance in predicted logits under dropout, while Ji et al. (2024) introduced an algorithm with theoretical
4

---

Published as a conference paper at COLM 2025
guarantees regret and query complexity. In parallel, Muldrew et al. (2024) explored active learning in DPO by first selecting sub-batch of prompts with high predictive entropy, then further filtering based on large reward gaps, interpreted lower uncertainty in the DPO model. Although these methods employ different exploration or exploitation strategies, they often require prior assumption about which response is preferred, computing acquisition scores under that assumption. Ideally, an active learning approach should consider all possible preference outcomes without relying on predefined guess. Our method fulfills this criterion, offering first attempt at risk-based perspective that balances exploration and exploitation more comprehensively.
Beyond dueling bandit frameworks, Zhang et al. (2024) introduced a bilevel optimization approach for DPO that favors potentially high-reward responses. Xiong et al. (2024) pro- posed an online exploration method and rejection sampling strategy for offline settings, formulated as a -ivers-kK--rggulariiedd contextual bandit.
4 Problem Setting
Consider practitioner who wishes to fine-tune a large language model (LLM) via rein- forcement learning from human feedback (RLHF) in a specific domain. The practitioner has access large pool of unlabeled data,
D = (x
where и is large, and each entry consists of prompt along with two candidate responses ya and yi2. Owing to the high cost and impracticality labeling every entry in D, only a small subset D,. C D can be annotated with expert preferences (i.e., which of yı or y12 is preferred).
Once the practitioner obtains b labeled triplets from DL, a direct preference optimization (DPO) update is performed the LLM. The model is then used to query new batch of unlabeled data for expert feedback, and this iterative process continues until the labeling budget is exhausted. The key challenge is to select the most informative triplets for labeling to maximize the final performance of the RLHI-fine-tunee
model under strict budget constraints.
To closely mirror practical scenarios of collecting and deploying preference data, we re- quire a criterion that identifies the most valuable prompts for human annotation. In our experimental setup, model this situation as follows:
1. For each prompt response pair in a large batch of size b p, evaluate a designed selection criterion, where p is user-defined fraction indicating the annotation budget. We use this strategy as a practical search procedure.
2. Rank all triplets based on the selection criterion and select the top b to label.
3. Using the labeled preference pairs and perform single DPO update.
Sharpe Ratio for Active Preference Learning
5.1 Method Description
We propose novel method to efficiently collect human preference data in an online setting. Our strategy maximizes the gradient magnitude derived from the DPO objective on the selected data, thereby using information about model parameters when deciding which samples will have the greatest training impact.
A key challenge arises because we cannot compute the DPO gradient without knowing which response is actually preferred. However, we do know that, for each prompt x with candidate responses y1 and 1/2, the gradient will assume exactly one of two possible forms: one if y/1 is preferred, and another if 1/2 is preferred. Let @ref denote the reference model. Depending which response is ultimately chosen, the DPO update takes one of the
5

---

Published as a conference paper at COLM 2025
following two forms:
G₁ = V0LD)o(x,,1112)) = = -Vologo (Blog Po(vilx)
Blog Po(v2|x)
=- - Prer(y1
Prer(y2|x)
x V log po(y1lx)) Ve log (o((xxx)
(Blog (Blog Po(y212)
(yer(y2|x) - Boog po(U1lz))
G2 VoLDro(x,82,1)) - Vo lologo
Prer(yslz)
(x) (xx,,))) x Ve (o Va log Po [x)].
Let be the random variable defined by the magnitude of the gradient update that is obtained by soliciting human feedback for the y2) triplet. Let P1 ply1 y2 be
the probability that y1 preferred to y/2 and p2 = pyy2 y1(x) be the probability that y2 is preferred to y/-
The expectation of is defined as:
E E|G] = Pr G +
(6)
The variance of G is defined as:
= Pc() E(G]
(7)
The expectation alone is not a good decision metric when selecting which responses should be labeled for several reasons. First, suppose that one response is gibberish, and the other is sensible. The gradient in which the gibberish response is the preferred response would likely be large, and therefore, the expectation would be high. However, selecting a tuple where one of the responses is gibberish will not lead to an informative update to the model. Thus, we need some way to account for the variance of G. To do this, we use a tool from statistical finance: the Sharpe ratio. The Sharpe ratio (Sharpe, 1966), invented by William Sharpe in the 1960s, evaluates not just the expected value of an investment portfolio but also the risk. For example, one would likely eschew investment opportunities that could result in losing one's entire life savings, even if these investment opportunities had a high expected value. We apply the same logic when selecting which preference pairs to label. We want to maximize the expected magnitude of our gradient updates but reduce the risk of getting a small gradient update if a certain response is preferred. By choosing to label the preference pairs that yield the highest Sharpe ratio, we accomplish this goal. Because we drew inspiration for our method of active learning from the Sharpe ratio metric, we name our method SHarpe Ratio-based Active Requested Preferences, or SHARP for short. The Sharp ratio triplet (x, (2) is defined as:
G
SR(G)
(8)
(G)
In our active learning setting, we select triplets that yield the highest Sharpe ratio. We define an acquisition function for the current policy Qø as:
0gs (x, 122 = SR(G).
99
No Prior Acquisition Function: Before querying the expert labeling of the prefer- ence, we might have no prior assumption about which response might be preferred to the other. In this case, we can assume that y11 and 1/2 are equally likely to be the better response, and therefore P1 = P2 = 0.5. We consider this the no prior version of our method, and we refer to it as SHARP.
SHARP:
W-SHARP: Paser-based Acquisition Function: The RLHF/DPO pipeline usually initializes the policy @o to the SFT policy previously finetuned data related to the same domain or topic of interest. This model can provide us with a prior for the preference probabilities P1 and p2- For instance, in the DPO setting, we derive implicit reward model from Qe (a(yix) and Piet, yx, = Blog ( and then set the probabilities Pl and p2 based on Equation during the active learning iterations. We refer to this version of our method as weighted
SHARP (W-SHARP).

---

Published as a conference paper at COLM 2025
5.2 Efficient Execution of SHARP with DPO
In practice, computing the Sharpe ratio would require computing the gradient for each element in the dataset twice and consequently backpropagating through the 2 2 x B times for each batch of size instead of a single batch-wise backpropagation. This procedure is computationally expensive terms of both time and memory. To overcome this bottleneck, we use the closed-form expression of the gradient of the DPO loss function to simplify the final expression of the SHARP acquisition functions. Given the final expression of the gradient of the DPO loss, we can express G2 as function of G1:
G2 (o yo()) V log po(v2)x) Vo logog go (y [y]]x)]
(xx((e(x))2)) fz()))1)) 1] [Velog (y((x))) Ve log Po 1 = G G [1- (ffa(x,z2) - f
Consequently, have:
|Gll = ||G|l 1|yll,
(10)
with || ||1
(Fal
Combining Equations 8, Equation 6, and Equation 7, we get an expression of the Sharpe ratio as follows:
P1 P || + 221
SR(G) G
G p2 G2 G
By substituting the expression of Sharpe ratio, in which the gradient terms |G|]| cancel out in both the numerator and the denominator.
|| G2]l from Equation 10, obtain the final form of the
+
SR(G)= = VPI - G[ p Gill( + p2llr)))
(11)
(p1 p2llr]))
(12)
VPI - (p: + P2 p2ll2>/ll)2++ p2(1>|]
(P1 +2(
p2/l))))
In the case of W-SHARP, we substitute the probabilities #1 and p2 by the preference proba- bilities obtained by combining the implicit reward model and the Bradley-Terry preference model (Bradley & Terry, 1952);
& Pp
W-SHAW SHARP (x, SR(G),
(13)
with SR(G) defined Equation 12. In the case of SHARP, where we assume that we do not have any prior about the preference probabilities, we have = P2 = 1. The acquisition function expression can be further simplified follows:
{1 1l
a
SHARP (x,221)122) 9:
(14)
V 1/ [1 11>l)) + 4lml }11 + ll>l))
By simplifying Equation 14, we obtain the final expression:
+ |>|
SHARR Ag
(x, (x,)),
= 1 1
(15)
By leveraging the gradient expression the DPO loss and the relationship between swapped-preferenne gradients and the Sharpe ratio, derivation provides *closse-foomm
7

---

Published as a conference paper at COLM 2025
formula** for per-tuple Sharpe ratios. This circumvents the need for multiple backpropaga- tions, significantly reducing both memory and computation costs and keeping the method tractable. Crucially, without this derivation, although the approach might still be conceptu- ally valid and useful, it would be prohibitively impractical in real-world applications.
We execute SHARP and W-SHARP each batch of incoming unlabeled prompt-responser triplets to select a sub-batch for human labeling. SHARP proceeds as in Algorithm
Algorithm SHARP Data Selection Algorithm
Inputs:
policy Pø, reference policy Pret, exploration parameter B, batch size b, number of iterations N, dataset D ((xi V1 V/2 } the fraction p of the batch that we can afford
to label.
Output: A subset of the data Dr D of triplets of expert labeling with [Pzl b x N, Updated Qg.
1: for =,..,,,,,A NN do
2: Draw large batch of triplets B {(xi V2a 912) (b.p) D.
3: for (x;, V12) B do
SHARR (xi, Vil 912) 4: If using SHARP method, compute a
5:
If using W-SHARP method, compute a W-SHARP (xi, yia, end for
6:
7: Let Bj be the top-b elements of B by the value of the acquisition function a.
8: Request the preferences labels from the expert and add them to DL
9: Update the policy Φe using a gradient step of the LDPO using Bj.
end for
11:
return D2 and Φo.
Win Rate GPT2 Pulicios on HB
Win Rate Pythie-2.88 Policies Hif
Wia Rate of Liama-3 4B Policies HH
Azerage Win Rata Orar Chosen
SHARU DPO
Average Wit Tatte Over Cioosen
SHARP-DD' DPO W-SHARP-DPO
Chooen
SHARP-DD DPO W-SHARF-DPO)
W-SHARF-DPO)
8.44
0.100
DPO
DPO
DPO
0.375
0.370
0.20
Average Win
0.43
2:4
0192
24576
53552
38304
24574
50352
300554
24576
Samples
Samples
Samples
Rate GPT Polictes
Pollcies
Liama Polictis
0.180
Averoge Wis Rato Over Ehossn
SHARP-DRO
Average Win Rate Over Chooee
SHARF-DPO
SHARP-DPO
W-SHIARP-DE
0.190
ALD01
DPO
DPO
DPO
APL
APL
APL
0.004
2LEED
0.125
8.175
0.000
0.10
Averaage
8.170
0.000
20105
0.001
0.100
8152
10384
24575
8102
16364
26376
8192
19364
26356
Samples
Samples
Samples
Figure 3: Comparison of W-SHARP-DPO and SHARP-DPO against DPO and APL across different models and datasets. The metric is the average win rate over chosen completions, computed with CPT-4o under swapped evaluation orders to reduce positional bias. Error bars indicate the standard error.

---

Published as a conference paper at COLM 2025
Win Raia of GPE2 Puliciess on HH
Win Rate of GPTR2 Policios on SHP
Average Win Rate Over Chossn
SHARP-DPO
Rate Chosee
0.100
SHARP-DPO W-SHARP-DPO
W-SHARP-DPO
0.180
DPO
0.005
DPO
0.375
EXP
0.038
EXP
0.091
0.092
0,000
Aueragee
2.100
0.06
0.13
0.044
1/114
Samples
Samples
Figure 4: Comparison against the expectation over the gradient on the GPT-2 model with both datasets.
Experiments
In this section, we provide the details of our evaluation pipeline. Our main goal is to determine we can achieve better or comparable performance as DPO while using smaller amount of labeled data. The standard DPO uses random selection from the dataset. To assess whether our approaches enhance data selection in DPO, we conduct experiments training large language models (LLMs) two datasets applied to three different models with different ranges of sizes. We additionally provide comparison results against the APL baseline proposed by Muldrew al. (2024). When comparing the approaches, we keep all parameters of the experiments identical except for the data selection method to isolate and verify its impact on performance. The code for our approach is publicly available github. www.beeakaaiia/aeeeeeeeaaaa iateeriaa/ssaae sharppeeeeeaaaaaaaaaaaaaae ratio-active active- 1lm- alignment-
Datasets We evaluate both methods on two public datasets: the Anthropic Helpful-Harmlee (HH) dataset (Bai et al., 2022b) and the Stanford Human Preferences (SHP) dataset (Etha- yarajh al., 2022).
Anthropic Helpful-Harmiess (HH): The HH dataset is designed to measure an AI assistant's ability to be both helpful and harmless. It contains two main types of examples: queries where the user's 's request is reasonable and the assistant should provide a helpful response, and queries where the user's request may harmful or inappropriate, requiring the assistant to prioritize safety by giving non-harmful response.
Stanford Human Preferences (SHP): The SHP dataset consists of Reddit posts and correspond- ing human-ger -eeenratd comments spanning 18 different categories. This broad coverage provides diverse human writing styles and topics. SHP focuses on modeling general human preferences across wide range of real-world conversations.
LLMs: We explore the impact active learning by evaluating three models of varying size and capacity: GPT-2, Pythia-2.8-B. and Llama-3-8B. These models span broad range of resource requirements and capabilities, allowing us to assess how active learning strategies perform under different constraints. We conduct six distinct experiments using the above datasets to provide a comprehensive analysis of each model's performance.
Pipeline Setup:
In the DPO pipeline, we begin by splitting each dataset into training and test sets. During the Supervised Fine-Tuning (SFT) phase, we finetune each model on the training split, updating all parameters each gradient step. In the subsequent DPO phase, to efficiently manage computational resources, we apply quantized Low-Rank Adaptation (LoRA) of each L.M.. This approach reduces memory footprint and speeds up experimentation without sacrificing the model's overall performance. We apply 4-bit quantization using double quantization strategy under the NF4 scheme while computing in bfloat16. In addition, rate of 0.05, tailored for causal language modeling tasks and omitting additional bias. We set the batch size of our training to 64 and set the fraction that would be labeled to p 6.
LoRA configuration with rank 16, alpha 32, and dropout
We evaluate model performance using the winrate against the dataset's designated "chosen" completions. Formally, the winrate indicates the proportion of generated responses that are

---

Published as a conference paper at COLM 2025
deemed preferable to those labeled as chosen in the dataset. We recompute this metric after every 4,096 training samples to track performance trends over time.
To underscore the benefits of active learning under constrained resources, we limit the DPO phase to a total of 28,672 training points across all experiments. Additionally, we use GPT-4o as an evaluation oracle to compare each newly generated response against the dataset's designated chosen completions. To mitigate position bias, each pair of responses is evaluated twice with reversed ordering, and we report the average winrate across these two evaluations.
Both W-SHARP- DEHORP--POOAO and SHARP-DPO consistently outperform the standard DPO baseline and the APL baseline (Figure 3). We attribute this improvement to our acquisition function a, which takes the risk (i.e., all possible gradient outcomes) into account when selecting data points. Interestingly, W-SHARP-DPO and SHARP-DPO achieve similar performance, suggesting that incorporating the implicit reward model as a prior does not necessarily yield further gains in this setting. This could indicate that while using a prior might help in other contexts, it not required for effective data selection and making no prior assumption could be beneficial for risk assessment.
The accuracy of the implicit reward model for experiments conducted on both datasets echos this result (Figure 5, Appendix), Although this metric is not our primary focus, the results indicate that both SHARP and W-SHARI tend to attain higher accuracy more quickly the test data, suggesting that these methods guide the model toward more effective reward predictions.
Ablation Study: We additionally provide ablation study to compare against the expected gradient (Figure 4). We conduct the experiments on the GPT-2 model using both datasets. We observe that the SHARP method performs better. Notably, we expect the performance gap to be even larger in noisy datasets that include response pairs where one option is unlikely or nonsensical. Due to the high memory and computational demands of extracting individual gradients for each data point, we were only able to run these experiments with the GPT-2 model. This computational challenge further underscores the strength of the SHARP approach, as its closed-form expression avoids the need for expensive per-sample gradient computations.
7 Summary, Future Directions, and Limitations
We present a novel active learning strategy for RLHF/DPO in large language models, designed to prioritize and label the most impactful data points under limited human annotation budgets. Central to our method is the use of Sharpe ratio-based acquisition function to evaluate potential gradient updates. By selecting examples with the highest Sharpe ratios, aim to target those most likely to produce substantial improvements in policy performance. Our empirical results suggest that this risk-aware selection can reduce annotation costs while enhancing the quality of the learned policy.
Our current approach focuses exclusively on high Sharpe ratio data, which may bias the distribution of selected examples. Although such selective sampling is typical in active learning scenarios, if a practical setting requires an unbiased estimate of the underlying data distribution, future methods could address potential deviations arising from risk-based sampling. Potentially, future methods could combine our Sharpe ratio-based approach with techniques like importance sampling or explicit expectation balancing address such requirements. Moreover, our computational study was limited by relatively modest resources, restricting the scale of DPO training and the range of datasets evaluated. While our findings demonstrate the promise of a Sharpe ratio-based framework, additional inves- tigation across larger tasks and more extensive experiments would establish its robustness and generalizability.
Although in this work we focused on Sharpe ratio, given its intuitive trade-off between the expected benefit (mean gradient magnitude) and the potential downside (variance), alternative risk-aware metrics such as the Sortino ratio could offer interesting inductive biases. Investigating the impact of such metrics is promising direction for future work.
10

---

Published as a conference paper at COLM 2025
References
Azar Alizadeh, Pooya Tavallali, Mohammad Reza Khosravi, and Mukesh Singhal. Sur- vey on recent active learning methods for deep learning. 2021. URL https://api. semanticscholar... www.aar.rrrrrrraaa ::tp//oopss:::::20m : 245763828.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022a. URL 05862.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arv preprint ar dxXie:2204.05862, 2022b.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from feedback. Xr preprint ar arXiv:2212.0807. 2022c
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/ 14):324-34444 1952.
Rui M. Castro and Robert D. Nowak. Minimax bounds for active learning. IEEE Transac- fions on Information Theory, 54:2339-2253, 2007. URL https://api. wttppp..smmaiisccooll.ee. .rg/ CorpusID: 2877584.
Rui M. Castro, Rebecca M. Willett, and Robert D. Nowak. Faster rates in regression via active learning- In Neural Information Processing Systems,, 2005. URL https://api. semanticscholar. org/CorpusID: 1343236.
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, A. Kalyan, Karthik Narasimhan, A. Deshpande, and Bruno Castro da Silva. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for 1lms. ArXiv, abs 0oss/4404085555, 2024. URL httpp:///ppt www.nticscholar.ooooo ottpc//cooooo..ooooooooooo0//200 269113377
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei.
Deep reinformcement learning from human preferences. In Adoances in Neural Information Processing Systems 30 (NIPS 2017), 2017a. URL https: wwwpppppppp....ps...ipppppspsssp www.nnnnncccioiiieioioioooiiitct sc/paper_files, http:/2117//aahh/seeeeaaaaae
html.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017b.
Paul Doucet, Benjamin Estermann, Till Aczel, and Roger Wattenhofer. Bridging diver- sity and uncertainty in active learning with self-supervised pre-training. arxiv preprint arXiv:2403.03728, 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Con- ference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp- -088-6008.. PMLR, 17-23 Jul 2022.
Teodor Fredriksson, David Issa Mattos, Jan Bosch, and Helena Holmström Olsson. Data labeling: An empirical investigation into industrial challenges and mitigation strategies. In Maurizio Morisio, Marco Torchiano, and Andreas Jedlitschka (eds.), Product-Focussee Software Process Improvement, pp. 202-216, Cham, 2020. Springer International Publishing. ISBN 978-3-030-641488-1
11

---

Published as a conference paper at COLM 2025
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, PP. 1183-1192. PMLR, 2017.
Siyi Gu, Minkai Xu, Alexander Powers, Weili Nie, Tomas Geffner, Karsten Kreis, Jure Leskovec, Arash Vahdat, and Stefano Ermon. Aligning target-aware molecule diffusion models with exact energy optimization. Advances in Neural Information Processing Systems, 37:44040-44063, 2025.
Hossein Hadian and Hossein Sameti. Active learning in noisy conditions for spoken language understanding, In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 1081-1090, Dublin, Ireland, August 2014. Dublin City University and Association for Computational Linguistics. URL https: 1elianttology.org//11-11111
Steve Hanneke and Liu Yang. Negative results for active learning with convex losses. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Stafistics, volume 9 of Proceedings of Machine Learning Research, pp. 321-325, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https://proceedingg mlr. wwwpppresssseeeeee //haneeee @0a html.
Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement learning from human feedback with active queries, arXiv preprint arXiv:2402.09401 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first International Conference on Machine Learning.
Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang, Weidong Guo, Yukang Gan, Xu Yu, Daniell Wang, and Ying Shan. Hrlaif: Improvements in helpfulness and harmlessness in open-domain reinforcement learning from ai feedback, 2024. URL
http:/oogg/ab////////// 08389.
Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via active exploration. 2023.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. ar Xiv preprint ar X0o:2402.08114, 2024.
Fredrik Olsson. A literature survey of active machine learning in the context of natural language processing. 2009.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2023a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirth- Conference on Neural Information Processing Systems, 2023b. URL
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yo- Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning, 2021. URL http::////// 0tt/22009.00233
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. ArXiv, abs/1700700347, 2017. URL https://api. semanticscholar. org wwwg///ooosss:::..oo 28000pps11::229052222 28695052.
Burr Settles. Active learning literature survey. 2009.
12

---

Published as a conference paper at COLM 2025
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ar arviv preprint arXXi:2402.03300, 2024.
William F. Sharpe. Mutual fund performance. The Journal of Business, 39(1):119-38,, 1966. ISSN 00219398, 15375374. URL htttp:://www..e
William F Sharpe. The sharpe ratio. Streetwise-the Best of the Journal of Portfolio Management, 3:169-185, 1998.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigat- ing length correlations in rlhf, 2024. URL
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining
and characterizing reward hacking. ArXiv, abs/2209.13085 2022. URL https://api.
semanticscholar.c wwtpg//Crpus::::::mm 252545255.
N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, Lowe, C. Voss, Radford, D. Amodei, and P. Christiano. Learning to summarize from human feedback. 2020.
Simon Tong and Daphne Koller. Active learning for parameter estimation in bayesian net- works. In Neural Information Processing Systems, 2000. URL https://api. URL.ttpss//ppp...eaatccoooooo semaaticcccctn..............
org/CorpusID:23386 2386340.
Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. Journal of machine learning research, 2(Nov):45-66, 2001.
Devis Tuia, Michele Volpi, Loris Copa, Mikhail Kanevski, and Jordi Munoz-Mari. A survey of active learning algorithms for supervised remote sensing image classification. IEEE Journal of Selected Topics in Signal Processing, (13):606-677, 2011. doi: 10.110///STSP2201.2133/9
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purush- walkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the TEEE/CVF Conference 2024.
on Computer Vision and Pattern Recognition, pp.
Haoran Wang, Qiuye Jin, Shiman Li, Siyu Liu, Manning Wang, and Zhijian Song. A comprehensive survey on deep active learning in medical image analysis. Xiv preprint arXie:2310.14220, 2023. URL https: / //ooi. 0rg/10. 00..48550/rraa wtt.///.vvv../////////oo/aaaaeva 2310 14230.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint. In Forty-first International Conference Machine Learning, 2024.
Yuzi Yan, Xingzhou Lou, J. Li, Yipin Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in llms. ArXiv, abs/2409.15366 2024. URL https: //api. semantiscccc org wrgg/CopussI::: /Corppus1::2228118811 272831831.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In CVPR, pp. 8941-8951, 2024. URL https: / org/10 1189/CVPRS2733. 2024.00854. 00854.
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. Xiv preprint arXio:2405.11 19332, 2024.
13

---

Published as a conference paper at COLM 2025
Appendix
A Additional Results
In this section, we provide additional results reporting the accuracy of the implicit reward model on the test set. provide informative results, we use exponential moving average smoothing.
Acnuracy of Rewvord of GPT 2 on
Accurnny Reware of 2ythls 2:68 cm HHH
Accurncy Reward of Lamo 3 83 on HH
BRD
SUMIISERO
DPO
0.020
SH43P EPO
202
3.51
3.10
2312
3132
16:244
38
20555
51.02
18164
201/8
2045
801/2
1hoot
245/3
Sumpptes
Sorpplex
Semrples
Accuracy Rowand of CPT 2 nn SEP
.Sccurscy of Reward Pythia 2.58 on SHP
Actursey of Reward of Liama 88 SHP
D20
0.00
DPO
SEARP DFD
0.82
20140 LIN
0.58
S2UAIP EIDO
D.62
1313
0.20
Buforouc: Accunaty
DU62
D00
0.54
1.54
0.52
002
0.02
0.50
3340
18351
24576
01.02
10184
20376
204-3
01:02
16114 Sarepies
28378
Sample
Semplex
Figure The accuracy the implicit reward model models.
14


Reviews:
[{"review_id": "Ya8bZqSc97", "reviewer": "Reviewer_ZGMy", "summary": "This paper presents an active learning method to acquire labeled preference tuples for LLM fine-tuning.\n\nFor each unlabeled preference tuple (consisting of a prompt and two candidate responses), they compute the magnitude of the the DPO gradient update under the two possible preference orderings, then they compute the Sharpe ratio (average / standard deviation) of these two values, and use this as an acquisition function to submit the tuple to the human annotator for labeling. The authors also provide a derivation which enables them to compute the Sharpe ratio efficiently by only doing one DPO computation rather than two.\n\nThey evaluate their methods on standard LLMs and standard datasets, comparing the win rate of the preferred response on a test set over vs the number of training examples, comparing against DPO (presumably with randomly selected examples) as the baseline, showing improvements.\n\nWhile the method is interesting, and, as far as I know, novel, I notice a few areas of concerns:\n- The authors mention running SFT on the positive examples before DPO or their methods, as it is standard practice, but how are the SFT examples labeled? If they use all the dataset, then this defeats the purpose of doing active learning.\n- The improvements are very small, and the authors only consider DPO, with presumably random chosen examples, as the baseline. How does their method compare to other active learning approaches?\n- I'm not sure I buy the argument for using the Sharpe ratio as opposed to the unnormalized magnitude of the gradient updates. This has to be evaluated empirically, and the authors did not do it.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting, novel, improvements.", "reasons_to_reject": "Weak baseline, small improvements, no relevant ablations.\n\nEDIT:\n\nThe authors reported more baselines and ablations.", "questions_to_authors": "- How are the SFT examples labeled?\n- How does using the expected gradient magnitude as opposed to the Sharpe ratio perform?", "extra_scores": {}}, {"review_id": "BDfNhum8Yo", "reviewer": "Reviewer_MNhU", "summary": "This paper introduces a novel active learning approach for Reinforcement Learning from Human Feedback (RLHF), specifically focusing on improving the efficiency of preference data collection for Direct Preference Optimization (DPO). The authors propose SHARP (SHarpe Ratio-based Active Requested Preferences), a method that uses the Sharpe ratio from statistical finance to evaluate and select the most informative prompt-response pairs for human annotation. The key innovation is assessing the risk-reward tradeoff of potential gradient updates before knowing which response will be preferred. The authors derive closed-form expressions for computing Sharpe ratios efficiently and introduce two variants: SHARP (assuming equal likelihood of preferences) and W-SHARP (incorporating a prior from an implicit reward model). Experiments across multiple language models and datasets demonstrate that their method outperforms standard DPO by up to 5% in win rates against chosen completions while using significantly less labeled data (less than 18% of available training tuples).", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper introduces a novel and well-motivated approach to active learning for RLHF that alleviates a practical challenge: the high cost of collecting human preference data. The use of the Sharpe ratio as a risk assessment metric for data selection is an innovative application of a concept from finance to machine learning.\n\n2) The authors provide a mathematically rigorous derivation of closed-form expressions for their acquisition functions, making the approach computationally efficient and practical for real-world applications. This avoids the need for multiple backpropagations, which would otherwise make the method prohibitively expensive.\n\n3) The empirical evaluation is comprehensive, testing the approach across different model sizes and on multiple established datasets, demonstrating consistent performance improvements over standard DPO with limited labeled data.\n\n4) The method is model-agnostic and can be integrated into existing RLHF/DPO pipelines without significant modifications, increasing its practical utility for the broader AI community.", "reasons_to_reject": "1) The paper lacks a thorough ablation study to understand the contribution of different components of the proposed method. For example, it would be valuable to see how performance varies with different definitions of risk or alternative formulations of the Sharpe ratio.\n\n2) While the authors mention that their approach can be applied to traditional RLHF beyond DPO, they only provide experimental results for DPO. This limits the understanding of how well the method generalizes across different RLHF frameworks.\n\n3) The evaluation methodology relies heavily on GPT-4o as an evaluation oracle, which introduces potential biases and limitations. The paper would benefit from additional human evaluation or alternative automated metrics to validate the findings.\n\n4) The paper does not sufficiently address potential distribution shifts that might occur due to selective sampling based on Sharpe ratios. It's unclear how this sampling strategy might bias the model training over time and whether this could lead to performance degradation in certain domains or tasks.\n\n5) The experimental setup is limited to a relatively small training budget, and it's unclear how the benefits of the proposed method scale with larger datasets or longer training regimes.", "questions_to_authors": "1) Could you provide additional ablation studies showing how different formulations of the risk component in the Sharpe ratio affect performance? For example, what happens if you use only the expected gradient magnitude without considering variance?\n\n2) Have you conducted experiments applying your method to other RLHF with explicit reward models? If so, how does the performance compare to the results presented in the paper?\n\n3) Have you investigated whether the data selected by your method exhibits any systematic biases compared to randomly selected data? If so, how might these biases affect the model's performance on diverse real-world queries?\n\n4) How does the performance gap between your method and standard DPO change as the labeling budget increases? Is there a point at which the benefits of active learning diminish?", "extra_scores": {}}, {"review_id": "PREklzwwZJ", "reviewer": "Reviewer_MhkP", "summary": "This paper introduces a novel active learning approach called SHARP aiming at selecting data points for annotation by using a risk assessment strategy based on Sharpe ratio. Drawing inspiration from Sharpe ratio, this method models selection criterion with both the expectation and the variance of the magnitude of gradient update. The paper claims the method could achieve more effective DPO training under same constrained annotation budgets. The paper proves the claim by comparing win rates of DPOs using different data selection methods (SHARP, weighted variant W-SHARP and standard) on HH-RLHF and SHP dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The method provides a new perspective to apply AL to RLHF methods without heuristic preference assumptions.\n2.The method outperforms DPO under standard data selection strategy and has a data assessment method that is computationally efficient.", "reasons_to_reject": "1.The proposed data selection strategy promotes the sample with higher gradient magnitude, which is not necessarily corelated to better influence. Lacks further discussion.\n2.Some details in the experiments remain unclear.\na)\tWhat is the data selection strategy for the standard DPO?\nb)\tThere seems to be an inconsistency regarding the definition of win rates. On page 9, win rates are presented as “the winrates against the dataset’s designated chosen completions.” However, the subsequent paragraph indicates that they “compare each newly generated response with the baseline DPO generation.”", "questions_to_authors": "Please refer to the reasons to reject.", "extra_scores": {}}, {"review_id": "qatvI1pJAR", "reviewer": "Reviewer_C2ud", "summary": "The paper proposes a novel active learning method for RLHF/DPO in LLMs, leveraging the Sharpe ratio—a financial risk-assessment metric—to prioritize data points that balance expected gradient update magnitudes and their variance. Two variants, SHARP (no prior) and W-SHARP (with reward model prior), are introduced. The authors derive closed-form expressions for efficient computation, avoiding multiple backpropagations. Experiments on the HH and SHP datasets demonstrate improved win rates over baselines with constrained data budgets (18% of training data), validated across models like Llama-3-8B and Pythia-2.8B.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Innovative Methodology: The application of the Sharpe ratio to active learning is novel, effectively addressing the exploration-exploitation trade-off in RLHF data selection.\n\nComputational Efficiency: The closed-form gradient derivation reduces memory and computation costs, making the method scalable for large models.", "reasons_to_reject": "Theoretical Limitations: The choice of the Sharpe ratio lacks rigorous theoretical justification compared to alternative risk-aware metrics (e.g., Sortino ratio). Why the authors choose sharpe ratio as the foundation metric? What happened if we choose other ones?\n\nNarrow Evaluation: Experiments are limited to two datasets and lack diversity in tasks (e.g., harmlessness vs. general preferences). Broader validation is needed.\n\nEvaluation Concerns: Reliance on GPT-4o as an oracle evaluator introduces potential positional bias, which is only partially mitigated by order swapping.\n\nBehind Reason: I don't think that the gradient of loss function measures the importance or quality of the samples. It only measures the influence of the sample to the parameters, but this is totally different with the quality of the sample itself. Please give a explanation here why these two metrics are related with each other and some experiments to show this.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new way to do RLHF, based on an idea from finance (Sharpe ratio) to select from the preference data. The authors use a closed-form solution for this, and it runs efficiently. The new ablation study and new baseline demonstrate the advantage of the approach.

This paper seems to be in possibly a good shape for publication, even though most of the reviewers asked for additional experiments. The authors added a specific set of experiments. I am not completely certain it addresses all the requests (it is a limited set of experiments that tries to address all of the requests), but reviewers, overall, seem to be satisfied.
Decision: Accept