Reinforcement Learning from Human Feedback with Active Queries
Authors: Kaixuan Ji
Jiafan He
Quanquan Gu
Venue: Accepted by TMLR ()

Abstract: Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (RLHF). Despite their superior performance, current RLHF approaches often require a large amount of human-labelled preference data, which is expensive to collect. In this paper, inspired by the success of active learning, we address this problem by proposing query-efficient RLHF methods. We first formalize the alignment problem as a contextual dueling bandit problem and design an active-query-based proximal policy optimization (APPO) algorithm with an $\tilde{O}(d^2/\Delta)$ instance-dependent regret bound and an $\tilde{O}(d^2/\Delta^2)$ query complexity, where $d$ is the dimension of feature space and $\Delta$ is the sub-optimality gap over all the contexts. We then propose ADPO, a practical version of our algorithm based on direct preference optimization (DPO) and apply it to fine-tuning LLMs. Our experiments show that ADPO, while only making about half of queries for human preference, matches the performance of DPO, establishing it as a data-efficient alternative to DPO. The codes are available at https://github.com/jkx19/ActiveQuery.

Published in Transactions on Machine Learning Research (09/2025)
Reinforcement Learning from Human Feedback with Active Queries
Kaixuan J1
kaimannjisss.cloe ucla. edu
Department of Computer Science University of California, Los Angeles
Jiafan He
jiafonhel 19 @ edu
Department of Compuler Science University of Colifornia, Los Angeles
Quanquan Gu
agu vcla edu
Deportment of Computer Science Universily of California, Los Angeles
Reviewed on OpenReview: https: // openreview. net/ forum? id= ESca QQRmz
Abstract
Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (RLHF). Despite their superior performance, current RLHF approaches often require a large amount of human-labelled preference data, which expensive to collect. In this paper, inspired by the success of active learning, we address this problem by proposing query-efficient RLHF methods. We first formalize the alignment problem as contextual proximal policy optimization
dueling bandit problem and design an active-query- (APPO) algorithm with an Ó(d² /Ad instance-dependen regret bound and O(d²/A²) query complexity, where d is the dimension of feature space and A is the sub-optimality gap over all the contexts. We then propose ADPO, practical version of our algorithm based on direct preference optimization (DPO) and apply it to fine-tuning LLMs. Our experiments show that ADPO, while only making about half queries for human preference, matches the performance of DPO, establishing it as a data-efficient alternative DPO. The codes available https://githh: wtw.ttuu...mmm/aaatt...te http://jxx9/Acttveeeeee
1 Introduction
Recent breakthroughs in large language models (LLM) significantly enhance the performances across a wide range tasks, including common sense reasoning, world knowledge, reading comprehension and math problem solving (Jiang et al. 2023 Touvron et al., 2023 Chiang et al. 2023 Tunstall et al. 2023). addition to
the prominent capabilities of traditional natural language tasks (Gao et al.) 2023a Yuan al.] 2023 Han
et al. 2023 Wei et al. 2023 they also demonstrate great potential in responding to human instructions Ouyang et al.) 2022).
One key step towards building these models is aligning them with human preference, where reinforcement learning from human feedback (RLHF) FFürnrranz et al.
2012 Casper et al. 2023
Ouyang et al.) 2022 Ziegler et al., 2019; Christiano et al. 2017 Rafailov et al, 2024 is widely employed.
The orthodox process of RLHF (Gao et al.) 2023b: Munos et al. 2024) is described as follows. At each time, the human user prompts the LLM with an instruction. Subsequently, the model generates several candidate responses and queries the users for their preferences. Then, a reward model is trained on this preference data mimic human evaluation. The language models are then updated using reinforcement learning (RL) algorithms such Proximal Policy Optimization (PPO) (Schulman al.) 2017) to optimize responses that maximize the reward. However, PPO requires an additional reward model and ouline sampling from LLMs, which is computational inefficient. Alternatively, Direct Preference Optimization (DPO) (Rafailov et al.
"Equal Contribution

---

Published in Transactions on Machine Learning Research (09/2025)
2024) directly treats the language models themselves as the reward models and optimize the LLMs the offline datasets. While its objective is mathematically equivalent to its canonical counterpart, it eliminates the requirement of additional reward modeling and online sampling.
Despite the notable success of RLHF in aligning language models with human preferences, its practical implementation often necessitates significant amounts of human-labeled preference data. For instance, the fine-tuning process of Sephyr-7b-best
through RLHF relies on the utilization of a sizable 62k UltraCat- binarized dataset (Ding et al. 2023) J. The collection of such a substantial volume of human preference data is both costly and inefficient. Therefore, there exists a pressing need to develop query-efficient RLHF methods for effectively aligning large language models with human preferences.
Following recent. theoretical advancements in RLHF (Xiong et al. 2023
Zhu et al. 2023
Sekhari et al.
2024), we formulate the RLHF problem as a contextual dueling bandit problem (Yue et al. 2012 Wu & Liu
2016 Saha 2021 Saha & Krishnamurthy 2022; Saha & Gaillard 2022 Wu et al. 2024
Di ct al.. 20240.
In this setting, the learner proposes a pair of actions and receives noisy leedback regarding the preference
between the dueling pair for each round. While numerous studies address regret minimization in dueling bandits, only few works Wang et al. 22023) Zhan et al. (2023); Wu & Sun (2023); [Skhari et al. (2024) have considered query complexity during the learning process. However, their results either exhibit a linear dependency on the size of the action set A, limiting the practical applicability of their methods, or fail to
provide instance-dependent regret, thereby missing the opportunity to exploit favorable large-suboptimal-gap
structures in RLHF In this paper, we adopt the principles of active learning (Zhang & Oles query-effciien algorithm, Active Proximal Policy Optimization (APPO) for linear contextual dueling bandits. In each round, APPO employs the maximum likelihood estimator (Di et al. 2024) to estimate the underlying parameter and constructs an optimistic estimator for the reward gap between different arms. Subsequently, APPO selects the best arms and estimates the uncertainty associated with the potential feedback. To reduce the query complexity, APPO selectively queries the dueling preference and updates the parameters only when the uncertainty of the observation exceeds a threshold.
2000 Hoi et al. 2006) to design a
We further extend APPO to direct preference optimization (DPO) (Rafailov et al. 2024) and introduce novel query-efficient method, Active Direct Preference Optimization (ADPO). Following the methodology of APPO, ADPO selectively queries human preference only for data where the model exhibits high uncertainty about the observation. For data where the model is less uncertain about, we employ the pseudo label predieted by the model to fin-ttne the model itself. Our contributions are summarized as follows.
We propose an active-learning based algorithm APPO for linear contextual dueling bandits. Theoretical analysis shows that our algorithm enjoys a constant dependet regret / where d is the dimension of the feature space, and A is the minimal sub-optimal gap. Mcanwhile, our proposed algorithm only requires O(d3/A² queries in total T rounds. Compared with previous instance deeenddnt regret bound 32 A) achieved by SSkhari et al. (2024 3 where A is the size of the action space, our regret bound is independent on the size of action space A, which is more favorable in practice.
We propose an serive-leaniing-basee DPO method named ADPO. We apply our method to train zephyr-7b-beta Ultrafeedback- binariize dataset (Ding et al. 2023) and zephyr-7o-gen -eagmma on dpo- mix-7k dataset. Our experiment shows that while ADPO only make less than half numbers of queries, the model trained by ADPO achieves a comparable or better performance than DPO on our selected benckmarks including MT-Bench (Zheng el al.] (024) and AlpacaEval 2.0.
Notation
We employ [n] to denote the set (1.....m). In this work, use lowercase letters to represent scalars, and denote vectors and matrices by lower and uppercase boldface letters respectively. Given a vector x E Rd, we denote the vector's L2-norm by ||x||2- We further define ||x|| = Vx Σx given a positive semidefinite matrix Σ E Raxd We use standard asymptotic notations D(),.()) ()))()),()))) and further use Ö()) to hide logarithmic factors other than the number of rounds T. We use (.} denote the indicator function.
For instance, in the Ultrafeedback-binae binarized dataset, the minimal reward gap is 0.5, within a range of 0 to 10. This occurs because users typically cannot perceive subtle quality differences between responses.
We notation O(-) to hide the log factor other than number of rounds T In our work, we only focused on the regret one selected action, which is slightly different from the two-arm regret in Sekhari et al. (2024) See Section 3 for further discussion.
2

---

Published in Transactions on Machine Learning Research (09/2025)
2 Related Work
Reinforcement Learning from Human Feedback Learning from human preference data dates back to Wirth et al. (2017):
Christiano et al. (0177) and is recently popularized by generative language models Achiam et al., 2023 Touvron et al.) 2023). This procedure usually takes place after supervised finetuning
(SFT). The canonical procedure of aligning with human preference includes two stages: reward modeling and
reward maximization (Ouyag et al. 2022 Bai et al. 2022 Munos et al. 2024). Another approach is direct preference optimization (DPO) (Rafailov et al. 2024), which treats the generative models directly as reward models and trains them on preference data. Compared with the first approach, DPO simplifies the aligning process while maintaining its effectiveness. However, both paradigms require a large amount of human preference data. In this work, we follow the DPO approach and study its query-efficient modification. The empirical success of RLHF also prompts a series of theoretical works, with a predominant focus on the reward maximization stage, modeling this process as learning a dueling bandit (Zhu et al. 2023 Xiong et al. 2023 Sckhari ct al.
2024). Among these works, Wang al. (2023): Zhan et al. (2023); Wu & Sun (2023): Sekhari
et al.]
2024) stand out for considering query complexity in the learning process. However, Wang et al. 2023)
(2023)- Wu & Sun (023)) focus either worst-case regret bounds or the sample complexity to Zhan et al.
identify an c-optimal policy, but fail to provide instance-dependent guarantees (Further discussion is deferred
to Appendix B .Only Sekhari et al. (2024) offer an instance- dependent analysis; however, their upper bound is oAA2 which depends on the size of the action set A, limiting the practical applicability of their algorithm. Compared to this work, we provide an instance-dependen regret guarantee without dependency on the action space. Furthermore, based on APPO, we derive practical algorithm, ADPO, which we empirically verify to demonstrate its superiority. We also notice two concurrent works that incorporate active learning with DPO. MMehta et al. (2023) incorporate active learning to DPO and use the variance of log-probabilitee under different dropouts as the uncertainty estimator, which is inefficient in practice. Mulldew et al. (2024)
also proposed an active learning- barning-based alternative of DPO and leverage reward difference as uncertainty estimator. However, their approach does not involve pseudo labels, which is key component of our approach.
Active Learning To mitigate the curse of label complexity, active learning serves as a valuable approach in supervised learning. The first line of work is pool-based active lcarning (Zhang & Oles 2000 Hoi et al.
2006 Gu el al. 2012 2014; Citovsky et al. 2021). In pool-based aclive learning, instead of acquiring labels for the entire dataset, the learner strategically selects a batch of the most informative data at each step and exclusively queries labels for this selocted data batch. The learner then employs this labeled data batch to update the model. Subsequently, guided by the enhanced model, the learner queries another mini-batch of labels and continues the training process. These steps are iteratively repeated until the model achieves the desired performance level. The strategic selection of informative data significantly reduces the label complexity for supervised learning. The label complexity of basel active learning has been extensively
studied by Dasgupta 2005): ; Dasgupta et al.
(2005) Balcan et al. 22006 2007 Hanneke & Yang (2015): been widely applied in tasks like robotics learning AAkrour
Gentile et al. (2022). This strategy has also
et al. 2012 Biyik & Palan 2019: Wilde et al.
2020 On the other hand, selective sampling (a.k.a., online
2009 Hanneke & Yang. 2021) is a learning framework that integrates online learning and active learning. In this framework, the algorithm sequentially observes different. examples and determines whether to collect the label for the observed example. In reinforcement learning, there are also lines of works focusing on the application of active learning. On theoretical side, Schulze & Evans (0188) Krueger et al. (2000;; Tucker et al. 22023) focuses on active reinforcement learning and directly integrates the query cost into the received reward. Krueger et al. (2020) laid the groundwork for active reinforcement learning by introducing a cost e associated with each reward observation and evaluated various heuristic algorithms for active reinforcement learning. Recently, Tucker et al. (2023) studied the multi-arm bandit problem with costly reward observation. Their work not only suggests empirical advantages but also proves an (TT233) regret guarantee. On the application side, there are also lines of works apply selective sampling to specific circumstance of RLHF in robotics (Lee et al. 2021a b. Liang et al. 2022
active learning) (Cesa-Bianchi et al.) 2005 2006
3 Preliminaries
In this work, we formulate the RLHF problem as a contextual ducling bandit problem (Saha 2021 Di ct al. 2024).. We assume context set X, and at the beginning of each round, a contextual variable 2, is i.i.d generated from the context set X with the distribution D. Based on the context 34, the learner then chooses
3

---

Published in Transactions on Machine Learning Research (09/2025)
two actions vi vi from the action space A and determines whether to query the environment for preferences between these actions. If affirmative, the environment, generates the preference feedback with the following probability P(o, = 1 1 vi vi) = o (r(I), (()) - r(æ,, vī)), where a--) ; IR + [0, 1] is the link function and r(.,)) is the reward model.
We consider a linear reward model, e-g., r(x, y) (0", Φ(x; v)), where 0t E Rd and Φ X x A - Red is a known feature mapping. For the sake of simplicity, we use Φf of to denote Φ(I,, vi), Φ(I,, vF)) Additionally, we assume the norm of the feature mapping Φ and the underlying vector 0 are bounded.
Assumption 3.1. The linear contextual dueling bandit satisfies the following conditions:
For any contextual x E X' and action y E A, we have ( 22 - 1/2 and r(æ, v) S 1.
For the unknown environment parameter 0", it satisfies | || B.
For the link function o, we make the following assumption, which is commonly employed in the study of generalized linear contextual bandits Fillippi et al. 2010 Di et al. 2024).
Assumption 3.2. The link function is differentiable and the corresponding first derivative satisfied I o(-,, where Ko > 0 is a known constant.
The learning objective is to minimize the cumulative regret defined as:
T r" (1t) - r(t,,
Regret (T) Σ
where (xxe) = r" (1t,, vt) max r (r y) stands for the largest possible reward context 2y- is worth noting that prior works (Di et al. 2024 Saha & Krishnamurthy 2022 Sekhari et al. 2024) in dueling bandits often define the regret on both action ot and v². However, in the context of RLHF, the model generates multiple candidate responses, and users will choose the most preferable response from the available options. Under this circumstance, sub-optimality is only associated with the selected response. Therefore, we choose the regret defined only on action vł-
To quantify the cost of collecting human-labeled data, introduce the concept of query complexity Query(T) for an algorithm, which is the total number of data pairs that require human feedback for preference across the first T rounds. Note that while some prior work (Tucker et al. 2023) counts the cost of requesting for human feedback together with the cost paid for taking certain action, in our approach, we distinguish between regret and query complexity as two separate performance metrics for an algorithm.
In addition, we consider the minimal sub-optimality gap (Simchowitz & Jamieson 2019 Yang et al. 2021 He et al. 2021), which characterizes the difficulty of the bandit problem.
Definition 3.3 (Minimal sub-optimality gap). For each context x E X and action E A, the sub-optimality gap A (z. y) and the minimal gap A are defined as
( = (x) r(æ,8), A min ZER
(A(c, A (x, + 03.
In general, a larger sub-optimality gap A between action and the optimal action y" implies that it is easier to distinguish between these actions and results in a lower cumulative regret. Conversely, task with a smaller gap A indicates that it is more challenging to make such a distinction, leading to a larger regret. In this paper, we assume the minimal sub-optimality gap strictly positive.
Assumption 3.4. The minimal sub-optimality gap is strictly positive, ie.. A>0.
Remark 3.5. In the context of RLHE, given a prompt, the minimal sub-optimality gap A represents the uniform gap between the best answers and the sub-optimal answers, where the optimal answers (or arms in the context. of bandits) might not be unique (See Definition 3.3). Typically, for arms with sub-optimality close to 0, is difficult for humans to discern a quality difference between them. Under this situation, we can roughly consider these arms also optimal arms (optimal arms may not be unique) and only consider the gap between sub-optimal arms and these optimal arms. Therefore, Assumption 3.4 is mild in the context of RL.F.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm Active Proximal Policy Optimization (APPO)
Require:
Regularization parameter À >> 0, and B, an upper bound on the εz-norm of 0", confidence radius 3, uncertainly threshold 0, learning rate 7)
Set initial policy T1(-1-) uniform distribution over the action set A, Eg t AI, Co 0
for t == ..... 7 do
Compute the MLE 0, in 4.1 and observe A, select yi Uniform(A)
Compute (x(,). v) = min| (0,, ( v) - Φf) + ( y) &ill,,
5: Choose y = argmax, (I,,
6:
if || 0 -, I then
7:
Keep Eg = -1 T(+1(a = (aass) and C. = Ci-1
else
Sample o2 Te (+1s,), query for the preference and observe 0t
10: Updale E. Σ.-1 (o) (8) Φi) and C. C. (1)
11: Update Te+1 (v/lr) x Te re exp (y.
12: end if
13:
end for
4
Algorithm
In this section, we introduce our proposed query-efficient method for aligning LLMs. The main algorithm is illustrated in Algorithm 1 At high level, the algorithm leverages the uncortainty-aware query criterion
(Zhang et al. 2023) to issue queries and employs Optimistic Proximal Policy Optimization (OPPO) (Cai
et al. 2020 He et al. 2022a for policy updates. In the sequel, we introduce the key parts of the proposed algorithm.
Regularized MLE Estimator For each round [T], we construct the regularized MLE estimator Fillippi et al. 2010: Li et al. 2017) of parameter 0" by solving the following equation:
o of 10--
1O
(4.1)
where C. denotes the of rounds up the t-th round for which the preference label is required. Compared with previous work on linear dueling bandits (Sahal 2021 Di et al., 2024), here only requires part of the human-labelle, preference. We construct the MLE estimator with only rounds e C.. In addition, the estimation error between 0, and 0r satisfies
110 Or S 5( [/Vooo/CC//a)) (CI/No))
After constructing the estimator 0,, the agent first selects a baseline action vi and compares each action y E A with the baseline action v For simplicity, we denote De(1s, v) = (0", Φ(11,)) - Φf) as the reward gap between y and action d Then, we construct an optimistic estimator D. for the reward gap with linear function approximation and Upper Confidence Bound (UCB) bonus, i.e.,
De( v) min( ((x,))))))) v) Φi) + -l| y) 1111,,1113
Here we truncate the estimation since the true reward is in [0, 1] and therefore their difference is bounded by 1. With the help of UCB bonus, we can show that our estimated reward gap D, is an upper bound of the true reward gap D.-
Uncertainty-Aware Query Criterion To mitigate the expensive costs from collecting human feedback, we introduce the uncertainty-based criterion (Line Gb (Zhang et al. 2023) to decide whether a pair of action y and yi requires human labeling. Intuitively speaking, the UCB bonus B - -
captures the uncertainty associated with the preference fcodback o,. Similar criterion has also been used corruption-robus, lincar contextual bandits (He et al.] (022b) and nearly minimax optimal algorithms for learning linear (mixture) + represents the uncertainty of certain action. For the action pair (vt vi with low uncertainty, where the observation
Markov decision processes (Zhou & Gu 2022 He et. al. 2023 Zhao et al. 2023 where B|

---

Published in Transactions on Machine Learning Research (09/2025)
is nearly known and provides minimal information, select the action yi v/ without querying human preference leedback. In this situation, the policy a(---) remains unchanged as there is no observation in this round. By employing the sneertaintybbsse data selection rule, we will later prove that the query complexity is bounded.
Proximal Policy Optimization In cases where the action pair (y) vi) exhibits high uncertainty and the uncertainty-awar query criterion is triggered, the agent resample the action yt from policy ne and queries human feedback for the duel y/. 3/ Upon observing the preference ot, this round is then added to the dataset C. Subsequently, the policy 1x+1 is updated using the Optimistic Proximal Policy Optimization (OPPO) method (Cai et al.
2020
He et al. 2022a). i.e.,
(yla)) x (x(yl) exp (v,).,,
In an extreme case where the uncertainty threshold I is chosen to be 0, the uncertainty-aware query criterion will always be triggered. Under this situation, Algorithm [ will query the human-labeled preference for each duel (yl (.), and Algorithm will degenerate to the dueling bandit version of OPPO (Cai al.) p020 Under this situation, Algorithm enjoys ((dTT) regret while having a linear query complexity with respect to the number of rounds T.
5 Theoretical Analysis
In this section, we present our main theoretical results.
Theorem 5.1. Tet A be the minimal sub-optimal gap in Assumption 3.4 If we set the parameters I' = O(A/vd), A= B=2, 0( A/d), and 3 = ((Vd/Ka) in Algorithm then with probability at least 1 - 8, the regret for Algorithm across the first 7 rounds is upper bounded by
Regret (T) = AA)
In addition, the query complexity of Algorithm ! is upper bounded by:
Query(T) = |Crl =
O(d/AA))
Remark 5.2. Theorem 5.1 suggests that our algorithm achieves a constant level of regret and query complexity respect to the number of rounds T. In theory, our algorithm requires a prior knowledge of the sub-optimal gap A. In practice where A is unknown, the learner can set the parameter A via grid search process.
Remark 5.3. In comparison to the destedeede regret Ö(A3d/A) obtained by the AURORA algorithm (Sckhari et al. 2024 our algorithm's regret eliminates the dependency of the action space A. Morcover, achieve improvement in the query complexity by factor of A3.
6 Practical Algorithm
In this section, we introduce a practical version of our proposed algorithm based on DPO (Rafailov et al. 2024 and the resulting algorithm named as Active Direct. Preference Optimization (ADPO) and summarized in Algorithm 2 At high level, our proposed method follows the basic idea of Algorithm and sets an uncertainly threshold filter out informative training samples. However, adapting our algorithm to neural network training requires several key modifications as below.
Direct Preference Optimization We follow the framework of DPO Rafailov et al. 2024 for policy oplimization. In detail, we consider the Bradley-Terry (BT) model [Bradley & Terry 1952). which corresponds to (α) = 1/(1 + e e=*). In RLHF, the objective is to maximize the expected reward regularized by the Kullback-Leible (KI.) divergence from the reference policy Tref:
max E Byna(()))
(r(,.) (r( [KL(
(6.1)
4In our work, only focused the regret one selected action, which slightly differs from the regret in Sekhari al. 2024

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm Active Direct Preference Optimization (ADPO)
Require: Regularization parameter B, uncertainty threshold ", learning rate 77, initial model parameter 0,,
batch size
for = ..... T do
Receive batch of data B, (ar . v.)
for i = - S do
4:
Set the confidence Co, (ri, vi vt) as in 6..3)
5:
if Ce. (x,, v. vř) S then
6:
Query for the human label and set os as the queried preference. 7: else
Set o; t (r,, ro, (x,
end if
10:
end for
11: Update 0.-1 t 0 V/VoL, To,
12: end for
where 3 >> 0 is the regularization parameter, D is the distribution of the prompts and Teef is the reference policy, which corresponds to the SFT checkpoint. The optimal policy of (6.1 is follows:
(yla) Trer(yl)) exp(r(x,,)) y)).
Therefore, given the final model parameter 0, we can rewrite the reward in the following form:
re(x, v) 3 log Te(y)>) log ()))(()>>) + 3Z(x),
where 7(x) is a constant independent of y. Plugging re(x. y) into the BT model and fitting the model with the preference labels in dataset D, we get the following DPO training objective:
log (o- (ra(æ, 3²) ro(o, ro(o,)))))
LDPO Tess) = -aaay "ta.."..= o)~D owwD
where y¹ and y² are the two responses to the given prompt and o is the human preference such that o indicates a preference for y² and o = -1 indicates a proference for y². Compared to standard RLHF, DPO bypasses the reward modeling process and thus eliminates the introduced reward noise.
Confidence Estimator The key to achieving query efficiency in Algorithm D is the confidence- based data filter. However, in real applications, rewards are no longer necessarily parameterized by linear function. Thus, the uncertainty estimator cannot be directly transferred to empirical cases. Since the model is essentially predicting the probability of human preference labels, i.e.,
P 1|x, .,."" = (ro( (², - re(a, 1²))
(6.2)
where stands for the preference label and re is the reward model. We can use the reward model's predicted probability as its uncertainty. Specifically, if (ro(z, y (z, y²)l is large, then the predicted probability is close to 0 or 1, which means the model is confident about its prediction. Otherwise, if [re y²) re ræ(x, y²1 is
close to 0, the predicted probability is close to 1/2, which indicates the modol's confidence is low. Therefore, we define the following function Cg:
Co ".,."",,))) y²) = (ro - ro (2)
(6.3)
as the confidence level of the model. Such approaches is also applied in previous works like He et al.
2024
Training Objectives One key design in ADPO the use of pscudo label, which inspired by previous methods such as Gentile et al. (2022). For given answer pairs, if the model is very confident in its preference label, then the preference label predicted by the model (i.e., pseudo label) for training. To be specific, given a prompt x and the corresponding answers y² and y², the predicted preference label can be defined follows:
if CoCox.v'.u .2) u=) S
(0 ,², sign(re v²) re(xx)?))
(6.4)
if Colx, v.,))

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Results on objective benchmarks. We use bold for the highest score and underline for the second highest. ADPO significantly outperforms DPO on ARC, TruthfulQA, and performs comparably to DPO on HellaSwag, resulting higher average performances. Besides, ADPO only makes 16k queries Zephyr-Beta and 3.6k queries Zephyr-Gemma, which is about only quarter to half of the queries made by DPO. Models
ARC
TruthfulQA
HellaSwag
Average
# Queries
Zephyr-Beta-SFT
58.28
40.36
80.72
59.79
7ephyr-Beta-DPO
61.17
45.15
82.08
62.80
62k
7ephy--Beta-PP.
61.35
45.15
82.17
62.89
30k
Lephyr-BettaAAE--DO
61.38
45.46
82.99
63.28
28k
Zephy--Bela-ADDOO
62.29
52.25
83. 11
65.88
16k
Zephyy-Gemmma-SFT
55.03
46.92
81.45
61.13
0
Zeplyr-Gemma-DPP
58.45
52.07
83.48
64.67
6751
Zephyr-Gemma-ANe
59.47
52.27
83.30
65.01
3456
60.41
54.03
83.68
66.04
4219
Zephy--Gemmna-ADDO
61.01
57.55
83.16
67.24
3652
where is the human preference upon query, sign(=) is the signal of and y is the confidence threshold (corresponding to the threshold I' in APPO, which can be empirically selected by grid search on small portion of the dataset). With the predicted preference labels of given prompts and answers, now we can formulate our training objective as the follows:
CT Tret) = -x .PDD log a (0o(z,))) (0 .3.) (re(x, v²) -o(
(6.5)
To make our approach more time efficient in practice, we follow the standard approach in DPO and use mini-batch gradient descent to update the parameters of our model. At each time step, we feed the model with batch of data {(1,, v' v We then compute the pseudo-labels and update the model parameters by one-step gradient descent.
Remark 6.1. Compared to the loss of vanilla DPO, the only additional term in 6.5 is the pseudo-labeling term og. which can be directly obtained from the difference of estimated rewards "0( y ra ra v²). Since the difference of estimated rewards is also required in the forward computation of DPO, the computation of 0g results to almost no extra computational overhead, making ADPO also computationally efficient.
7 Experiments
In this section, we conducted extensive experiments to verify the effectiveness of ADPO. Our experiments reveal that ADPO outperforms DPO while requiring only up to half of the queries. Additionally, our ablation studics show that involving pscudo-labels plays a key role in the training process.
7.1 Experimental Setup
Models and Datasets
We start from two different base models zephty-77-sftt-ful11 (Zephyr-Beta-SFT) 10 (Zephy--Gemma-5 Gemma-SmmF SFT), which is wwwrrviieddffnetenee from Mistral-7B
and Sooohyr--b-gmmaaa
(Jiag et al. 2023) model and gemma-7B (Team et al.] 2024) correspondingly. Zephyr-Beta-SFTT is obtained
by conducting SFT on Ultrachat-200k (Ding et al. 2023) dataset and Kephyr-Gemma-SHe is obtained by
conducting SFT on deita-10k-v0-se (Luu et al. 2023 ). We follow the approach in aligument-handboo and adopt the corresponding human-preet preferecce datasets. In the experiments, we use pre-labeled datasets for simplicity Specifically, we use wwwpeeeeodbccc-biibbieeeee
(Ding et al. 2023) to train Zephyr-Beta-Se and
dpo-mix-77 to train Jephyr-Gemma--SPP
$https://hugginngaan wwtp://w..../aaggaaaeeeeeeee alignment-handlonnee 7
"htlps://huugiinggann www..c..o.oogggaa..e HuwggiggFaccel///////o httr////wphhy---e/eeee 7b-
hhttps://github.orr btttchuu..oo//ugggggggec.. wtngggiigface/lligmmentt hignnennt-handboo
8In real world sccuarios, the label might be queried in-tho-loop. While this process does impose for automatic
labeler (e-g., LLMs), might be slower than gathering a big dataset of labels all human labelers.
"https://huuee http///gggggccc...... httgg//nccc..c//aaaeeteeeeeo aaaaasttt//rrllllllaaa /dpo-x///pp--
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Results on subjective benchmarks including AlpacaEval 2.0 and MT-Bench. We use bold for the highest score and underline for the second highest. Here WR stands for win rate and LC stands for length controlled. ADPO achieves comparable performance with DPO starting from dephyr-Beta-SF7 and outperforms DPO starting from Zepbyr-GGmmma-SFT. The checkpoints evaluated here exactly the same as those in Table 1 so they share the same number of queries.
MT-Bench Second Turn
Alpaca Eval 2.0
Models
First Turn
Average
LC WR.
WR
Avg. Length
Leptyr-Beta-SFT
6.82
5.94
6.39
4.59
4.69
1741
Zophyr-Beta-DPO
7.55
7.27
7.41
13.57
12.67
1735
Jephyr-Beta-APL
6.93
6.63
6.80
10.08
9.58
1801
Beepyy--eta-AEE-DAe
6.78
6.50
6.64
11.89
11.34
1588
Zephyr-Beta-ADPOO
7.31
7.08
7.20
12.67
12.02
1801
Gephys-G
5.62
5.56
5.59
0.13
0.62
4296
Zeplyt--Gemma-DPOO
5.94
5.49
5.72
3.68
10.70
9064
Geplay--GemmaAAPL.
6.62
5.59
6.10
2.71
11.64
17474
Zepmyy--Gemma-A--DDD
6.31
5.59
5.96
3.54
16.22
16858
6.53
6.49
6.51
3.81
15.85
8967
Baselines and Evaluation We consider DPO as our baseline and use full-fineunne to optimize the models for both DPO and ADPO. Please refer to Appendix C for more details regarding the selection of hyperparamcters. In addition, we also consider two concurrent similar DPO-based active learning approaches, namely AE-DPO (denoted by AE-DPO) proposed by Mehta el al. (2023) and Active Preference Learning (denoted by APL) proposed by Muldrew al. (2024). as our baselines. We adopt both objective and subjective evaluation techniques to evaluate the resulting models. Specifically,
employ ARC (Clark al. 2018),
HellaSwag (Zellers et al.) 2019) and TruthfulQA (Lin et al. 2022) benchmarks for objective evaluation
Among these datasets, ARC (Clark et al. 2018) and HellaSwag (Zellers et al. 2019h focus on the language models' capability of commonsense reasoning, while TruthfulQA (Lin et al. (022) focuses on human falsehood mimic. For subjective benchmarks, we consider AlpacaEval 2.0 (AlpacaEval) and MT-Bench (Zheng et al. 2024). AlpacaEval employs AlpacaFarm (Dubois et al. 2024), which made up of general human instructions, as its set of prompts. During evaluating, the model responses and the reference response generated by GPT-4-Turbo fed into GPT-4-Turbo for preference amnotation and the win rate measures the models capability. MT-Bench is composed of 80 high-quality multi-turn open-ended questions covering variety of topics. The generated answers are also judged by GPT-4, which gives scores directly without comparison. Please refer to Appendix C for more detailed discussion of the datasets and evaluation.
52
DPD
DPO
62
ADPO
ADPO
50-
ADPO (w/a PL)
(w/o PL)
Accuracy (96)
61
(%6)
Accuracy
82.0
60
81.5
ADPO
42
81.0
ADPO (wlo PL)
40
0
10
20
30
40
50
60
10
20
30
40
50
60
10
20
30
40
50
60
Number Queries
Number Queries
Number Querles
(a) ARC Challenge
TruthfulQA
(c) HellaSwag
Figure 1: The test accuracy curve of DPO and ADPO starting from Zephyr-Beta-SPF The x-axis is the number of queries and the y-axis is the metric for corresponding dataset. Compared to DPO, ADPO enjoys faster performance improvement and higher performance upper bound.
9

---

Published in Transactions on Machine Learning Research (09/2023)
DPO ADPO
58
DPG
83.5
DPO
60
56-
ADPO
ADPO
55--
54
Accurary (86
58-
52
57-
50
82.0
56-
55
81.5
4000
4000
6000
2000
4000
6000
Number of Queries (k)
Number of Queries (k)
Number of Queries (k)
(a) ARC Challenge
(b) TruthfulQA
(c) HellaSwag
The test accuracy curve of DPO and ADPO starting from Zephyr-Gemma- Gemma-SFT. The x-axis is the number of queries and the y-axis is the metric for corresponding datasct. Compared to DPO, ADPO enjoys a faster performance improvement and a higher performance upper bound.
Figure 2:
7.2 Experimental Results
Results on Objective Benchmarks The results on subjective benchmarks are presented in Table | We see that DPO and ADPO improve the average score by a large margin starting from both ZSphyr-Beta-SFT and Zephyr-Gemma-SFT
As for Zephyr-Beta, on TruthfulQA, ADPO outperforms DPO prominently by a margin of 7.1%, and also outperforms DPO on ARC and HellaSwag by 1.08% and 1.03% respectively. Reflecting on the average score, we see that ADPO outperforms DPO by a margin of 3.08%. As for Zephyr- Gemma, ADPO outperforms DPO prominently by margin of 5.48% on TruthfulQA and 2.56% on ARC. ADPO also reaches a performance comparable to DPO on HellaSwag, Finally, reflecting the average score, we see that ADPO outperforms DPO by margin of 2.57%. Furthermore, when comparing to other active-learning- based DPO variants, we see that ADPO outperforms both APL and AE-DPO consistently under both models and datasets except HellaSwag, where the three methods performs almost the same. In terms of average performance, ADPO outperfomms the baselines by a significant margin of 2.60% for Zephyr-Beta and 1.2% for Zephyr-Gemma. In summary, results on both models shows the superiority of ADPO. Besides the performance on the benchmarks, we see that ADPO only requires 16k queries for Zephyr-Bet Beta and 3.6k for Zephyr-Gemma, Gemma, which is only about half of the size of the training dataset and comparable with other aetive-laarning-bare based DPO methods.
Results on Subjective Benchmarks The results on subjective benchmarks are presented in Table 2 For Zephyr-Beta, we that ADPO achieves comparable performance with DPO. In detail, On MT-Bench, we see that ADPO improves the average performance from 6.39 to 7.20, which is much more significant comparing to its gap with DPO of 0.21. Similarly, on AlpacaEval, ADPO also improve the LC win rate by ar margin of 8.08, which is much more significant. than its gap to DPO. As for other baselines, while we see that these mothods also significantly improve the performance over the base model, ADPO also outperform these baselines. For Zephyr-Gemmn we see that ADPO outperforms DPO by a considerable margin. In detail, On MT- we see that ADPO achieves performance of 6.51 compared 5.72 achieved by DPO. Similarly, on AlpacaEval, ADPO achieves LC win rate of 15.85, which also surpasses DPO by large margin. Comparing with other baselines, we see that ADPO outperforms APL and AE-DPO in terms of MT-Bench average score and Alpaca Eval length-controlled win rale.
Query Efficiency To further demonstrate the query efficiency for ADPO, we plot the test accuracy curves for ADPO and DPO as the numbers of queries increase selected datasets. The curves for ARC, HellaSwag and TruthfulQA starting from Zephyr-Beta-SFTT and Sephhr--Gemma -Frmmm--SFTT.. are shown in Figure D and Figure 2 respectively. For Zephry-Beta, we see that the growth of DPO's performance for ARC almost stops when query number reaches about 30k. This trend can also be observed on TruthfulQA after 20k queries and HellaSwag after about 15k queries. In contrast, the performance of ADPO enjoys a faster improvement when training with the first about 10k results and maintains at a preferable level after that. For Zephyr-Gemma, we observe a similar pattern. The growing speed of the performance of DPO either slows down significantly after making 3k to 4k queries, as shown by Figure ((a and Figure
(b) maintains at very low level
10

---

Published in Transactions on Machine Learning Research (09/2025)
(Figure 2(c) These results suggest that ADPO can effectively select the most informative data and only make queries for these preference labels.
8 Ablation Studies
In this section, consider the impact of the two important parts that are crucial in ADPO, namely pseudo-labeling and the choice of uncertainty threshold. Due to time and computational constraint, all the ablations starts from Zephyr-Bela-SPT and evaluated on objective benchmarks.
8.1 Impact of Pseudo Labels
Table The effect of pseudo-labels. ADPO performs better than ADPO (w/o PL) in terms average scores with fewer queries.
Model
ARC
TruthfulQA
HellaSwag
Average
# Queries
DPO
61.17
45.15
82.08
62.80
62k
ADPO (w/o PL)
61.18
45.28
82.25
62.90
34k
ADPO
62.29
52.25
83.11
65.88
16k
An alternative to active learning is to directly follow Algorithm 1 and simply neglect those training data with high confidence. Since neglected samples will not affect the loss and the corresponding gradient, we set the label to 0 so that they will not contribute to VeC in Eq. (6.5) during the learning process. Formally, we define the label og as follows:
{ o Ca 5
of (F -6)
if ('
We keep the remaining part of our method the same and denote this method "ADPO (w/o PL.). The performance of the trained models is shown in Table 3 We also plot the training curve in Figure | The results show that, without pscudo-labels, the performance suffers from significant downgrade in average score compared to ADPO and does not demonstrate a clear advantage against vanilla DPO. The training curves further indicate that, without pseudo labels, the training dynamics are much more similar to vanilla DPO. These results show that the pseudo-labels plays a crucial role our designed active learning process.
Table 4: The quantitative study of the error rate of the pseudo labels. For each steps, we report the number of all the predieted labels before this step, and the total number of all the correctly predicted labels before this steps. The results show that, the training proceeds, the correct rate of pseudo-labels remains at a relatively low level below 25%, which demonstrate reliability of the pseudo-labels.
Training Steps
50
100
200
300
450
600
750
950
# Total Predicted Labels
24
952
4904
9840
17448
25432
33232
43904
# Correctly Predicted Labels
16
808
3936
7792
13432
19392
25384
33488
Correct Rate (%)
66.67
84.87
80.26
79.19
76.98
76.25
76.38
76.28
We further count the correctly and incorrectly predicted labels as the training proceeds to quantitatively study the quality of the pseudo-labels. The results are compiled in Table The results show that, during the training procedure, about three quarters of all the predicted preference labels align with the ground true. This result shows that when the model has a high confidence, its predictions are generally reliable. Such pseudo labels helps the training process makes use of these samples and therefore enhances data-efficiency.
8.2 Value of Confidence Threshold
We study the impact of different confidence thresholds by varying the value of 7 from 0.2 to 4.0. For each 7, we count the preference labels used by the models and evaluate the trained models on the objective benchmarks. We plot the results in Figure and also provide some numerical results in Table 5 As shown in Figure 3 when y is small, even about 6k samples are sufficient for ADPO match the performance of DPO. When y is large, ADPO queries for more labels and the performance become close to DPO. Another
11

---

Published in Transactions on Machine Learning Research (09/2023)
ABC
HellSww
5TT
... B73
830
SIT
6.3
625
66.3
66.0-
58.
163
e20.31
63.0 8./60
10/2a)
20 (M))
arle
12/2b
13/00
20107
a20.5
20109)
83
as
23
Ormanee Thna Number Buenm Ic0
Candence Threales Numbar of Cuerta
Grntance Mamber Quenks
(a) ARC Challenge
(b) TruthfulQA
(c) HellaSwag
Figure 3: The performances on three objective benchmarks and the average performances as 7 varies from 0.2 to 4.0. The results shows that ADPO performs comparable with DPO over a wide range of y and can matches the performance of DPO even with only fewer than 6k samples. These results demonstrate the robustness of ADPO over 7
Table 5: The effect of confidence threshold in ADPO. We vary the value of y and report the evaluation results. When y is increasing, ADPO made more queries and the performance pattern is getting closer to DPO.
Method
y
ARC
TruthfulQA
HellaSwag
Average
# Queries
DPO
61.17
45.15
82.08
62.80
62k
1.0
61.43
52.76
82.55
65.58
13k
ADPO
1.3
62.29
52.25
83.11
65.88
16k
1.5
60.75
52.97
83.29
65.67
21k
observation ADPO matches the performance of the DPO baseline over wide range of which implies that ADPO not very sensitive to the uncertainty threshold and an coarse grid search of confidence threshold can introduce a fairly good performance.
9 Conclusion and Future Work
In this work, we considered query-efficient methods for aligning LLMs with human preference. We first formulated the problem as a contextual ducling bandit. Under linear reward and sub-optimal gap assumption, we proposed an active-learning based algorithm, APPO. Our theoretical analysis shows that our algorithm enjoys a constant instance-dependent regret upper bound and query complexity. We then adapted our algorithm to direct preference optimization and proposed query efficient DPO method, ADPO. We conducted experiments starting from two models, Zephyr-Beta-SFTT and Zephyr-Gemma-SPYy and evaluated the resulting models on both objective benchmarks and subjective benchmarks. Results show that, ADPO achieves a comparable or even better performance compared to DPO with only less than half the demands on the human preference labels. Despite the good performance ADPO achieves, since it uses DPO as the framework of our practical method, our theoretical analysis of APPO cannot be directly applied to ADPO. We leave the thcoretical analysis of ADPO as our future work.
Broader Impact Statement
This paper studics aligning LLMs with human preference in query-efficient manner. We believe that this topic has the following social impacts. First, LLM-based chatbots have demonstrated substantial capabilities as AT assistants and they are now increasingly relied upon by individuals. The key step towards building helpful Al assistant is aligning LLMs with human ethics and preferences. Secondly, aligning LLMs requires a large number of human preference labels, necessitating considcrable human labor and material resources. this paper, We propose a query-efficient method to align LLMs with human preference. Our experiments results indicate that method can better align LLMs with human preference with significantly fewer queries for human preferences. Therefore, we believe that our method potentially alleviate the labor and resource demands within this process.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Acknowledgments
We thank the anonymous reviewers and the action editor for their helpful comments. We also thank Qiwei Di for feedback at the early stage of this work. KJ, JH and QG are supported in part by the NSF grants and IIS-2403400. JH is also supported by UCLA Dissertation Year Fellowship. The views and conclusions contained this paper are those of the authors and should not be interpreted as representing any funding agencies.
IIS-1906169,
References
Yasin Abbasi- Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint MKKv:2333.0877,, 2023.
Riad Akrour, Marc Schoenauer, and Michèle Scbag. April: Active preference lcarning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp. 116-131. Springer, 2012.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International conference on machine learnóng, pp. 263-272. PMLR, 2017.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. aiv preprint arXiv:2204.0m 2556022 2022.
Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. In Proceedings of the 23rd international conference on Machine learning, pp. 65-72, 2006.
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference on Computational Learning Theory, pp. 35-50. Springer, 2007.
Akshay Balsubramani, Zohar Karnin, Robort E Schapire, and Masrour Zoghi. Instance-dependent regret bounds for dueling bandits. In Conference on Learning Theory, pp. 336-360. PMLR, 2016.
Viktor Bengs, Aadirupa Saha, and Eyke Hüllermeier. Stochastic contextual ducling bandits under lincar stochastic transitivity models. In International Conference on Machine Learning, pp. 1764-1786. PMLR, 2022.
Erdem Biyik and Malayandi Palan. Asking easy questions: A user-friendly approach to active reward learning. In Proceedings of the 3rd Conference on Robot Learning, 2019.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)) 1952.
Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration policy optimization. In International Conference on Machine Learning, pp. 1283 1294. PMLR, 2020.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Schourer, Javier Rando, Rachel Preedman, Tomek Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023.
Nicolo Disa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Nicolo Cesa-Bianchi, Gábor Lugosi, and Gilles Stoltz. Minimizing regret with label efficient prediction. IEEE Trunsactions on Information Theory, (1):22155 2162, 2005.
Nicolo Disa-Bianchi, Claudio Gentile, Luca Zaniboni, and Manfred Warmuth. Corst-case analysis of selective sampling for linear classification. Journal of Machine Learning Research, 7(7), 2006.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Nicolo Claudio Gentile, and Francesco Orabona. Robust bounds for classification via selective sampling. In Proceedings of the 26th annual international conference on machine learning, pp. 121-128, 2009.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://Amsys. org.byss..g//logog
3023-03-30-vicume
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34: 11933 11944, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXie preprint arXiv: 1att:883..00 807.05457, 2018.
Sanjoy Dasgupta. Coarse sample complexity bounds for active learning. Advances in neural information processing systems, 18, 2005.
Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. Analysis of perceptron-based active learning. In International conference on computational learning theory, pp. 249 263. Springer, 2005.
Qiwei Di, Tao Jin, Yue Wu, Heyang Zhao, Farzad Farnoud, and Quanquan Gu. Variance-aware regret bounds for stochastic contextual dueling bandits. In The Twelfth International Conference on Learning Representations, 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arxiv preprint Xie:22305 14233, 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Miroslav Dudik, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual ducling bandits. In Conference Learning Theory, pp. 563-587, PMLR, 2015.
Moein Falahatgar, Alon Orlitsky, Venkatadheerai Pichapati, and Ananda Theertha Suresh. Maximum selection and ranking under noisy comparisons. In International Conference on Machine Leorning, pp. 1088-1096. PMIRR 2017.
Moein Falahatgar, Ayush Jain, Alon Orlitsky, Venkatadheeraj Pichapati, and Vaishakh Ravindrakumar. The limits of maxing, ranking, and preference learning. In International conference on machine learning, pp- 1427 1436. PMLR, 2018
Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. Advances in Newral Information Processing Systems, 23, 2010.
Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preferonce-based reinforecment learning: a formal framework and a policy iteration algorithm. Machine learning, 89:123 156, 2012.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. Exploring the feasibility of chatgpt for event extraction. arXiv preprial arXic:2303.088 03836, 2023a.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835 10866. PMLR, 2023b.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Clandio Gentile, Zhilei Wang, and Tong Zhang. Achieving minimax rates in pool-based batch active learning. In International Conference on Machine Learning, pp. 7339- 7339-7367. PMLR, 2022.
Quanquan Gu, Tong Zhang, Jiawei Han, and Chris Ding. Selective labeling via error bound minimization. Advances neural information processing systems, 25, 2012.
Quanquan Gu, Tong Zhang, and Jiawei Han. Batch-mode active learning via error bound minimization. In UAI, pp. 300-309, 2014.
Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. orXiv preprint arXiv:2305. 14450, 2023.
Steve Hanneke and Liu Yang. Minimax analysis of active learning. J. Mach. Learn. Res., 16(1):3487-3602. 2015.
Steve Hanneke and Liu Yang. Toward general theory of online selective sampling: Trading off mistakes and queries. In International Conference on Artificial Intelligence and Statistics, pp. 3997-4005. PMLR, 2021.
Jiafan He, Dongruo Zhou, and Quanquan Gu. Logarithmic regret for reinforcement learning with linear function approximation. In International Conference on Machine Learning, pp. 4171-4180. PMLR, 2021.
Jiafan He, Dongruo Zhou, and Quanquan Gu. Near-optimal policy optimization algorithms for learning adversarial linear mixture mdps. In International Conference Artificial Intelligence and Statistics, pp. 4259-4280. PMLR, 2022a.
Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. Advances in Neural Information Processing Systems, 45:34614-34625 2022b.
Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear markov decision processes. In International Conference on Machine Learning, pp. 12790 12822. PMLR, 2023.
Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao. Semi-supervised reward modeling via iterative self- training. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7365-7377, 2024.
Reinhard Heckel, Max Sinchowitz, Kannan Ramchandran, and Martin Wainwright. Approximate ranking from pairwisc comparisons. In International Conference on Artificial Intelligence and Statistics, pp. 1057 1066. PMLR, 2018.
Steven CH Hoi, Rong Jin, Jianke Zhu, and Michael R Lyu. Batch mode active learning and its application to medical image classification. Proceedings of the 23rd international conference on Machine learning, pp- 417-48 424, 2006.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Kevin Jamieson, Sumeet Katariya, Atul Deshpande, and Robert Nowak. Sparse dueling bandits. In Arlificial Intelligence and Statistics, pp. 416-424. PMLR, 2015.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arxiv:2310. 06825, 2023.
Chi Jin, Zeyuan Allen-Zhu, Sebastion Bubeck, and Michael I Jordan. Is qlearning provably efficient?
Advances in neural information processing systems, 31, 2018.
15

---

Published in Transactions on Machine Learning Research (09/2025)
David Krueger, Jan Leike, Owain Evans, and John Salvatier. Active reinforcement learning: Observing rewards at cost. arxiv preprint or urXi:2011.06700, 2020.
K Lee, L Smith, A Dragan, and P Abbeel. B-pref: Benchmarking preference based reinforcement learning. Neural Information Processing Systems (NeurIPS), 2021a.
Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152 6163. PMLR, 2021b.
Lihong Li, Yu Lu, and Dengyong Zhou, Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning, pp.
PMLR, 2017.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference- based reinforcement learning. In 10th International Conference on Learning Representations, ICLR 2022.
International Conference on Learning Representations, 2022.
Stephanic Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Mcasuring how models mimic human falschoods. In Proceedings of the 60th Annual Meeting of the Associalion for Compulational Linguisties (Volume 1: Long Papers), pp. 3214 3252, 2022.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Represenlations, 2023.
Hao Lou, Tao Jin, Yue Wu, Pan Xu, Quanquan Gu, and Farzad Farnoud. Active ranking without strong stochastic transitivity. Advances in newral information processing systems, 35:297-3000 309, 2022.
Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via. active exploration. arXiv preprint arXiv:2312.0026% 2023.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Proceedings of the 4 1s International Conference on Machine Learning, pp. 2024.
Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhao- han Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. Nash learning from human feedback. In International Conference on Machine Learning, pp. 08743-36766 PMLR, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishknn, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730 27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Newral Information Processing Systems, 36, 2024.
Siddartha Y Ramamohan, Arun Rajkumar, and Shivani Agarwal. Dueling bandits: Beyond condorcet winners to general tournament solutions. Advances in Neural Information Processing Systems, 29, 2016.
Wenbo Ren, Jia Kevin Liu, and Ness Shroff. On sample complexity upper and lower bounds for exact ranking from noisy comparisons. Advances in Neural Information Processing Systems, 32, 2019.
Aadirupa Saha. Optimal algorithms for stochastic contextual preference bandits. Advances in Neural Information Processing Systems, 34:30050 -00002,00222 2021.
Aadirupa Saha and Pierre Gaillard. Versatile ducling bandits: Best-of- both world analyses for learning from relative preferences. In International Conference Machine Learning, pp. 1901 19011-19026, PMLR, 2022.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Aadirupa Saha and Akshay Krishnamurthy. Efficient and optimal algorithms for contextual dueling bandits under realizability. In International Conference on Algorithmic Learning Theory, pp. PML.R, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177::7077.034477 2017.
Sebastian Schulze and Owain Evans. Active reinforcement learning with monte-carlo tree search. arXiv preprint Xix: 049264 2018.
Ayush Sekhari, Karthik Sridharan, Wen Sun, and Runzhe Wu. Contextual bandits and imitation learning with preference-based active queries. Advances in Neural Information Processing Systems, 36, 2024.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Max Simchowitz and Kevin G Jamieson. Non-asymptotic gap-dependent regret bounds for tabular mdps. In Advances in Neural Information Processing Systems, pp.
2019.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya. Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint KKK::2403.08299 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arxiv preprint orXiv:2307.09255 099888 2023.
Aaron Tucker, Caleb Biddulph, Claire Wang, and Thorsten Joachims. Bandits with costly reward observations. In Uncertainty in Artificial Intelligence, pp. 2147-2156. PMIRR 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Youes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Yuanhao Wang, Qinghua Liu, and Chi Jin. is rlhf more difficult than standard rI? theoretical perspective. Advances in Neural Information Processing Systems, 36:76006-76032, 2023.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang. Xin Zhang, Shen Huang, Pengjun Xic, Jinan Xu, Yufeng Chen, Meishnn Zhang, al. Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10000 2023.
Nils Wilde, Dana Kulić, and Stephen L Smith. Active preference learning using maximum regret. In 2020 [EEE/RSJ International Conference on Intelligent Robols and Systems (IROS), pp. 10952 10952-10959. 10959. TEEE, 2020.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(16)): 2017.
Huasen Wu and Xin Liu. Double thompson sampling for dueling bandits. Advances in neural information processing systems, 29, 2016.
Runzhe Wu and Wen Sun. Making with preference-based feedback efficient via randomization. arXiv preprint Xrx::22310. 14554, 2023.
Yue Wu, Tao Jin, Hao Lou, Pan Xu, Farzad Farnoud, and Quanquan Gu. Adaptive sampling for heterogeneous rank aggregation from noisy pairwise comparisons. In International Conference on Artificial Intelligence and Statistics, pp. 11014 11036. PMLR, 2022.
Yue Wu, Tao Jin, Qiwei Di, Hao Lou, Farzad Farnoud, and Quanquan Gu. Borda regret minimization for generalized linear dueling bandits. In International Conference on Machine Learning, pp. 53571-53596. PMI.R, 2024.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human leedback: A provable kl-constrained framework for rlhf. arxie preprint arXiv:231.. 11456, 2023.
Kunhe Yang, Lin Yang, and Simon Du. Qllearning with logarithmic regret. In International Conference on Artificial Intelligence and Statistics, pp. 1576 1584. PMLR, 2021.
Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. Zero-shot temporal relation extraction with chatgpt. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1588-1556 2012.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hcllaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason D Lee. How to query human feedback efficiently in rI? 2023.
Tong Zhang and F Oles. The value of unlabeled data for classification problems. In Proceedings of the Seventeenth International Conference on Machine Learning, (Langley, P., ed.), volume 20, pp. 0. Citeseer, 2000.
Weitong Zhang, Jiafan He, Zhiyuan Fan, and Quanquan Gu. On the interplay between misspecification and sub-optimalitt gap linear contextual bandits. In International Conference on Machine Learning, pp- 41111-41132. PMLR, 2023.
Heyang Zhao, Jiafan Hc, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. In The Thirty Sixth Annual Conference on Learning Theory, pp. 1977-5020. PMLR, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-asaa-judge with mt-bench and chatbot arena. Advances in Newral Information Processing Systems, 36, 2024.
Dongruo Zhou and Quanquan Gu. Computationally efficient horizon-free reinforcement learning for lincar mixture mdps. Advances in newral informalion processing systems, 35:36337 36349, 2022.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. Tn International Conference on Machine Learning, pp. 13037-43067. PMLR, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Finc-tuning language models from human preferences. arXiv preprint arxiv: 100.19099089, 2019.
A Additional Related Work
Dueling Bandits Dueling bandits represent variant of the multi-armed bandit problem, incorporating
preference foodback between two selected arms (Yue et al. 2012). Existing results in this domain generally
fall into two cat categories, shaped by their assumptions about preference probability. The first category of work
Yue et al.) 2012 Falahatgar et al.) 2017 2018 Ren et al.) 2019 Wu et al.] 2022 Lou et al.) 2022) assumes a
transitivity property for preference probability and focuses on identifying the optimal action. Our work also
belongs this category. The second category of work (Jamieson et al.) 2015 Heckel al.
2018 Saha 2021
Wu et al.] 2024 Dudík et al.) 2015 Ramanohan al.) 2016 Balsubramani el al.) 2016) locuses on general preferences with various criteria for optimal actions, such as Borda winner and Copeland winner.
Expanding beyond the standard dueling bandit problem, Dudík et al. (2015) was the first to incorporate contextual information into the dueling bandit framework. Subsequently, Saha (2021) studied the K-arm
18

---

Published in Transactions on Machine Learning Research (09/2025)
contextual dueling bandit problem and proposed algorithm with a near-optimal regret guarantee. In order to addressing the challenge of potentially large action space, Bengs et al. (2022) also considered linear function approximation and extended these results to the contextual linear dueling bandit problem and obtained a regret guarantee of (dd/T). Recently, Di et al. (0244) introduced a layered algorithm, improving the results to a variance-aware guarantee of Eot) where of denotes the variance of the observed preference in round t.
B Further Discussions on Related Work
In this section, provide further discussion on previous works (Wang et al. 2023 Zhan et al. 2023 Wu &
2023 Sekhari et al. 2024), which consider query complexity in the dueling bandit setting, and explain why they fail to achieve an instance-dependent regret guarantee.
Sun
Comparison with Wang et al. (2023) Wang et al. (2023) proposed a general framework, P2R, for efficiently querying human preferences, and later extended it to a white-box algorithm (P-OMLE) with a specialized analysis. However, the P2R algorithm relies
comparison oracle that is stronger than ours. In the bandit setting, the oracle in Wang et al. (2023) can return preference labels between responses to different prompts, which often exceeds the abilities of typical users. In contrast, our oracle only requires preferences between responses generated from the same prompt. Furthermore, P2R algorithms necessitate multiple independent comparisons between a baseline trajectory and user-gertt aver-generated trajectories, making it impractical to ask a single user for multiple independent preferences on the same qucry. Our ADPO algorithm, by contrast, only requires one preference feedback per query, making it much more user-friendly. Additionally, in the linear reward setting, the query complexity for the Preference- based OMLE (white-box) algorithm is D(df//2// which is the same as ours. However, P-OMLE requires solving an optimization problem over complex confidence regions, resulting in an intractable planning phase. In comparison, our APPO algorithm introduces an explicit confidence bonus to bypass this complexity and uses a policy optimization method, which more tractable and closely aligned with practical RLHF methods, while still achieving the same query complexity.
Failures in Achieving Instance-Dependent Regret Guarantees Recently, Zhan et al. 2023) proposed a pure-exploration style algorithm (REGIME) that can identify the e-optimal policy with a query complexity of D(dL/e") where d is the dimension of the feature space. However, it is important to note that the output policy may be a randomized policy, and does not guarantee a constant regret, even under the assumption of a minimal sub-optimality gap A. Specifically, the minimal sub-optimality gap does not prevent a randomized policy from incurring regret between O and A. Thus, even for an e much smaller than the sub-optimality gap A, there no guarantee that randomized algorithm will always achieve zero regret. For example, a policy that selects the optimal action with a probability of 50% and a A-suboptimal action with a probability of 50% will result. in regret of A/2. In this situation, the REGIME algorithm may lead to linear regret. with respect to T and fail to achieve instance-dependent regret guarantees.
A similar issue arises when transferring the sample complexity guarantee in Wang et al. (2023) an instance- dependent regret bound. Additionally, we observe that in Proposition 5 of Wang al. (2023), , the P2R, framework achieves finite sample complexity using the UCBVI algorithm (Azar et al. 2017 It is important to note that the original UCBVI algorithm employs deterministic policies for each episode. However, Azar lt al. (211)) only provides vT regret guarantee for the first T rounds, rather than a sample complexity guarantec. To address this, Jin et al. (2018) demonstrates that any algorithm with sublincar regret can derive a finite sample complexity by randomly selecting a policy from the first T rounds, resulting in a final randomized policy. After this step, the output policy may no longer be deterministic and fails to provide instance-dependent regret guarantees.
Another related work, Wu & Sun (2023), proposed a sampling-based algorithm (PR-LSVI) that provides a Õ(d' regret guarantee for the first I' rounds, which is not directly related to the sub-optimality gap. Consequently, a random mixture over the first T rounds is required to identify a near-optimal policy, and it fails to achieve constant regret even the presence of a positive sub-optimality gap.
10 query complexity of P-OMLE has logarithmic dependency on the reward function space For the linear reward function class, where log d, their complexity becomes doog R/ A d2/A²
19

---

Published in Transactions on Machine Learning Research (09/2025)
As demonstrated above, all of these works can only find random policy that achieves e-optimality, which cannot provide an instance-dependent regret guarantee, even with the assumption of minimal sub-optimality gap.
C Additional Experiment Details
paper-parameters for Training Zephyr-Beta We trained our models on 4XNVIDIA A100 GPUs, with about 80G memory for each GPU. We set the learning rate to 5e-7 for both DPO and ADPO. We use a linear learning rate scheduler with a warm-up ratio of 0.1. The batch size per device is set to 4 and the gradients are accumulated every 4 steps, resulting in equivalent batch size 64. We set dropout to 0.1 and the regularization parameter 3 = 0.1 for both DPO and ADPO. For both ADPO and its counterparl without. pseudo labels, we set the uncertainty threshold y 1.3. We trained one epoch for both DPO and ADPO, which takes roughly 9 hours for both methods.
prrer-parameters for Training Zephyr-Gemma We trained our models on 4XNVIDIA A100 GPUs, with about 80G memory for each GPU. We set the learning rate to 5e-7 for both DPO and ADPO. We use linear learning rate scheduler with a warm-up ratio of 0.1. The batch size per device is set to 4 and the gradients are accumulated every 4 steps, resulting in equivalent batch size 64. We set dropout to 0.1 and the regularization parameter 3 0.1 for both DPO and ADPO. For ADPO, set the uncertainty threshold y == 1.5. We trained one epoch for both DPO and ADPO, which takes roughly hour for both methods.
Evaluation Setup
For subjective evaluation benchmarks, we follow the standard setup specified in the few-show learning to prompt the LLMs. Specifically,
original repositories. For objective benchmarks, the few-shot number of ARC is set to 25, HellaSwag to 10 and TruthfulQA to O. We use acc_norm as the metric for ARC and HellaSwag, and mc2 for TruthfulQA.
D Additional Experiment Results
In this section, present the additional results which are obtained by starting from Zephyr-Beta--ee and optimizing the model with LoRA-finetuing (Hu et al. 2021
Experiment Setup We trained our models on 4xNVIDIA RTX A6000 GPUs, with about 49G memory for each GPU. We set the LoRA rank to 64, a 16, and dropout to 0.1 and learning rate to 1e-5. We use linear learning rate scheduler with warm-up ratio of 0.1. The batch size per device is set to 4 and the gradients are accumulated every 4 steps, resulting in equivalent batch size 64. We set the regularization parameter 8 = 0.1 for both DPO and ADPO. For ADPO, we set the uncertainty threshold y 1.5. We trained one epoch for both DPO and ADPO, which takes roughly hours. We only evaluate the obtained checkpoints on objective benchmarks due to the costly nature of calling external large language models as the judge.
Table 6: Result on objective benchmarks for LoRA finetuning on Zephyr-7B-Beta. ADPO significantly outperforms DPO on ARC, TruthfulQA and HellaSwag. Besides, ADPO only makes 32k queries, which is about only a quarter to half of the queries made by DPO.
Models
ARC
TruthfulQA
HellaSwag
Average
# Queries
7ephyr-Beta-SFT
58.28
40.36
80.72
59.79
Zophyr-Beta-DPO (LoRA)
60.58
41.88
82.34
61.60
62k
BepD (LoRA)
61.26
45.52
83.21
63.33
32k
Benchmark Performances fall-finetunig. The results show that both DPO and ADPO improve the average score by large margin. ADPO outperforms DPO on TruthfulQA by relatively large margin of 3.64% and also reaches an at-loast comparable performance on other three datasets. Finally, reflecting on the average score, we see that ADPO outperforms DPO by a margin of 1.73%. Besides, we see that ADPO only requires 32k queries, which is only about half of the size of the training datasct. These results show that with much less number of queries, ADPO can reach a comparable or even superior performance than DPO, which is consistent with results under full-finetune.
The results are summarized in Table G We observe similar results for
20

---

Published in Transactions on Machine Learning Research (09/2023)
61.0
45-
Accuracy 186
Accurace suuracy (16)
60.0
(96)
DPO (LoRA)
43-
ADPO (LORA)
82.0
59.5
ADPO (LoRA)
42-
81.5-
59.0
DAO (LoRA)
DPO (LoRA)
58.5
ADPO (LoRA)
ADPD (LoRA) (W/O PL)
81.0-
ADPO (LoRA) (w/o PL)
ADPO (LoRA)
0
10
20
30
40
50
60
0
10
20
30
40
50
60
0
10
20
30
40
50
60
Number of Queries
Number of Queries
Number of Queries
(a) ARC Challenge
(b) TruthfulQA
(c) HellaSwag
Figure 4: The test accuracy curve of DPO, ADPO (w/o PL) and ADPO under LoRA-finetuee. The x-axis is the number of queries and the y-axis is the metric for corresponding dataset. Compared to DPO and ADPO (w/o PL), ADPO enjoys a faster performance improvement and higher performance upper bound.
Query Efficiency We plot set of similar test accuracy curves for ADPO and DPO for LoRA-finetunnn. The results are presented in Figure 4 Here we observe a similar pattern. The growing speed of the performance of DPO either slows down significantly after making 15k to 30k queries, as shown by Figure ((a) and Figure
[(b)] or maintains at a very low level (Figure [[c) These results suggest that ADPO can effectively select the most informative data and only make queries for these preference labels.
D.1 Ablation Studies
Table 7: The effect of pseudo-labels under LoRA-finetune setting. ADPO performs better than ADPO (w/o PL) in terms of average scores with fewer queries.
Model
7
ARC
TruthfulQA
HellaSwag
Average
Queries
ADPO (LoRA) (w/o PL)
0.8
60.49
41.39
82.23
61.37
38k
ADPO (LoRA) (w/o PL)
1.0
60.49
41.62
82.32
61.48
40k
ADPO (LoRA) (w/o PL)
1.2
60.49
41.79
82.47
61.58
43k
ADPO (LoRA)
1.5
61.26
45.52
83.21
63.33
32k
Impact of Pseudo Labels Following the setup in Section 8.1 we consider the counterpart of ADPO without pseudo labels under LoRA finetuning. We pick 7 to be 0.8, 1.0, and 1.2 for ADPO without pseudo labels. The performances of the trained models are shown in Table 7 We also plot the training curve in Figure 4 The results show that, without pseudo-labels, the performance suffers from a significant downgrade in average score compared to ADPO. The training curves further indicate that, without pseudo labels, the training dynamics are much more similar to vanilla. DPO. These results show the crucial role of pscudo-labels in the aclive learning process.
Table 8: The effect of confidence threshold in practice setting. We vary the value of 7 and report the evaluation results. When y is increasing, ADPO made more queries and the performance pattern is getting closer to DPO.
Method
ARC
TruthfulQA
HellaSwag
Average
# Queries
DPO (LoRA)
60.58
41.88
82.34
61.60
35.7k
1.0
61.01
48.41
83.35
64.26
16k
1.3
61.43
47.56
83.48
64.16
24k
ADPO (LoRA)
1.5
61.26
45.52
83.21
63.33
32k
1.8
60.92
43.20
82.13
62.08
43k
Value of Confidence Threshold We also study the impact of different confidence thresholds in LoRA fine-tuning setting. We varies the value of y to 1.0, 1.3, 1.5, and 1.8. For each 7, we count the preference labels used by the models and evaluate the trained models on the datascts. As shown in Table g when the confidence threshold is small, with more predicted labels, these models perform better on the TruthfulQA dataset. On the other hand, when the confidence threshold goes larger, the models making queries,
21

---

Published in Transactions on Machine Learning Research (09/2025)
and the performance patterns become closer to the DPO baseline. Another observation is that for all our chosen y, ADPO consistently outperforms the DPO baseline, which implies that ADPO is not very sensitive to the uncertainty threshold and an coarse grid search of confidence threshold can introduce fairly good performance.
E Proof of Theorems in Section 5
In this section, we provide the proof of Theorem 5.1 and we first. introduce several lemmas. The following lemma provides an upper bound on the query complexity and the corresponding dataset size |Cr).
Lemma E.1 (Modified from Lemma 4.5, Zhang et al. 2023). Given uncertainty threshold 0 < I' S 1, if we set the regularization parameter = B-2, then for each round / E [T], we have |C, S |Crl < 16d log(3LBr
For a finite dataset Cr, the following lemma provides a upper bound for the estimation error between O and 0*
Lemma E.2. Suppose we have ||0||| < B, [[((,,))))))))))))))))))())))[ s L/2. Then with probability least - 8, for each
round (T), we have
100 0
S 1 Ki
. + 22d (C2122/200))
Based on Lemmas E.1 and E.2 the following auxiliary lemma proposes proper choice for the uncertainty threshold I' and confidence radius 3 in Algorithm [
Lemma E.3. If we set the uncertainty threshold r = ne (/2d1) and confidence radius B = ki (1 + 4vdi2 + /2dc3), where 4 (226. (,))) + / (8log(1/5) l2 log log(3LBr-1) and i3 log ((1 + 161.2 (2 then we have 2 < A and
B (VXB+ 2d (T) (Cr2/((+(C
Kip
With these parameters, we now define the event El as
& = (vt [T), 00 0011,,1 S $..
According to Lemma E.2 and Lemma E.3 have Pr(&)) 1 - 5. Conditioned on the event &,, the following lemma suggests that our estimated discrepancy is no less than the actual discrepancy.
Lemma E.4. On the event Eι, for each round t [T], context X and any action y A, the estimated discrepancy De y) satisfied
D,(x, v) 2 D,(2, v) = (0", p(r,)) - -7))
On the other hand, we have
Dr(x, S Dz(x, v) + 28 v) - $ |
It is worth to notice that in Algorithm (Line we update the policy म. with online mirror descent and the following lemma provides the regret guarantee for this process.
Lemma E.5 (Modified from Lemma 6.2, He et al. 2022a For any estimated value function D. :), we update the policy (+11() by the exponential rule:
T++((12)) x Tx(1lx) exp (nDa(<r,)) :),
(E.1)
then the expected sub-optimality gap at round T can be upper bounded as follows:
Ey-D,y- (1 (1)) y)] EaND (D, y)]
< 21 + 17 EEAD (+1x)(()))) - (-<0())))]===<(((()))
22

---

Published in Transactions on Machine Learning Research (09/2025)
With the help of these lemmas, are now ready to prove main theorem.
Proof of Theorem 5.1 Now we start the regret analysis. For simplicity, for each round t E [T], we use Φi to denote Φ ye). Initially, the episodes and their corresponding regret can be decomposed into two groups based on whether episode is added the dataset Cr:
Regret(T)
0 .6) -00 (1))
T
Σ Dr(ris vi) - De(urs wi))
Σ
Da vi) D. v/)
E
De(ae, Di(
(E.2)
LCC
e4Cr
where Dz(z, y) (0", ((x,1) y) oi) denotes the reward gap between action E and selected action ył at round 6.
Now, we bound this two term separately. For the term 41 we have
1
Σ Di( v()
IECT
D.(xr,o)) (i E
Di(zn De(
tec
J1
Σ De w)l gy w)1
J2
Σ (i(z., - D. IECr
Σ
E (1) 8)] (t vel. (E.3)
IECr
where the inequality holds due to Lemma For the term J1, we have
E.4
J1 = Σ
De( wt) Dr (xe:
tEC
Σ min(4,288144 - &6
<
1CC
$ 48
|Crl- Σ min(1, 11
Φ7 E
LECT
Ad + [Cr
- 8//
[Cr|dlog
(E.4)
Ad
where the first incquality holds due Lemma E.4 with the fact that -2 S D2 v!) < 2, the second inequality holds due to Cauchy-Schwan inequality and the last inequality holds due to the elliptical potential lemma (Lemma G.7
23

---

Published in Transactions on Machine Learning Research (09/2025)
The term J2 reflects the sub-optimality from the online mirror descent process and can be upper bounded by Lemma E.5 For simplicity, we denote Cr {f1,... (x} where K = -Crl. Thus, we have
J2
Bun D.y (11x) DD (Fee U)1 Bn,, D.ymm (1x) (ex (F U)]
K
Σ
(2n) +
E RA (T (110) KT ((>))
k=1
2n + 1) FAND (T - (1)])
< 21/ +1 F KT (T (+1=))
s 2v 32d1 log B |A|,
(E.5)
where the first inequality holds due to Lemma E.5 the second equation holds due to policy keeps unchanged for & E Cr, the second inequality holds due to KT-(-II-) 2 0 and the last inequality holds due to 77 v Log log A/33d According to Seumm--Hoeffding inequality (Lemma G.6 with probability at least 1-8, 8, the term Jg can be upper bounded by
with the fact that is uniform policy,
J3 2 2 2 log(1/8).
(E.6)
Substituting F.4
E.5
and
E.6 into
we have
Ad + |r
1 J1 J2 + Ja s 8
Crldlog
+ 2 32d logg==l)3(LLBB))]]1 |A|
2|C log(1/0)
Ad
s <((Bd/T)
=ō d2
(E.7)
where the last inequality holds due Lemma F.1 Now, we only need to focus on the term 12. For each round $ Cr, we have
De(re, vi) - De(re, vi) = (0- 0. Φं - (3) + (0., o; - 0,) (0" of +33)
s $ ol ب + (0,, of Φ)) (0", 0 07)
< B|Φ of 2 (0,, Φf Φ)) (0" Φb +3)
< 28]1o]
|
where the first incquality holds due Lemma E.4 the second inequality holds due the selection rule of
action &f and the last inequality holds due to Lemma E.4 According to the definition of set Cr in Algorithm for each round t Cr, we have | 31I|, < r. Therefore, the sub-optimality gap at round t is upper bounded by
2
Φe &e11 < 23T < A,
where the second incquality holds due to Lemma (Assumption 3.4h, this indicates that the regret yielded in round & Cr is Summing up over 1. & Cr, we have
E.3
According to the minimal sub-optimality assumption
I = Σ D.(x..))) Di vt) = 0.
(E.8)
ET
Combining the results in F.7 and E.8
complete the proof of Theorem 5.1
24

---

Published in Transactions on Machine Learning Research (09/2025)
F Proof of Lemmas in Appendix E
In this section, we provide the proofs of the lemmas in Appendix
F.1 Proof of Lemma
E.1
The proof follows the proof Zhang et al. 20233) Here we fix the round t. to be T the proof and only provide the upper bound of Cr due the fact that C2 is monotonically incrcasing w.r.t. the round For all selected episode / Cr, since we have | oi -
Proof of Lemma E. 7
r, the summation of the bonuses
over all the selected episode & E Cr is lower bounded by
E min { L, 1, &il }
[Cr] min ( 12 |CT|122
(F.1)
ICCr
where the last equation holds due to 0 s I S 1. On the other hand, according to Lemma G.3 the summation is upper bounded by:
Ad -d+1(r1r2)
Σ min {I, 10: - 23 33 S 2d
(F.2)
Ad
tECr
Combining F.1 and
F.2
we know that the total number of the selected data points |Cr| satisfies the
following inequality:
Ad + |Cr|Zz
racr1 s 2d
Ad
For simplicity, we reorganized the result as follows:
r Crl
2L [2]
S log
11
(F.3)
2d
r 2d
Notice that x = B-2 and 21.²
22 1 therefore, if |Crl is too large such that
C
42
p2
> 4 log
+ 2 4 4og
2d
r
r2
2L2
then according to Lemma
G.1
F.3
will not hold. Thus the necessary condition for F.3 to hold is:
[²|Crl
(4L2
B2
22B (-r) r
2LBe
3LB
<
4log
+1 = 8 8log
+ log(e)) = 8 log
< 8 8log
2d
r
r
Applying basic calculus, we oblain the claimed bound for |Cr| and thus complete the proof of Lemma E.1
F.2
Proof of Lemma E.2
Proof of Lemma E.2 This proof follows the proof Di et al. 2024 For each round t E (T], we define the
following auxiliary quantities:
Go(0) XO + Σ
(((6)- - To u((Ø) 0z) o²)Tor) -0?)
TEC.-1
m(()) )²) 07)
Z, = Σ e ( - Φ²).
rEC
By defining 02 the solution to 4.1 we plug the equation into the definition of G2 and we have
G.(O.) = 10, +
Σ
[u(())
T²)To,) 0.) or + o, - =m(0² - -の) T.) [0) -6)
TEC
10, +
[((0) Φ.) 0.) or 10) (3) + Σ bor -n((0) - -3) o3)T0') (.)
TEC
7EC-11
Zr.
25

---

Published in Transactions on Machine Learning Research (09/2025)
Therefore, we have that
C,(O.) G,(0") - -.-(((0') 74 - 20"
On the other hand, applying Taylor's expansion, there exists the following equation holds:
[0. 1] and 0: ab.; (1- -a)0", such that
- 1(0, -0)+ E [u((;) - o;) 0) u((0) - -0,) 00000 Φ.,)
TEC--1
=[XI+
[ E (( - -7, ( ( - 02) - -0)
TEC.-1
F(. 0*),
where we define F(O,) AI + Erec.., /¹ (( (.)) (Φ) (z)(0) 2²² Thus, we have:
0 0- 0' = (Z) - (2-))0)) F(O,) E. ((0)) '(Z) 200)
(Z - (0 E 20")
N/
Zi 10 10
where the inequality holds due to F O()) ž Kip
- Now we have:
1
0
0" 1'11z,
5
1| |0
Kip
1
(||zel>z,,
|1 x
Kpp
< Kpp
(1Iz|l>z), + V\B),
where the second inequality holds due to triangle inequality and last inequality holds due to E,- 2 AI. Now only need to bound ||Z]]] x
According to Lemma G.2 with probability at least 1 - have
/ell(E. /dct(Enn (Eo)8
det det(En (1-1))
1c/LL//d
1ellz,1 I
V2 2og
< 2 log
2d
X48
where the first incquality holds duc to Lemma combine the two term and have
G.2 and the last incquality holds due to Lemma
G.5
Now we
A
|C,112/d
0 0 x S
VBB+ + 2d 2d
Ad8
which concludes our statement.
F.3 Proof of Lemma E.3
This proof follows the proof in Zhang et al. (023 First, recall that I' /2d1 and 3 = (1 44 + /2dzz). We will first demonstrate that the selection of B satisfy the requirement in Recalling that A = B B-2, through basic calculation, we have
Proof of Lemma E.3
Lemma E.3
K 2 + / /2dlog ((1 + 12
2 + / /2d log (233C33()))
VBB / 2og + (
26

---

Published in Transactions on Machine Learning Research (09/2025)
where the first inequality holds by neglecting the positive term 4v and d 2 1, the second inequality holds due to Lemma F.1 and the last equation holds by plugging in X = B-2. Now we come to the second statement. First, by basic computation, we have
221g 22og(( 16L3BTT-212 (llog(1/8))
Notice that we have L 1, B2 1, and r S 1, which further implies that LBI 2 leading
2 2+ 4/122 S 612, 2log(( 16I 312) S 312-
Therefore, we have:
2 + 4/ 221 s 962 + (0/o0(1/5
"xp'') s glog(6LLBv " (1) + 2 2o/og(1/0)
By Lemma G.1
identify the sufficient condition for the following inequality
(6LBV K )in 2 966LBV7 "x -x + " 1og(1/6) (F.4)
is that
2 36 (10 - + (ool))+Vllg(111 /8log(1/8),
-xp which naturally holds due to our definition of Eliminating the 6LBvdA - term in F.4 yields that
4 2 2 42 +V23,
which implics that
Axp (1 + 22 diz + 223)) < A.
2
2vdr Ky
Thus, we complete the proof of Lemma E.3
F.4 Proof of Lemma E.4
Proof of Lemma
E.4 For each context x E X' and action E A, we have
(D(x,)) (0r, (xx,))
of 10, 0" ",,((x))) +.:
< 10: 0 0'|| || v) - Φ2 ||
< B y) - & b
(F.5)
where the first incquality holds due to Cauchy-Schwand incquality and the second inequality holds due to event &. Therefore, we have
o ((.,v) Φ?) + 31 &ill... 2 Dz(x,y).
where the first inequality holds due these results, we have
F.5
addition, we have Di(x,y) = (0', "(,,(( - +Z) < 2. Combing
De y) = min (0,, ((x, Φ3) + +1]1((( v) - (1 2 D((a,v)-
On the other hand, we have
D,(x, < 0 ((2,)(
-3- + v) &Zll.. S De v) 2,11 v) &ill.,
where the first incquality holds due to the definition of D, y) and the second incquality holds due to Thus, complete the proof of Lemma E.4
F
27

---

Published in Transactions on Machine Learning Research (09/2025)
F.5 Proof of Lemma E.5
Proof of Lemma E.5 The proof follows the approach in He et al. (022a)) Recall that we assume the policy
is updated in round according to the update rule EE.1 for all contexts a E X. Thus, we have:
( T<(U(o) exp (),
exp (DD((((2))))
(F.6)
(
(y
where p= Eyer (y(la) exp (nDr(I, is the regularization term that is the for all actions y E A. Therefore, we have
Σ (E (v|x) (++1())))) (+((((()())))))
(log p + log T (1/=)) log Nz (()l)) Te+ (
sEA
Σ
x" ('(lz) (1yl)) - ( - Te+ (z/ ( (1o - log n. (y/lx))
DEA
Σ
(y]lz) (log - (vlx) +
Σ
n
log "r" (v|x:)
PEA
yEA
Σ (1+1(y/=)( ( log re (B)(2))
VEA KL (1=))
(F.7)
where the first equation holds due to 0. Consequently, we have
F.6 and the second equation holds due to Eyen (T012)))++1(122))) =
Eysrr (-1x) [D((r, Ey [)( (
Σ D((E,))( ()(2²))
VEA
Σ Di( (11(0) + Σ Dz (F (v(xx)
yEA
DEA
Σ (((r)))TT( - ()/() + 2 (11))) - (1))))(()))]
WEA
" (KTT(T((((())))))) KI. (())))
+ =21mmc4(((1)-
$ 77 1 ([[]((())) (z) (1)
-
2n
< 21 + (K
KL
(-<x)()))) [[ (|
(F.8)
where the first inequality holds due to the fact that 0 D. Pinsker's inequality and the last inequality holds due to the fact that ax - bx² < a² 22/4b. Finally, taking expectation over x ~ finishes the proof.
v) < 2, the second inequality holds due to
G Auxiliary Lemmas
Lemma G.1 (Lemma A.2, Shalev-Shwartz & Ben- David 2014). Let a 2 1 and b 2 0, then x: 2 4a logg(22) (2a)+2b + results in 2 a logg b.
28

---

Published in Transactions on Machine Learning Research (09/2025)
Lemma G.2 (Theorem 1, Abbasi-Yadkori al. 2011). Let {Fi) be filtration. Let {4) be real-valued stochastic process such that is F:-measurable and is conditionally R-sub-Gaussian for some Rt 2 O. Let { be an -aluuee stochastic process such that Φ; is F;-1 measurable and || S L for t. For any t 2 0, define U, = + Ei=1 Φ Then for any 5 > 0, with probability at least for all A
2 0, we have
det(Uu) .
2R log
(eo((U
Lemma G.3 (Lemma 11,
Abbasi- Yadkori al. 2011 Let {øili= be a sequence in R define U,
AI + 0 then
Σ min {1, 11lbbbll } 2d
Ad +d+
Ad
The following auxiliary lemma and its corollary are useful
Lemma G.4 (Lemma A.2, Shaley-Shwartz & Ben- 2014). Let 2 and b >> 0. Then 2 4a log(2a))+ +2b
yields x 2 a log(s) +
Lemma G.5 (Lemma C.7, Zhang et al. 2023
Suppose sequence (x: C RR and for any i S |x < L. For any index subset C [ define U + Eiec x,X, for some > 0, then det(U) s (A+ 1C+1c//3/d)
Lemma G.6 (Azuma-Hoffdding inequality, Cesa-Bianchi & Lugosi 2006). Let {z be martingale difference sequence with respect to a filtration {9i} satislying [x, S M for some constant M, Ia is 9,+1- measurable, [[[r[[]]] = 0. Then for any 0 < ő < 1, with probability at least. - 6, we have
n
Σ
24 S M 2nloogg(1//))
Lemma G.7 (Lemma 11 Abbasi-Yadkori et al. positive definite matrix and define U, = Ei= Φ Φ. have
2011)
Let { be a sequence in Rd, U x d ||0i112 S and Amin (U) 2 max(1, L2), then we
det U
(U,-1) 1. s 2log
det U
29


Reviews:
[{"review_id": "kYqfTOBAMr", "reviewer": "Reviewer_Ga6d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. This paper tackles a highly relevant and practical problem in LLM alignment, i.e., the high cost of collecting human preference data.\n\n2. APPO is rigorously analyzed with solid theoretical guarantees, featuring regret and query complexity bounds that improve over prior work by removing dependency on the action space.\n\n3. ADPO introduces a compelling pseudo-labeling mechanism for training under uncertainty, which is empirically shown to be critical for data efficiency.\n\n4. Experimental results are comprehensive, covering both objective and subjective evaluation settings, and including ablation studies on pseudo-labeling and uncertainty thresholds.\n5. The work contributes to bridging theoretical insights with practical LLM training pipelines.\n\n## Weakness\n1. The pseudo-labeling mechanism, while empirically justified, lacks a quantitative analysis of its error rates or possible accumulation of bias during training.\n2. Comparisons are limited to DPO; other related baselines such as PPO with active querying, AURORA, or recent active RLHF strategies are not included\n3. The choice of uncertainty threshold γ is empirically tuned but lacks adaptive mechanisms or principled justification", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a query-efficient framework for reinforcement learning from human feedback (RLHF) by incorporating active learning strategies. The authors formulate RLHF as a contextual dueling bandit problem and propose a theoretically grounded algorithm called Active Proximal Policy Optimization (APPO), which achieves instance-dependent regret bounds independent of the action space size. Furthermore, the authors develop a practical instantiation of APPO named Active Direct Preference Optimization (ADPO), which integrates pseudo-labeling into a DPO-style training pipeline. Experimental results on multiple large language models (LLMs), including Zephyr-7b variants, across both objective (e.g., ARC, TruthfulQA, HellaSwag) and subjective benchmarks (e.g., MT-Bench, AlpacaEval 2.0), demonstrate that ADPO achieves performance comparable to or better than standard DPO while requiring significantly fewer preference queries.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Same as the weakness above.", "broader_impact_concerns": "This work addresses an important bottleneck in RLHF by reducing reliance on human-labeled preferences, which has direct implications for scalability and accessibility of LLM alignment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ie6GFMyhYX", "reviewer": "Reviewer_n1tV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\nThe paper is exceptionally clear and well‑structured: concepts are introduced logically, and the training procedure for Active DPO is laid out step by step. Diagrams and pseudocode are easy to follow, and the appendix provides all necessary implementation details.\n\nThe ablation studies are thorough, quantifying how the confidence threshold γ affects both query volume and end‑task performance. This not only validates the core idea but also offers practical guidance for tuning.\n\nAdding a simple, online‑trained logistic regression “gate‑keeper” represents a small yet highly effective extension to vanilla DPO—cutting human/LLM queries by roughly half with minimal loss in accuracy. The elegance of this minimal intervention makes it broadly applicable.\n\nExperimental evaluation on the Zephyr backbone demonstrates real gains, and comparisons against both always‑query DPO and a passive baseline firmly establish the benefits of the active approach.\n\nWeakness :\nAll experiments use the Zephyr architecture. It remains unclear how well Active DPO transfers to other larger LLMs (LLM with more params) or multi-model.\n\nThe approach relies on a fixed confidence cutoff. In reality, the optimal γ may drift over time or vary by data domain; a dynamic or adaptive threshold could offer further gains.\n\nLogistic regression may struggle on more complex tasks where the decision boundary between outputs is highly nonlinear. Testing richer classifiers  could reveal whether the choice of gate‑keeper matters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the high cost of human (or proxy) preference queries in Direct Preference Optimization (DPO) by introducing Active DPO, which adds a lightweight “gate‑keeper” model to decide when to ask for a preference label. In standard DPO, every pair of model outputs \nis sent to an expensive reward model—a human or large LLM—to obtain a preference label. Active DPO interposes a simple logistic regression classifier trained online: for each input, the base model proposes two outputs, the gate‑keeper predicts which is better and with what confidence, and only if that confidence falls below a threshold γ does the system defer to the oracle and add the true label to the classifier’s training data. By trusting the gate‑keeper whenever, the method auto‑labels “easy” cases and spares the oracle, while still querying on hard examples where its cost is justified. Experiments using a Zephyr‑based architecture show that Active DPO can cut total queries by roughly half with negligible loss in task accuracy. An ablation over different γ values (1.0, 1.3, 1.5) further illustrates the trade‑off between query volume and performance: higher thresholds yield fewer oracle calls but incur a small accuracy drop. In essence, Active DPO retains nearly all the benefits of full DPO at a fraction of the labeling expense by adaptively routing only uncertain cases to the costly reward model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No, I think paper has covered everything in detail. If you could explain few things in detail?\n1. One thing thats bothering me, are we really using simple logistic regression or is the paper using NN based logistic regression?\n2. Is reward model another LLM pre-trained on preference dataset or its human labeller who gets the prediction queries to label as per preference?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qvk7O2X1pK", "reviewer": "Reviewer_grAN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- This paper provides both a theoretical analysis and a practical application of the proposed active learning algorithm.\n- The paper is well structured and clearly motivated.\n- Experiments are well presented with two models trained on instruction-tuned datasets and evaluated on well-recognized datasets.\n- The work provides a data-efficient alternative to RLHF, making this process cheaper to run and more accessible.\n\n## Weaknesses\n\n**[W1]** The paper does not report the additional computational cost of using the proposed method in the experimental section. While active learning decreases the number of queries made (or samples used), the evaluation of the model, the potential query informativeness and other criteria probably adds computational costs. It would be nice to report such metrics to better visualize the tradeoff. Metrics like GPU hours or FLOPS could be used for that.\n\n**[W2]** While the method relies on fewer samples, in practice, one does not know how many samples will be needed in advance. As such, active learning methods only make sense in a labeling process with human labelers in the loop, which is slower (though cheaper) than gathering a big dataset of labels all at once. For simplicity, the paper currently uses an already labeled dataset of a fixed size. It would be nice to discuss this limitation in the main text.\n\n**[W3]** Finally, the ablation study of the confidence threshold currently covers values from 1.0 to 1.5, requiring from 13k to 21k samples, and all achieving better performance than the DPO baseline (Table 4). It would be very interesting to also report much slower thresholds and to identify the minimal threshold before the performance of the model collapses. What happens in practive if the thresholds is 0.0 and the learner never asks for a label? Recent work, such as “_Reinforcement Learning for Reasoning in Large Language Models with One Training Example_” suggest that only a very small amount of data is needed to achieve good reasoning performance. While these results are to be taken “with a grain of salt”, the work presented in this paper could potentially shed some light into these minimal training data paradigms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel active learning variant of PPO and DPO (APPO and ADPO, respectively) as a sample-efficient way to perform reinforcement learning from human feedback (RLHF). The proposed method formulates the RLHF problem as a contextual dueling bandit problem, where an estimator for the reward gap between the two arms is trained with maximum likelihood. Importantly, the active learning algorithm only queries human preferences for data where the model exhibits high uncertainty.\n\nAfter presenting the query-efficient Active Proximal Policy Optimization (APPO) algorithm, the paper presents some theoretical analysis about the efficiency (level of regret and query complexity) of the algorithm. It then presents the more practical Active Direct Preference Optimization (ADPO) algorithm before conducting experiments with ADPO.\n\nTwo base models (Zephyr-Beta-SFT and Zephyr-Gemma-SFT) are trained with DPO and ADPO on the same human instruction datasets (Ultrachat-200k and deita-10k-v0-sft, respectively) and compared on multiple tasks (ARC, TruthfulQA, HellaSwag, MT-Bench, and Alpaca Eval). Results show that the ADPO-trained models perform similarly (if not better) than the DPO-trained variants at a fraction (¼ to ½) of the number of queries. Ablation studies also show the importance of the pseudo-labels and the effect of the confidence threshold for the learner to make a human label query.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add some sort of compute or time requirement for each model presented in Table 1 & 2 to have a tradeoff discussion in the results.\n\n- Explore lower threshold values in Table 4 (at least 1). Ideally, a figure with the X-axis being the threshold (with corresponding number of queries in parentheses) and the Y-axis being the average performance would be amazing, especially with very high and very low thresholds.\n\n- The caption of Table 2 has a copy-pasted sentence from the caption of Table 1: “_Besides, ADPO only makes 16k queries … made by DPO_”. But Table 2 does not report the number of queries made. It would be nice to report the number queries made on MT-Bench and Alpaca Eval.\n\nTypos:\n- Under “Uncertainty-Aware Query Criterion” paragraph in Section 4: “_a pair of action requires y^1_t and y^2_t requires human-labelled preference._”.\n- In Theorem 5.1: “_… in Algorithm 1, then with with probability at least…_”\n- In Appendix B, Evaluation setup paragraph: “_For obejctive benchmarks_”", "broader_impact_concerns": "The current broader impact statement in the paper is good.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)