ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
Authors: Xiaoqiang Lin
Arun Verma
Zhongxiang Dai
Daniela Rus
See-Kiong Ng
Bryan Kian Hsiang Low
Venue: ICLR 2026 Poster (2026)
Keywords: Active Data Selection
Direct Preference Optimization
Human Feedback
LLM Alignment
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose an active learning algorithm that uses a theoretically grounded selection criterion while using LLM to parameterize the reward model for efficiently collecting human preference feedback when the latent reward function is non-linear.

Abstract: The recent success in using human preferences to align large language models (LLMs) has significantly improved their performance in various downstream tasks, such as question answering, mathematical reasoning, and code generation. However, achieving effective LLM alignment depends on high-quality datasets of human preferences. Collecting these datasets requires human preference annotation, which is costly and resource-intensive, necessitating efficient active data selection methods. Existing methods either lack a strong theoretical foundation or depend on restrictive assumptions about the reward function, such as linear latent reward functions. To this end, we propose an algorithm, ActiveDPO, that uses a theoretically grounded data selection criterion for non-linear reward functions while directly leveraging the LLM itself to parameterize the reward model used for active data selection. As a result, ActiveDPO explicitly accounts for the LLM's influence on data selection, unlike methods that select data without considering the LLM that is being aligned, thereby leading to more effective and efficient data collection. Our extensive experiments demonstrate that ActiveDPO outperforms existing methods across various models and real-world preference datasets.

Published as a conference paper at ICLR 2026
ACTIVEDPO: ACTIVE DIRECT PREFERENCE OPTIMIZATION FOR SAMPLE-EFFICIENT ALIGNMENT
Xiaoqiang Lin'
Arun Verma Zhongxiang Dai
Daniela Rus 1.2 See-Kiong Ng² Bryan Kian Hsiang Low1.2
Deparrmeett of Computer Science, National University of Singapore, Singapore Singapore- MIIT Alliance for Research and Technology Centre, Singapore 'The Chinese University of Hong Kong, Shenzhen, China
*CSAIL, Massachusetts Institute of Technology. USA
Institute of Data Science, National University of Singapore, Singapore
xi xiaoqiane ang. 1 inuu. nus edu, verma aammammaatt...mm smart memaart.miie .iit.edu, da wrizhhnnxiannge cuuh edu. cn, rus@csail mitt. edu, seekiong@nus edu.sg 1owkh0comp nus edu.sg
ABSTRACT
The recent success in using human preferences to align large language models (LLMs) has significantly improved their performance in various downstream tasks, such as question answering, mathematical reasoning, and code generation. However, achieving effective LLM alignment depends on high-quality datasets of human preferences. Collecting these datasets requires human preference annotation, which costly and resource- intensivee, necessitating efficient active data selection methods. Existing methods either lack strong theoretical foundation depend on restrictive assumptions about the reward function, such as linear latent reward functions. To this end, we propose theoretically grounded data selection criterion for non-linear reward functions while directly leveraging the LLM itself to parameterize the reward model used for active data selection. As a result, Act i veDPO explicitly accounts for the LLM's influence data selection, unlike methods that select data without considering the LLM that is being aligned, thereby leading to more effective and efficient data collection. Our extensive experiments demonstrate that Act iveDPO outperforms existing methods across various models and real-world preference datasets.
algorithm, Act i veDPO, that uses a
INTRODUCTION
Large language models (LLMs) (Google, 2023; OpenAl, 2023; Touvron et al., 2023; Anthropic, 2023) have demonstrated impressive performance across various tasks, including question-anse answering (Taori et al., 2023), mathematical reasoning (Wei et al., 2022), code generation (Chen et al., 2021), and many others (Zhao et al., 2023). However, L.MSs often fall short when required to produce responses that conform specific formats align with human values (Ji al., 2023; Anwar et al., 2024). To address this, methods such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Bai et al.. 2022) and Direct Preference Optimization (DPO) (Rafailov al., 2023), use binary preference feedback collected from human annotators, who indicate which of two LLM responses they prefer, better align LLM outputs with human preferences real-world applications. Both RLHF and DPO require high-quality human preference datasets to achieve effective LLM alignment. However, collecting these datasets requires skilled human annotators, making this process both costly and resource-intensive (Liu et al., 2024; Carvalho Melo et al., 2024; Muldrew et al., 2024).
To overcome these challenges, recent works (Mehta al., 2023; Das et al., 2024; Liu al., 2024; Muldrew et al., 2024) have proposed methods for actively selecting a smaller subset of preference data (i.e., triplets consisting of a prompt and two responses) for human preference annotation while maintaining alignment performance. Specifically, some existing works (Liu al., 2024; Muldrew et al., 2024) have proposed heuristic methods for actively selecting preference data to collect human
"Corresponding author.

---

Published as a conference paper at ICLR 2026
preference feedback. However, these methods lack rigorous theoretical foundation and therefore do not guarantee reliable performance across different tasks and LLMs (see Fig. 1 in Section 4). In contrast, some works (Mehta et al., 2023; Das et al., 2024) have developed methods with theoretical guarantees to achieve sample-efficient L.MM alignment. However, these methods require strong assumptions about the underlying latent reward function (e.g., linearity), which may not hold in the context of LLM alignment. Furthermore, another potential limitation of some existing works (Mehta et al., 2023; Das et al., 2024; Liu et al., 2024; Muldrew et al., 2024) is their dependence on separate reward model or a selection method that works independently of the LLM being aligned. As result, these methods often cannot directly account for how data selection affects the LLM itself.
These limitations naturally lead to the following question: How can we develop an active preference data selection algorithm that both theoretically grounded and practically effective? To answer this, we propose Act i veDPO, novel active preference data selection algorithm. Act VeDPO built on DPO, which has shown comparable or superior empirical performance to RLHF while avoiding the complexity of reward model training and the reinforcement learning process, making it a compelling choice for aligning LLMs with human preferences (Rafailov et al., 2023). Furthermore, ActiveDPO uses theoretically grounded criterion for selecting preference data for complex, non-linear reward functions, while using the LLM itself as a reward model to guide that selection.
Specifically, we establish an upper bound on the error in estimating the reward difference between any pair of responses and their ground-truth reward for a given prompt, expressed in terms of the gradient of the neural network, which represents the current estimate of the DPO reward function associated with the aligned LLM (Proposition 1 in Section 3). This result enables us to leverage the LLM's gradient derive an uncertainty measure as a criterion for preference data selection, thereby explicitly accounting for the LLM's influence on the data selection process. To improve the efficiency and practicality of Act iveDPO, we introduce novel techniques, such as batch selection and random projection with LoRA gradients (more details are given in Section 3.3) to reduce computational cost and storage requirements. These additional techniques make Act i veDPO both theoretically grounded and practically effective. Finally, extensive experiments demonstrate that Act vvDPO consistently outperforms existing methods across various LLMs and datasets.
The key contributions can be summarized as follows:
In Section 3, we propose novel algorithm, Act i veDPO, that uses theoretically grounded active preference data selection criterion for LLM alignment. By leveraging an implicit reward function parameterized by the LLM itself, ActiveDPa ensures that the selected preference data is better suited to the specific LLM being aligned.
Section 3.3, we introduce techniques such as batch selection and random gradient projection to reduce the computational and storage requirements of Act making it more practical for large-scale models.
In Section 4, we empirically demonstrate that Act i veDPO achieves efficient and effective active preference learning across diverse LLMs and datasets.
2 PROBLEM SETTING
In LLM alignment, we start with preference dataset D in which each data point contains a triplet (z, V1, 12) where x E X' is a prompt and //1 y2 E y are two responses (which can be written by humans or generated from LLMs). The X' and y are the prompt space and the response space, respectively. Denote n as the number of data points in D. We aim to find k-sized subset D' C D and ask human annotators to provide binary preference feedback on the responses denoted as tie > Vt where you and y/ denote the preferred and rejected response, respectively. Note that y is not the numan-preferent
label but the corresponding response to the prompt. We train the LLM to generate responses that better align with human preference on the labeled data subset D' using DPO. The objective is to obtain an LLM that gives the most desirable responses (defined by win-rate and reward score as we will discuss later) within the fixed labeling budget of k.
Direct preference optimization (DPO). We first start by discussing the DPO method, as introduced in Rafailov et al. (2023). DPO starts by training an LLM through supervised fine-tuning (SFT) carefully curated, high-quality dataset that is specifically tailored to a particular downstream task, resulting in model, denoted by TSr.- The objective of the SFT is to enable the L.LM to effectively
2

---

Published as a conference paper at ICLR 2026
follow instructions for specific downstream task. Let To ar) denote the conditional log- likelihood of generating y given the prompt æ, where the model is parameterized by 8. Within DPO, an implicit reward function is defined follows:
re( Toly x) x))
Teer(y
where Tee is the reference LLM, which is usually chosen to be the SFT LLM TSFT, and / is the regularization syper-paramete used in DPO. Based this implicit reward function, DPO uses Bradley-Terry-Lun Terry-Luce (BTLL) to model the preference feedback. Specifically, BTL assumes that the probability of response y1 being preferred over y2, conditioned on the prompt æ. is given by:
exp (r0 (1.)))
ply1 y2 x)
(ro(2,, 91) re v2)),
(1)
exp (re(z, v1)) + exp (ra (2))
where (xx) 3/(+tep((=x)) DPO uses the following training objective to train the LLM: LDPO (T) Teet
E (2,
wi)= (
log (yo xx) ro(u |=)]]
(2)
METHODOLOGY
Overview of ActiveDPO. Act i veDPO starts with generating responses from an initial data D, which consists of instructions/prompts tailored to a specific task. We use the initial LLM model (i.e., TSF) to obtain the responses to form the dataset D,, which forms the pool of selection (Section 3.1). After that, we select a batch of triplets (I,, 3/. 3/2 with size according to our selection criterion (Section 3.2). Then,
ask the human annotator provide preference feedback on the responses for the selected batch of data to obtain the labeled dataset. Finally, we train the LLM with the DPO training objective on the newly labeled dataset. We run this process for T iterations and obtain final trained LLM that generates responses aligned with human preferences.
ActiveDPO Active Direct Preference Optimization
1: Input: Initial dataset D; Reference LLM Tee TSF:: Initial LLM Teo = TSFT' parameterized by 0o: Iteration T;; Batch size B;
2: for tort=1 1,..,, ,, do
3: Generate m pairs of responses from previous LLM y1, V/2 ~ To. (y x)) for each D to obtain the dataset D..
4:
Dÿ =
5:
for 1y=1.... B do
6:
Select the (I V6,1 26,2) using Eq. (3)
7:
Dj Dj U { yi, 21
8:
Update V according to Eq. (4).
9:
end for
10: Ohtain the preference feedback Yu > yt x for each data point in D2 to get the labeled dataset
D
11:
Update the LLM T&.., using DJ with the DPO training objective in Eq. (2) to obtain me.
end for
13:
Return the trained LLM T0r
3.1 GENERATION OF THE PROMPT-RESPONSES DATASET
In each iteration of ActiveDPO, we regenerate the responses for each instruction/prompt in the dataset for two main reasons. Firstly, even though some tasks already have responses written by humans or generated by powerful L.LMs, most do not have good responses for each instruction the start. Generating responses is necessary for these tasks before asking human annotators to provide preference feedback on these responses. Secondly, even though some tasks already have responses for each instruction, these responses are not updated as the LLM improves over time. This is undesirable, since the LLM will not be able to learn to generate better responses (relative to the responses in the original dataset) it improves. Consequently, we generate new responses for all instructions using the latest model obtained from ActiveDPO, enabling Act iveDDO training to further improve the L.M with higher-quality responses.
3

---

Published as a conference paper at ICLR 2026
3.2 SELECTION OF DATA TO GET HUMAN PREFERENCE ANNOTATIONS
The selection strategy of our Act i veDPO is inspired by principled neural dueling bandits (Verma al., 2025), which derives uncertainty quantification of the human preference for the latent non-linear reward function estimated using a neural network (NN). Inspired by this, we derived the uncertainty quantification on human preference for our LLM trained by DPO and show its empirical effectiveness in Section 4. Consequently, our selection strategy is theoretically grounded and provides empirical effectiveness instead of using the heuristic- based method (Muldrew et al., 2024).
Proposition 1 (Estimation error of the reward difference (informal version of Proposition 2)). Let ra denote a fully connected neural network with a width of m in each layer and depth of L. Let ò (0, 1). Assume that there is a ground truth reward function and that human preference is sampled from BTL preference modeling. As long as m 2 M, then with a probability of at least 1 - 6,
[ro__] (x, ro,-, 92)] [r(x, - r(x, r(x,,2)]] v2)] I
1 (Vro, (x, Vro, . (x,, 12 +E
uT m
for all x X and V1. 32 E V,t E [T] when using the DPO objective defined in Eq. (2) with additional regularization term to train this reward function ro--1- Vi-1
E E Do Pt-1( V1. 12))::1 (2, y1, 82) and (t-1(x, y1, 82) Vro (T, (2)). The definition of M, MT, E can be found in the Appendix A.
1 vm (Vro, (I,91)
Proposition is based on the theoretical results from neural dueling bandits (Verma et al., 2025).
m This result suggests that if J. (Vro, (I. v1) Vra, (2r, va)) is smaller, the estimation error of the reward difference will be smaller. Note that the reward difference directly decides the human preference according to the BTL preference modeling as shown in Eq. (1). Consequently, the reward function _ will have a more accurate estimation of the human preference on the two responses 91; U/ given On the other hand, if | (Vro,, (1) vn) - Vro (2. u2)) (2))ll is large, this indicates that the reward model will potentially have an inaccurate estimation of the human preference for the responses and hence a higher uncertainty on the human preference. Therefore, natural selection criterion arises with the uncertainty defined in Proposition 1. Based on this selection criterion, our selection strategy selects triplet context and pair of arms (z. V1, 322 as follows:
I, 31 y22 = argmax,,, 3e x.n. AD Vro, (I, v1) Vro. (2, 12)ll
(3)
The selection strategy in Eq. (3) uses the implicit reward function ro,-, which is parameterized by the current LLM ny.-.. Note that we remove 1/ /m from the selection criterion and ( since it only affects the scale of the gradient, and the depth is undefined for the LLM. The selection criterion quantifies how uncertain the current implicit reward function is about the human preferences for the responses y/ and 1/2- Specifically. larger value of the selection criterion in Eq. (3) means that the prompt-response triplet (x, V1, V2) is more different from the previously selected triplets. Therefore, using this selection criterion, our strategy encourages selecting responses that very different from the previous data and thereby explores the prompt-response domain to obtain more informative human preference feedback. This exploration helps improve the implicit reward function, as the reward function is trained on human feedback on diverse data in the domains. Although Proposition I derived for a fully connected neural network, we argue in Section A that its conclusions extend to the transformer architecture used in our experiments.
The matrix V=] in Eq. (3) plays an important role as diversity regularizer: it down-weights directions in the gradient space that have already been well-explored by previously selected data points. Specifically, as more data points are selected, V,-1 accumulates the outer products of their gradient features, causing data points with gradient features similar to those already selected to yield smaller values of the selection criterion. This encourages selecting data points whose gradient features are diverse and complementary to those previously selected. However, directly computing and storing the matrix Vi-1 (and its inverse) is intractable for modern LLMs, as the gradient dimension equals the number of model parameters. We address this computational challenge in Section 3.3 through LoRA gradients and random projection, which reduce the gradient dimension to a tractable size (c.g.. 8192).
4

---

Published as a conference paper at ICLR 2026
In addition to being theoretically grounded, our selection strategy offers two other advantages. Firstly, our uncertainty criterion is defined using the LLM we are training, rather than extemnl models used in existing methods (Carvalho Melo et al., 2024; Das et al., 2024). Using the uncertainty criterion defined without the L.M implicitly assumes that different LLMS require the same data for preference alignment, which does not hold in practice (as we will show in the experiments). Therefore, our selection strategy is specific to the LLM used and can select data that better aligns with human preferences for human preference alignment. Secondly, our selection strategy selects data that directly improves the reward function defined by the LLM and, hence, its generation, through the use of DPO. This strategy contrasts with prior work that focuses on selecting data points improve the reward function used in RLHF, An additional reinforcement learning process is needed to obtain the final LLM. This complication makes the data points selected not necessarily helpful for the LLM alignment performance, as having better reward function does not always result in better RL-trained LLM.
3.3 PRACTICAL CONSIDERATIONS
Our selection criterion in Eq. (3) requires the computation of gradients of the implicit reward function with respect to the LLM parameters for each prompt-response pair, as well as updating the LLM using the DPO training in every iteration. These steps are computationally expensive and require significant storage for gradient updates. To address these inefficiencies, we propose two acceleration techniques to improve the efficiency of our selection strategy, which we deseribe in detail next.
Batch selection. In each iteration, we select a batch of size B to be labeled by human annotators. We keep TB k keep the annotation budget the same. The batch selection accelerates the selection in two ways: 1) We only need to recalculate the gradient for each prompt-response pair (i.e., Vrg._, (z. y)) every selections of data instead of every selection; 2) We only need to update the model via DPO training every B selections.
Batch selection dramatically reduces the computational cost of our selection strategy; however, at the cost of the loss of information. Specifically, the data selected in the current batch will be different from previous batches but the selection within the batch may not enjoy similar results. To remedy this, we propose to update V within the batch. Specifically, after a data point is selected, we update Vi-1 using the new data point (ar vh,1 2
V V--1 + Pr-1 (zf, v6.1 Pt- v(,2)
Consequently, the next data point to be selected will also be different from the current one even though they are in the same batch, hence further encouraging exploration.
LoRA gradient with random projection. The computation of gradients in our selection criterion is expensive and requires a large storage space. Specifically, the full gradient of the LLM is the same size as the LLM model weight and we need to calculate and store the gradients for all data points. To reduce both computational cost and storage requirement, we propose to use LoRA (Hu et al., 2022) to obtain the gradient efficiently. However, the LoRA gradient is still 2% percent of the full model weight which still requires a lot of storage and computation for our selection criterion. Consequently, apply random projection to further reduce the gradient to fixed dimension. This random projection is justified by the Johnson-Lindenstene which shows that the inner product of the original vector can be approximated by the inner product of the projected vector via random projection. Consequently, we can reduce both the computational and storage costs dramatically without sacrificing too much on the selection performance (as shown in Section 4). Similar techniques have been used in Xia et al. (2024). The random projection also reduces the computational cost of the matrix inverse in Vi-1 1 in our selection criterion.
lemma (Dasgupta and Gupta, 2003)
Gradient normalization. Existing work (Xia et al., 2024) has demonstrated that the LLM gradients will have lower magnitudes in their 12 norms when the training data are longer in their length (i.e., sentence length). This means if we use the selection criterion defined in Eq. (3), we will have a higher chance of selecting training data with shorter lengths. This is undesirable, especially for the application of question-answering in which humans may prefer medium to long answers that contain more elaboration on the response. To remedy this, we propose to normalize all the gradients to the unit norm (i.e., le norm being 1) before we use these gradients to calculate the selection criterion, consequently avoiding the criterion favoring shorter sentences. We have empirically shown the effectiveness of normalization before calculating the selection criterion in Section 4.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
In our experiments, we show the effectiveness of our selection criterion in terms of selecting data to train an LLM that can generate responses that better align with human preference. We compare with multiple existing baselines using two widely-used LLMs across two preference alignment tasks.
Datasets. We consider two tasks that require human preference alignment: 1) TLDR summarization dataset (Liu al., 2020; Völske et al., 2017) which contains posts from Reddit and the corresponding summarization written by humans; 2) WebGPT dataset (Nakano et al., 2021) which is long-form puession-answwete dataset that marked suitable for human preference alignment. These two datasets contain human preference feedback from human annotators and will be used later as an oracle to obtain real human preference feedback.
Models. We performance experiments using 3 different LLMs: Llama-2-7B (Touvron et al., 2023), Gemma-2B (Team et al., 2024) and Qwen3-4B (Yang et al., 2025). Using these LLMs is able to show the effectiveness of alignment on 3 different model families (i.e.. Llama, Gemma, and Qwen) and 3 different model sizes (i.e., models with 7 billion parameters, 2 billion parameters, and 4 billion parameters).
Baselines. We compare 4 different selection criteria in our experiments: 1) Random: randomly select data points from the dataset to get human preference feedback; 2) APO (Das et al., 2024): theoretically grounded method in the setting of RLHF alignment. Their theoretical results are based on the assumption of linear reward function and is designed for RLHF training: 3) API.P (Muldrew et al., 2024), an active learning method for DPO that uses heuristic uncertainty/ccrrainty quantification select the data to be labeled; 4) Our proposed method Activeiee Note that, for fair comparisons, we only vary the way to select data points to be labeled for different methods and share the same model training and data labeling pipeline among different baselines. Consequently, the only variable that leads to different performance is the way to select data among different methods.
Obtaining human preference feedback. As new responses are generated by the updated model in each iteration, these responses are not part of the original preference dataset and hence do not have human preference feedhack. To make our experiments feasible, we train a reward model using the original human preference feedback and use this reward model as an oracle to provide the preference feedback for newly generated responses in each iteration.
Evaluation. The reward model can be used to evaluate the extent to which the LLM generates responses that align with human preference. To evaluate the performance, we use the trained LLM to generate multiple responses for 100 prompts sampled from the dataset for each task. After that, use the reward model to obtain the average reward for all the prompt-response pairs and report the performance. Ideally, if the LLM can generate responses with higher rewards, it aligns better with human preference since the reward model is trained on real human preference.
Hyper-parameters. For each task, we train the initial LLM with supervised fine-tuning with the SET dataset provided in each task for 1 epoch with the learning rate of 2e 05. In each iteration, we randomly select 1000 prompts from the dataset to generate 3 responses for each prompt. Consequently, each prompt will form 3 corresponding triplets (Ir, y1, y2) (i.e.. (3) number of pairwise combinations) and hence 3000 data points the dataset D,- We select 50 data points in each iteration using different selection strategies. We train the model using DPO objective based on the labeled dataset for 4 epochs with the learning rate of 1e - 4. As for the LoRA gradient, we use the rank of 128 with a of 512. We project all the LoRA gradients 8192 dimensions, a dimensionality that balances performance and computational costs as we will show later.
Computational cost vs. label efficiency. We emphasize that the primary goal of ActiveDPO is label efficiency: achieving better alignment performance with fewer human annotations. While Act vvDDCC incurs additional computational costs for gradient computation and data selection compared to simpler methods such random selection, the cost of obtaining high-quality human preference annotations typically far exceeds computational costs in practice. Therefore, the additional
Note that, for APO, we implement the original algorithm (Das ct al., 2024) which does regencrate responses using the new models.
We use the reward function that is already trained and availahle in HuggingFace. Specifically, we use the model from ApenAssittant (2024a) for TLDR dataset and AsenAssistant (2024b) for WebGPT datasct.

---

Published as a conference paper at ICLR 2026
Random
Rardom
APO ActiveDPO
APO APLP ActiveDFO
1.0
Rewerd
Dewwern
ActiveDPO
Rewerd
0.5
0.5
Iteration
iteration
teration
(a) TL.DR with 1lma-2-7B
(h) TLDDR with Gemma-2B
(c) TLDRR with Qwen3-4B
0.0
Randori
Rardom
Random
APO
APO
0.0
APG
Reward
ActivaDPO
ActivaDPO
-0.2
ActivalIP
-1.0
-0.4
-0.6
-0.6
-0.8
-0.8
Beration
Iteratio
thenation
(d) WebGPT with Llama-2-78
(c) WebGPT with Gemma-28
(f) WebGPT with Qwen3-4B
Figure 1: Comparison of average rewards for responses generated by the LLM using different
selection strategies.
computational overhead of Act Aceiv is justified by its superior label efficiency. We provide a detailed computational cost comparison among different selection strategies in Section A.
Results. We have provided the comparison of the average reward of the responses generated by the LLM trained on the data selected by different selection strategies in Fig. 1. The LLM trained with data selected by our Act ivDDPO consistently generates responses with higher rewards compared other selection strategies across different LLMs and datasets. Consequently, our ActiveDPO outperforms all other baselines in selecting data for a fixed number of labeling budgets APLP performs well the Gemma model, however, it performs even worse than random on Llama-2. This is likely due to the heuristic design of the uncertainty quantification method in APLP, which does not work consistently well in different settings. Specifically, APLP uses the difference of the estimated rewards for two responses given prompt as part of the selection criterion. This criterion allows APLP select triplets with incorrect human preference predicted by the estimated reward function in the early stage when the reward function is inaccurate, hence improving the reward function estimation. This partially explains why APLP performs well in the first iteration for both TLDR and WebGPT the Llama-2 model. However, more human preference is collected, the reward function estimation is more accurate, and hence, the triplet with a large reward difference can be data points with correct human preference predicted by the estimated reward function and with large reward margin, which do not help to improve the reward function. Consequently, API.P performs badly in the later iterations. We provide more detailed discussion the differences between gradient-based criterion and APLP's APO also performs inconsistently in different settings. This is likely due the unrealistic assumption of the reward function, which does not hold in real applications (e.g., the implicit reward function in DPO is non-linear).
criterion in Section A. On the other hand,
The impact of LLM on data selection. We perform additional experiments to verify that different I.MS indeed need different data to achieve better performance. Specifically, we train the Gemma-2B model on two different SFT datasets to obtain two different L.Mss: Model and Model 2. The way we construct the 2 SFT dataset is by using the sentence- transform each data point to embedding and use k-means to cluster the dataset into two subsets using the embeddings. We obtain 3 different DPO data subsets using the same approach. We train these
(Reimers and Gurevych, 2019) to
'The result of DPO alignment training is cooblem-dependee (depending on the dataset and model). For datasets that are very noisy, it is expected that different active learning methods will perform similarly. For larger models, different selection methods also tend to perform more similarly than for smaller models (which explains Fig. (a) where our method performs similarly other methods the last iteration).
7

---

Published as a conference paper at ICLR 2026
2.1
Model
0.84
Model 1
Model 2
Model
2.0
Reward
Win-rate
0.82
1.9
0.80
1.8
0.78
0.76
0
1
2
Dataset ID
Dataset ID
(a) Reward
(b) Win-rate
Figure 2: Different models require different data to achieve good alignment performance. We train the Gemma model using two different SFT datasets to obtain Model and Model 2. We construct 3 different human preference datasets and perform DPO training on these datasets for these two models respectively.
two LLMss on DPO data subsets respectively, and evaluate their performance. From Fig. 2, Model and Model achieve very different performance using these 3 DPO data subsets. Specifically, Model 2 achieves the best performance on Dataset 2, while Model achieves the worst performance on the same dataset (i.e., in terms of the win-rate). Consequently, the choice of model has a substantial impact on performance and must be considered when selecting data for achieving better model performance. Intuitively, this is because Model 1 and Model 2 are trained on very different SFT dataset, and hence require different new data to make up what they missed in the previous SFT fine-tuning.
The effect of random projection on the performance. Our method uses random projection to reduce the dimensionality of the LoRA gradients, reducing the storage requirement and computational cost for our Act iveDPO (as described in Section 3). To further study the effect of random projection to the performance
our Act iveDPO, we perform experiments on using different dimensionalities for the random projection and evaluate the performance of our Act ivDPO. The results in Figs. 3c and 3d show that a lower dimensionality leads to poorer performance of Act iveDP.. However, when larger
the dimensionality is 8192 or above, the performance of Act iveDOO does not improve dimensionality is used. Consequently, we use the dimensionality 8192 across all our experiments achieve good performance while keeping the computational cost and storage requirement low. We provide more analysis on the computational complexity and memory requirement for Act ivDDPO Section A.
0.75
0.9
-0.
-1.0
05
200
400
602
820
200
400
002 Z00
400
602
300
Namber ot dato selectod
Numbor of deto seacted
Number of dato selected
Number of data selected
(a) Reward
(b) Win-rate
(c) Reward
(d) Win-rate
Figure 3: Left two figures: Effect of normalizing LoRA gradients on the performance of ActiveDPO. Right two figures: Effect of Random Projection Dimensionality of LoRA gradients.
The effect of the normalization of the gradient on the performance. We perform experiments to verify the effect of normalizing the gradient in our Act veDPO. Specifically, as described in Section 3, we normalize LoRA gradients to unit-norm before we use them to calculate the selection criterion. We perform the selection of data using our selection strategy with gradient normalization compared to the one without gradient normalization. Figs. 3a and 3b shows the performance of our Actived iveDPO with/without the gradient normalization on WebGPT dataset using Gemma-2B model. These results show that normalizing the LoRA gradients helps to improve the performance of our selection strategy. As described in Section 3 our method will not favor the data points with shorter responses compared to the ones without normalization. Long responses with clear reasoning may
8

---

Published as a conference paper at ICLR 2026
sometimes be preferred by humans instead of shorter ones. Consequently, our Acti with gradient normalization performs better. We have included additional results for the TLDR dataset in the Appendix in which normalization does not affect the performance by much.
5 RELATED WORK
Learning from preference feedback has been extensively studied for over decade (Yue and Joachims, 2009; Fürnkranz et al., 2012; Christiano et al., 2017; Verma et al., 2019; 2020a;b; Zhu et al., 2023; Verma et al., 2025). In this section, we review work on dueling bandits, active preference learning, LLM alignment, and active LLM alignment, which are most relevant to our problem.
Dueling Bandits. One of the earliest works (Yue and Joachims, 2009; 2011; Yue et al., 2012) considers finite-armed dueling bandit problem in which the learner's 's goal is to find the best action using available pairwise preference between two selected actions. Several follow-up up works considers different settings involving different criteria for selecting the best action (Zoghi et al., 2014b;a; Ailon et al., 2014; Komiyama et al., 2015; Gajane et al., 2015) and we refer readers to Bengs et al. (2021) for a compressive survey covering these details. The standard dueling bandits has been extended to different settings, such contextual dueling bandit setting (Saha, 2021; Bengs et al., 2022; Di et al., 2023; Li et al., 2024; Verma et al., 2025).
Reinforcement Learning with Human Feedback. Preference feedback has also been extensively studied in reinforcement learning (Fürnkranz et al., 2012; Akrour, 2014; Christiano et al., 2017; Zhu et al., 2023) introduced preference- based policy iteration, a method that relies solely on preference feedback to guide reinforcement learning, with subsequent developments by (Akrour, 2014). (Christiano et al., 2017) demonstrated the effectiveness of human preference feedback in training agents for Atari games and simulated robot locomotion. On the theoretical side, research has progressed from bandit settings to reinforcement learning (Zhu et al., 2023), providing deeper insights into the optimal use of preference feedback for decision-makie and policy optimization. For more comprehensive overview, we refer readers to a survey on preference-bes based reinforcement learning (Wirth et al., 2017).
LLM Alignment. Recent works have introduced methods like Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; Lee et al., 2024) and Direct Preference Optimization (DPO) (Rafailov et al., 2023) to align LLMs with specific formats or human values. For comprehensive overview of various aspects of LLM alignment, we refer readers to surveys on the topic (Ji et al., 2023; Anwar et al., 2024).
Active LLM Alignment. Actively select preference queries for human to provide relative preferences between two queries allows efficiently learn reward functions that capture human intent. Some of works has already considered actively selecting queries in domain like autonomous (Sadigh et al., 2017; Biyik and Sadigh, 2018). Recent work on active preference data selection for LLM alignment has explored both heuristic methods (Carvalho Melo et al., 2024; Muldrew et al., 2024) and approaches with theoretical guarantees (Mehta et al., 2023; Das et al., 2024). A key distinction among these recently proposed theoretical methods lies in their data selection strategies. On the other side, existing methods with theoretical guarantees (Mehta et al., 2023; Das al., 2024) are based on the assumption of linear latent reward function, which may not hold in real weal-world applications such as LLM alignment in which reward functions often highly non-linear and complex.
6 CONCLUSION
In this paper, we propose a data selection method for actively selecting data to obtain human preference feedback for L.MM alignment, aiming to achieve better alignment performance with as few annotations as possible. To this end, we introduce a theoretically grounded method, Act iveDPO, and demonstrate that achieves superior alignment performance under the same labeling budget across different models and datasets. Notably, the selection criterion in Act i veDPO requires computing the gradient of the LLM with respect to model parameters for each data point, which is computationally expensive and demands substantial storage for storing gradients. We propose several techniques to improve the efficiency of our method. Although further efforts could be made to accelerate gradient computation, this is beyond the scope of the current work and is left for future research.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLLP222440001. This research supported by the National Research Foundation (NRF), Prime Minister's 's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. The Mens, Manus, and Machina (M3S) is an interdisciplinary research group (IRG) of the Singapore MIT Alliance for Research and Technology (SMART) centre.
REFERENCES
Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In Proc. ICML. pages 127-135, 2013.
Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In Proc. ICML, pages 856-864, 2014.
Riad Akrour. Robust Preference Learning-hased Reinforcement Learning. Phl) thesis, Université Paris Sud-Paris XI, 2014.
Anthropic. Introducing claude 2.1. https: /www//wwwwww/////////////w http::/:/www..tttrrrop......o//s 2023. [Online; accessed 01 February 2008].
1/,
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, al. Foundational challenges in assuring alignment and safety of large language models. 20449924,. 2024.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. reinforcement learning. In Proc. ICML, pages 263-272, 2017.
Minimax regret bounds for
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.0588 05862, 2022.
Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Part and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey, Journal of Machine Learning Research, pages 1-108, 2021.
Viktor Bengs, Aadirupa Saha, and Eyke Hüllermeier. Stochastic contextual dueling bandits under linear stochastic transitivity models. Proc. ICML, pages 1764-1786 2022.
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Proc. CRL. pages 519-522 1199528.11 2018.
Luckeciano Carvalho Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. Deep bayesian active learning for preference modeling large language models. In Proc. NeurlPS, pages 118052-118088 2024.
Baiyuan Chen, Shinji Ito, and Masaaki Imaizumi. Optimal dynamic regret by transformers for non-stationary reinforcement learning. arXiv:2508. 16027, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv:2107.03374 2021.
Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Proc. NeurIPS, pages 4302-4310, 2017.
Nirjhar Das. Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. In JCML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024.
10

---

Published as a conference paper at ICLR 2026
Sanjoy Dasgupta and Anupam Gupta. An elementary proof of theorem of johnson and lindenstrauss. Random Structures Algorithms, 22(1):60-65 2003.
Qiwei Di, Tao Jin, Yue Wu, Heyang Zhao, Farzad Farnoud, and Quanquan Gu. Variance-aware regret bounds for stochastic contextual dueling bandits. Stkiv23300000900 2023.
Johannes Fürkrranz Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning. pages 123-156, 2012.
Pratik Gajane, Tanguy Urvoy, and Fabrice Clérot. A relative exponential weighing algorithm for adversarial utility-based dueling bandits. Proc. ICMI., pages 218-227, 2015.
Google. Palm 2 technical report. arXiv:2305. /00::3335//403,, 2023.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proc. ICLR, 2022.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, al. Ai alignment: A comprehensive survey. arXiv:2310.19852 2023.
Junpei Komiyama, Junya Honda, Hisashi Kashima, and Hiroshi Nakagawa. Regret lower bound and optimal algorithm in dueling bandit problem. In Proc. COLT, pages 1 1141-1154, 2015.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with feedback. Proc. ICML, pages 26874-26901, 2024.
Lihong Li, Wei Chu, John Langford, and Robert Schapire. A contextual- approach to personalized news article recommendation. In Proc. WWW, pages 661-670, 2010.
Xuheng Li, Heyang Zhao, and Quanquan Gu. Feel-good thompson sampling for contextual dueling
bandits. arXiv:2404. N.Xx:22444.00113, 2024.
Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. Proc. ICLR, 2024.
Fei Liu et al. Learning to summarize from human feedback. In Proc. ACL, 2020.
Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Sample-efficient alignment for llms. arXiv:2411.0149) 2024.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. kernel-based view of language model fine-tuning. In Proc. ICML, pages 13610-23641 PMLR, 2023.
Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via active exploration. arXiv:2312.00267, 2023.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Proc. ICML, pages 36577-36590, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv:2112.09333,, 2021.
OpenAl. Gpt-4 technical report. arXivv2303.08774, 2023.
OpenAssistant. DeBERTa large summarization reward model. https: http:::/ttt/////uee/eee hww.gggggggccceeeeeee/o co/ OpenAssistant wttp://eeeeweedddddddddeeeeeeeee
2024a. Accessed: 2025-02-19.
11

---

Published as a conference paper at ICLR 2026
OpenAssistant. DeBERTa large summarization reward model v2. https:/ https://huugingface 2025-02-19.
www.arrddoooel-ddddde watpa//ddoddodooaaaa------a----- 2024b. Accessed:
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Proc. NeurIPS, pages 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proc. NeurIPS, 2023.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BeRT-networks. In Proc. EMNLP. Association for Computational Linguistics, November 2019.
Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. In Proc. RSS, 2017.
Aadirupa Saha. Optimal algorithms for stochastic contextual preference bandits. In Proc. NeurIPS, pages 30050-30062, 2021.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proc. NeurIPS, pages 3008-3021, 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-followim model. Stanford Center for Research on Foundation Models. https://crfm. stanford. htt/22223/3/13/apppeee/ html, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv:2403.08295, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXivv2307.009288 2023.
Arun Verma, Manjesh K Hanawal, Csaba Szepesvári, and Venkatesh Saligrama. Online algorithm for unsupervised sensor selection. In Proc. AISTATS, pages 3168-3176, 2019.
Arun Verma, Manjesh K Hanawal, and Nandyala Hemachandra. Thompson sampling for unsupervised sequential selection. In Proc. ACML, pages 545-560, 2020a.
Arun Verma, Manjesh K Hanawal, Csaba Szepesvári, and Venkatesh Saligrama. Online algorithm for unsupervised sequential selection with contextual information. In Proc. NeurIPS, pages 778-788, 2020b.
Arun Verma, Zhongxiang Dai, Xiaoqiang Lin, Patrick Jaillet, and Bryan Kian Hsiang Low. Neural dueling bandits: Principled preference-hased optimization with non-linear reward function. In Proc. ICLR, 2025.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Sunmarization, pages 63, 2017.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Proc. NeurIPS, 2022.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, pages 1-46, 2017.
12

---

Published as a conference paper at ICLR 2026
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning (ICML), 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006. 145444 2020.
Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems dueling bandits problem. In Proc. ICML, pages 1201-1208, 2009.
Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proc. ICML, pages 241-248, 2011.
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, pages 1538-1556, 2012.
Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural Thompson sampling. In Proc. ICLR, 2021.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv:2303. 18223, 2023.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedhack from pairwise or k-wise comparisons. In Proc. ICML, pages 43037-43067, 2023.
Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten Rijke. Relative upper confidence bound for the k-armed dueling bandit problem. Proc. ICML, pages 10-18, 2014a.
Masrour Zoghi, Shimon A Whiteson, Maarten De Rijke, and Remi Munos. Relative confidence sampling for efficient on-line ranker evaluation. In Proc, WSDM, pages 73-82, 2014b.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
COMPUTATIONAL RESOURCES, DATASETS AND MODELS
Experiments are run on a server with AMD EPYC 7763 64-Core Processor, 1008GB RAM, and 8 NVIDIA L40 GPUs.
Dataset license. TLDR dataset: MIT License; WebGPT dataset: Apache License 2.0.
Model license. Llama-2: LLAMA 2 Community License Agreement. Gemma: Gemma License.
Qwen3: Apache License 2.0.
A.2 ADDITIONAL, DISCUSSION ON PROPOSITION
Although our results in Proposition I rely on neural tangent kernel (NTK) theory, which is primarily developed for fully connected networks, recent works (Lin et al., 2024; Chen et al., 2025) have shown promising directions for partially extending NTK theory to transformers. Furthermore, recent studies (Lin et al., 2024) have demonstrated that sufficiently large transformer models, when pre-trained on offline interaction sequences, can approximate near-optimal online reinforcement learning algorithms such as LinUCB (Li et al., 2010) and Thompson Sampling in multi-armes bandits (Agrawal and Goyal, 2013). as well as UCB value iteration for tabular Markov decision processes (Azar et al., 2017). In addition, transformers have been shown to effectively handle non-stationary RL environments, achieving near-optimal performance by minimizing dynamic regret (Chen et al., 2025).
standard results from Neural Tangent Kernel (NTK) theory: (i) Kernel constancy: along training, the NTK remains (asympiotically) constant (i.c., it converges to a deterministic kernel independent of the training step); (ii) GP limit of the predictor: the trained predictor converges to the Gaussian process induced by that kernel. Result (i) has been established for transformer architectures via the tensor programs framework (Yang, 2020). By contrast, general proof of (ii) for transformers is not yet available: however, extensive empirical evidence supports Gaussian process behavior in large-width networks (Malladi et al., 2023). Accordingly, the principal theoretical gap in our analysis is a formal proof of (ii) for transformers, which is challenging problem that we leave as future work. Nonetheless, these assumptions align with existing theory and are corroborated by the strong empirical performance of our method, which together provide credible justification for applying our theoretical insights to transformer- these ideas and existing results, we could potentially extend Proposition to transformer architectures; however, this is beyond the scope of the current paper and is therefore left for future work.
Our analysis relies
LLMs. Equipped with
A.3 ADDITIONAL ANALYSIS ON THE COMPUTATIONAL COMPLEXITY OF ACTIVEDPO
We provide a theoretical calculation of the computational complexity and memory requirement
for ActiveDPO.
Assume that we have n number of prompts with m number of responses, the number of parameters is k and the projection dimension is d. Calculating the gradient for all the data points requires Onn?k). Projecting all the gradients to dimensions requires O(nm²kd) and hence a total of O(nm²kd) kd). Assume that we have selected s number of data points. The calculation of gradient and projection for these s number of selected data points with the new model is O(skd). Calculating our acquisition function for all data points is O(rm²d + d²).. Therefore, for each iteration, we have the complexity of Ommm² ? dd + d + skd + nm km). Note that the projection dimension is controllable and a hyperparameter, and m is chosen to be small in most applications. Therefore, the overall computational complexity is small. The memory requirement is mainly dominated by storing the projected gradient, which is O(mm2d) and is again reducible by reducing and m.
Asymptotic vs. practical complexity. We note that the asymptotic complexity analysis above does not directly reflect the practical wall-time of different operations. practice, the dominant computational cost of Acti veDDO comes from the forward and backward passes through the LLM to compute gradients for each data point, rather than the random projection step. Specifically, while the random projection has higher asymptotic complexity O kd), the actual wall-time is dominated by the gradient computation through the LLM, as modern GPU-accelerated matrix multiplications make the projection step relatively fast.
14

---

Published as a conference paper at ICLR 2026
Computational cost comparison across selection strategies. We provide comparison of the computational overhead of different selection strategies in Table 1. Random selection has negligible selection cost. APO (Das et al., 2024) requires computing features from separate reward model and solving a linear algebra problem. resulting in moderate overhead. APLP (Muldrew et al., 2024) requires forward passes through the L.LM to compute reward estimates for all candidate data points. Our Act i veDPO requires both forward and backward passes through the LLM (using LoRA) to compute gradients, followed by random projection and the selection criterion computation. The additional computational cost of Act i veDPO is justified by its superior label efficiency, as the cost of human annotation typically far exceeds the computational cost of data selection.
Table 1: Comparison of computational overhead per iteration for different selection strategies. n. number of candidate data points; k number LoRA parameters; d: projection dimension; B: batch size.
Method
Selection Cost
Extra Storage
Random
((1) Onnk + B²B)
None
APO
((nk)
APLP
(n) forward passes
(
ActiveDPP (Ours)
O (nkd - ndf + dc)
O(nd)
A.4 COMPARISON WITH EXISTING WORK
Differences with APLP (Muldrew et al., 2024). APLP selects data based on the reward difference between two responses, i.e. the absolute value of the difference in estimated rewards |ra (z, V1) (z, 92)l. While intuitive, this criterion only captures the current model's confidence in ranking the two responses and does not account for how labeling particular data point would improve the model after a parameter update. In contrast, our Act i ivDDPO uses the gradient difference ||Vre._, (z. y1 Vra. 2 (2,, 32)llv as the selection criterion, which is inherently more forward-looking the gradient captures the direction and magnitude of the parameter change that would result from training on data point. By selecting data points that maximize the gradient-based uncertainty, Acti veDPO selects data that would lead to the largest potential improvement the reward model after parameter update, rather than simply identifying data points where the current model is uncertain. Furthermore, APLP's diffeerreee criterion can be problematic as training progresses: when the reward model becomes more accurate, data points with large reward differences are likely to be those with correct but highly confident predictions, which provide diminishing returns for model improvement. This partially explains the performance degradation of APLP in later iterations observed in our experiments (Fig. I). Our gradient-base criterion avoids this issue since it measures informativeness through the lens of the model's parameter space, which remains meaningful regardless of the model's prediction confidence.
Additionally, the V;= matrix in our selection criterion acts as diversity regularizer that accounts for the information already gathered from previously selected data points. This ensures that newly selected data points provide complementary information, property that APLP's selection criterion does not explicitly enforce.
A.5 ADDITIONAL EXPERIMENTAL RESULTS
Fig. 4 shows the win-rate of different selection strategies. In general, our ActiveDPO still outperforms other selection strategies in the last few iterations.
A.6 PROOFS FOR PROPOSITION
Define the following objective function:
Mogo(re(Va)) (x) re(m x)]] + A| - 0
L(Ø)- =
Σ
(5)
2
(x.
15

---

Published as a conference paper at ICLR 2026
Random
Random
Random
APO
Winrate
ActiveDPO
Wine
0.60
0.65
ActiveOPO
0.7
ActiveDFO
0.55
0.6
0.5
0.50
Iteration
Iteraion
Reeation
(a) TL.DR with llama-2
(b) TT ILDR with Gemma
(c) TLDRR with Qwen3-4B
0.80
Random
Random
APO
APO
Wireete
0.75
ActiveDPO
Win-rtte
0.65
ActivedPO
Winrets
0.7
0.60
Random
0.65
0.5
0.60
0.55
APLP
veDPO
Beertion
Eeration
Reration
(d) WchGPT with Llama-2
(c) WebGPT with Gemma
(f) WebGPT with Qwen3-4B
Figure 4: Comparison of the win-rate of the responses generated by the LLM trained by DPO with the responses generated by the initial L.LM with different selection strategies.
Define H as the NTK matrix following the same definition in Verma al. (2025). Define Vr following the same definition Verma et al. (2025). Define as the size of the selection dataset D in each round.
We make the following assumption:
Assumption 1. Assume that
the reward function is unknown, non-linear, and bounded function,
Ky = yy (r(x,, 81) (r(xxx(z))))
the reward function is bounded: |r(x, v)1 S 1, Va E K.yeV.
there exists λ > S.t.H 2 AgI, and
the reward function function takes a vector 2 (which is the representation vector for the concatenation of x and y) as impul and z satisfies: ||=||2 and [=]] [#lj+d/2 all x E X and y Y.
Denote (1, U1) = 2 211(2, M1 32) 32) where ς(x v1. = 1 (Vre, (x, v1)
7m
Vro, y2)) and V.- EE 2. w2 plz, y1, 32) vı, v1,42) following Lemma, which is direct extension from Theorem
AI. We give the F. Verma et al. (2025):
Lemma 1. Güven that Assumption holds, let E (0,1), E,, = Cm-1/6 Vlog ( for some absolute constant C > As long as m poly(T, L, K, 1/Kp, 1/da, 1/1, log(1 (og(1/8)) then with
probability of at least 1 8,
(r, re. (a, y2)] - [r(æ, r(æ, 82)] < (T 01. + Em.
for all x E X and 92 E V,t [T' when using the objective defined in Eq. (5) to train this reward function T0.-1"
Proof. This Proposition is immediately true by concatenating the prompt and response replace the imput used in Theorem of Verma et al. (2025) and instantiate the link function in Verma et al. (2025) the sigmoid function. Specifically, assume that the reward takes the representation
16

---

Published as a conference paper at ICLR 2026
vector of the concatenation of and y input and assume that this satisfies the corresponding conditions in Assumption
Denote ( 1/1 V2) A 11 V1, 12))lll where Pe- 91 91, Mi Vro, (2. 32)) and V1-1= - Ep Ex.n -22- D3 9t- (2t, v1, 92) (z, 91, ./2) + Myp
- (Vro,- 1 (x, 91) vm
AI
Lemma 2. Given that Assumption holds, for some absolute constant > 0, we have that: 1/6 v2) 0c-1 91 82)1 - CA 5/ 1)4/3m m/ Vlog mL 9/2
10:-1 (2, V1.
(6)
Proof. Following the proof of Lemma B.4 in Zhang et al. (2021), we can show that
00-1 (Ir, v1, 92)
o (x, 1/1.922 92)l
ro (z, v1)
ro:-1 (Ir (2)
rep (I. v1)
rBu (z, 1/2)
Vm
vmn
C
ro (Ti, Ma.) ro, (I 16.2) Too (2 Hi,1) - To (F 31.2)
Vm
Vm
for some absolute constant C > addition, according to Lemma 3 of Verma et (2025), we have that
1-1
1/3
[|ro (2., v) ro (2, 1/3 /log
1
Ye WEX.yED,
Consequently, we have that
lor- (2, M1, 82) 1 M1. ;1.
L
1/3
0²
Vx (1 - 1) x x
x C/m 1/3
Vog m
1,7/2
Vm
Cx - 1) -1/6
1/6
Vlg
for some absolute constant C 0.
The result of Lemma says that as long as the width m of the NN is large enough, we can ensure that the difference (Fr,)) ( 21,2) is upper-bounded by small constant. Consequently, can show the following formal version of Proposition 1.
Proposition (Formal version of Proposition 1). Given that Assumption holds, let E (0, 1), Em,t Cm-1/6 //og/ 3logmL3 ( + CA 55/ 1)4/3m - 1166 Vlog for some absolute 1/66
constant C 0. As long as poly(TT L, K, 1/mp, 1/>o, 1/,, log(1/8)), then with probability of at least 8,
[ro_ v1) T8.., (x, y2)]
[r(x, 91) r(z, y2)l Pror-1 V1, V2) + Em.t
for all x E X and y1, 82 V,, [T] when using the objective defined in Eq. (5) to train this reward function T0--1"
Proof. Combining Lemma and Lemma 2, we get that the proposition is true.
Remark 1. Note that the objective function of Eq. (5) is almost the same as Eq. (5), with Eq. (5)
scaling the Eq. (5) by constant and having additional regularization term. The design of Eq. (5) is for the theoretical results. Empirically, we still use the standard Eq. (2) and adjust the regularization by adjusting the 3 in Eq. (2).
17


Reviews:
[{"review_id": "XPmkvk9KB4", "reviewer": "Reviewer_u5Vv", "summary": "This paper introduces ActiveDPO, which integrates active learning into DPO for reinforcement learning from human feedback (RLHF). The key innovation is the linearization of the DPO objective at the policy’s last layer, enabling the application of D-optimal design to select the most informative feedback—either online (ADPO) or offline (ADPO+). The authors prove that the logit error decays as $O(d/\\sqrt{n})$ and validate their algorithms on both synthetic log-linear models and large language models (Llama-3.2, Phi-3) using the Nectar dataset.", "questions": "- Refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- First theoretical and algorithm formulation of active learning for DPO.\n- Provides algorithm for both online and offline settings, enabling flexible application.\n- Validate the theoretical foundation with reasonable empirical results.", "weaknesses": "- Reliance on a log-linear policy approximation may lead to shallow alignment, easy reward-hacking and neglected task complexity.\n    - The paper assumes (Assumption 1) that *all policies* are log-linear in the last layer features. This assumptions is the key assumption that enables the D-optimal design analysis, but concurrently restricts the model's expressive capacity.\n    - In realistic settings, such as aligning LLMs, the relation between prompt/response pairs and human judgements is likely nonlinear.\n    - Therefore, ADPO may select prompts that maximize separation in the linear space, but those may correspond to shallow differences (e.g., length of the reponse).\n    - This concern could be releaved by providing analysis of how *linear* each tasks are.\n- Evaluation scale\n    - The LLM experiments are relatively small; and conclusions about the practicality of ADPO in largs-scale PO remains unclear.\n- The comparison of the Active DPO baselines in the perspective of computational overhead is needed.\nI am willing to further raise the score if my concerns are addressed properly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50mACVvqDR", "reviewer": "Reviewer_ubRE", "summary": "The paper proposed a new objective to conduct active preference learning in the LLM alignment task. The proposed method used something similar to the influence function, which is a intuitive choice.", "questions": "See the 2/3 points above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The active preference learning research topic is very important. I like the method which is similar to the influence function to conduct active learning. The writing is clear and the scale of the experiment is OK.", "weaknesses": "**1. Discussion on Comparison with Active Preference Learning for Large Language Models**\n\nA detailed comparison with Active Preference Learning for Large Language Models (arXiv:2402.08114\n) would strengthen the paper. In particular, while the APL paper focuses on reward difference—which only reflects the immediate step before updates—the current work’s use of gradient difference captures the potential improvement after updates. This distinction highlights a more forward-looking and theoretically meaningful criterion. Including a discussion that formally articulates this advantage would enhance the theoretical depth of the paper.\n\n**2. Evaluation of Win-Rate Over More Iterations**\n\nIt would be valuable to present win-rate comparisons over a larger number of iterations to examine how performance evolves as active learning progresses. Since the proposed approach operates in an online learning setting, it is important to investigate whether performance degradation occurs due to catastrophic forgetting. Prior work suggests that active learning strategies may sometimes exacerbate forgetting compared to random sampling, so empirical results here would be insightful.\n\n**3. Impact Study on Batch Diversity Encouragement Term ($V$)**\n\nThe inclusion of the batch diversity encouragement term, $V$, is an elegant design choice that facilitates batch updates while promoting sample diversity. However, an ablation or impact study isolating the contribution of $V$ would help clarify its empirical importance and guide future design choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mi2m9gxzCC", "reviewer": "Reviewer_PcQD", "summary": "This paper develops an active learning approach for training models with direct preference optimization (DPO). Grounding their choice in the theory around dueling bandits, the paper develops a gradient-based uncertainty criterion which is used to select prompts for human labelers to consume. The method has significant computational and memory requirements: several mitigations are developed in order for the method to be deployed tractably. The method is evaluated on several different data sets and shown to help perform similarly to active learning approaches for DPO.", "questions": "Empirically, what is the amount of flops needed to run the various different selection strategies and how do they compare with the flops required to compute the gradient and do the forward/backward passes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The motivation of the Active DPO method is theoretically grounded and principled.\n+ The implementation details which allow the method to become tractable are quite clever and dramatically improve the tractability of the method.\n+ The ablations are quite in-depth and demonstrate effectively which parts of the algorithm are important; and show that their design is robust to various different models and datasets.\n+ The Active DPO method appears to outperform all other Active DPO approaches in the experiments and on some experiments significantly outperform the random baseline.", "weaknesses": "+ The description of the algorithm is slightly unclear at times. Particularly around the description of the matrix V_t. This is presumably an outer product of the gradients, but the authors don't comment on the fact that this is obviously intractable to store for modern LLMs, let alone invert. I appreciate that the matrix is tractable when projected to 8192 dimensions with the approximations made later on, but the authors should highlight this difficulty earlier on.\n+ The computational requirements of this approach are pretty large, even with the author's mitigations. As far as I understand from the paper, ActiveDPO requires computing the gradient on every example T times, and so (assuming that the fwd/bwd pass is the bulk of the computation, and that the alternative is optimization with T steps, computing the gradient on each example only once), ActiveDPO will take many more FLOPs and wall-time than random selection. The authors don't address this point very much, and it would be very interesting for them to provide the results from figure 1, but normalized by FLOPs instead of iteration. With this amount of extra computation, a practitioner could even potentially even use a method like GRPO, training a separate reward model on random samples, and potentially even get better results than using ActiveDPO. \n+ The computational complexity section in the appendix is overly simplified and even a bit misleading. For instance, the authors say that the projection of the gradients to d dimensions is going to dominate the model computation of the gradients, but this is not likely to be very true given the small size of the LoRA gradients vs. the large number of FLOPS required to compute the full gradient with a full forward or backward pass.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGsrX7Wwaq", "reviewer": "Reviewer_GSMF", "summary": "This paper aims to develop an active preference data selection algorithm.\nThe authors use the gradient information to estimate the difference between the current model's reward and the ground-truth, i.e., the uncertainty of the model.\nBased on this, they propose to select data that maximizes the uncertainty for annotation.\nThe authors further introduce batch selection and random projection with LoRA gradients to reduce computational cost.\nEmpirical results on several models and datasets demonstrate that the propsed method outperforms baselines in terms of trained models' reward.", "questions": "1. $V_{t-1}$ is an important term in the proposed method, but there is little explanation about its meaning. Can you provide more details about it? \n2. How does the proposed method compare with full data annotation (i.e., no selection) in terms of computational cost and performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Theoretical Grounding**: The proposed method is well-grounded in theory, with clear derivations from the estimation error in reward modeling to the data selection criterion.\n2. **Practical Considerations**: The authors address the computational challenges of the proposed method. These practical techniques are useful and insightful for applying gradient-based data selection in real-world scenarios. The authors also provide in-depth ablation studies to validate the effectiveness of these techniques.", "weaknesses": "1. **Training and Evaluation Settings**: The preference dataset is annotated by small reward models (OpenAssistant/reward-model-deberta-v3-large), and the evaluation metric is also this reward signal.\nHowever, a more convincing and up-to-date training and evaluation protocol is to use more powerful RMs (e.g., ArmoRM) for annotation and alignment benchmarks (such as AlpacaEval, Arena-hard, etc.) for evaluation aside from only the RM score [1].\n2. **Computational Cost**: Although the authors propose several techniques to reduce the computational cost, the method still requires significant resources due to gradient computations. \nFor example, the gradient computation may outweigh inference cost of an external RM for data selection.\nThe authors should at least provide a more detailed analysis of the computational overhead compared to baseline methods and annotation time, since the goal of active learning is to reduce the training cost. \n\n[1] Meng, Yu, Mengzhou Xia, and Danqi Chen. \"Simpo: Simple preference optimization with a reference-free reward.\" Advances in Neural Information Processing Systems 37 (2024): 124198-124235.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)