RLTHF: Targeted Human Feedback for LLM Alignment
Authors: Yifei Xu
Tusher Chakraborty
Emre Kiciman
Bibek Aryal
Srinagesh Sharma
Songwu Lu
Ranveer Chandra
Venue: ICML 2025 poster (2025)
Keywords: RLHF
Reward Modeling
Primary area: deep_learning->large_language_models
TL;DR: RLTHF is a human-AI hybrid framework that iteratively refines reward model alignment by leveraging LLM-labeled data and strategic human annotations, achieving oracle-level performance with minimal human effort.

Abstract: Fine-tuning large language models (LLMs) to align with user preferences is challenging due to the high cost of quality human annotations in Reinforcement Learning from Human Feedback (RLHF) and the generalizability limitations of AI Feedback. To address these challenges, we propose RLTHF, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations to achieve full-human annotation alignment with minimal effort. RLTHF identifies hard-to-annotate samples mislabeled by LLMs using a reward model's reward distribution and iteratively enhances alignment by integrating strategic human corrections while leveraging LLM's correctly labeled samples. Evaluations on HH-RLHF and TL;DR datasets show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort. Furthermore, models trained on RLTHF's curated datasets for downstream tasks outperform those trained on fully human-annotated datasets, underscoring the effectiveness of RLTHF.

RLTHF: Targeted Human Feedback for LLM Alignment
Yifei Xu 12*
Tusher Chakraborty'
Emre Kıeıman' Ranveer Chandra'
Bibek Aryal
Srinagesh Sharma' Songwu Lu
Abstract
preferences, including intent, style, grounding, and compli- ance requirements (Atreya, 2024; Microsoft, 2024; Sharma, 2024; AI, 2024). A key approach to achieving this align- ment is Reinforcement Learning from Human Feedback (RLHF), which has emerged as widely adopted technique in the literature for refining model behavior based on human feedback (Bai et al., 2022a; Stiennon et al., 2020; Rafailov et al., 2024; Wang et al., 2024a; Ouyang et al., 2022).
Fine-tuning large language models (LLMs) to align with user preferences is challenging due to the high cost of quality human annotations in Reinforcement Learning from Human Feedback (RLHF) and the generalizability limitations of Al Feedback. To address these challenges, we pro- pose RLTHF, human-AR AI hybrid framework that combines L.LM-base initial alignment with selec- tive human annotations to achieve full-human an- notation alignment with minimal effort. RLTHF identifies hard-to- sarr-too-anottan annotate samples mislabeled by LLMs using a reward model's reward distribution and iteratively enhances alignment by integrat- ing strategic human corrections while leveraging LLM's correctly labeled samples. Evaluations on HH-RLHF and TL;DR datasets show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort. Furthermore, models trained on RLTHI's curated datasets for downstream tasks outperform those trained on fully human-annotated datasets, under- scoring the effectiivness of RLTHF.
The effectiveness of RLHF techniques heavily depends on high-quality human annotations, which are both costly and time-consuming to obtain (Pang et al., 2023; Lee et al., 2023; Wang et al., 2024a). To mitigate this challenge, Reinforce- ment Learning from AI Feedback (RLAIF) has been intro- duced, leveraging LLMs to replace human annotators the feedback loop (Lee et al., 2023; Lee et al.; Bai et al., 2022b). While RLAIF can approximate human judgment to some extent, it is sensitive to factors such as prompt optimization, task complexity, model bias, cenerator-discriming and the capability of the judge model, limiting its ability to fully replicate human annotations (Huang et al., 2024; Sharma et al., 2024; Lee et al., 2023; Zeng et al., 2024; Huang et al., 2023). Our evaluation also provides evidence of these limitations. Furthermore, the samples that chal- lenge a judge model are often the ones most critical for adapting base models to specialized fine-tuning tasks (Etha- yarajh et al., 2024; Yuan et al., 2024; Huang et al., 2023), The cost of human annotation is further exacerbated by pri- vacy and security constraints that restrict fine-tuning service providers' access to an entire customer data corpus. In such cases, only subject matter experts (SMEs) within the cus- tomer organization have full visibility into the data, making it particularly difficult to optimize prompts effectively across the entire corpus, especially for hard-to-annotate samples.
gap,
1. Introduction
In recent years, large language models (LLMs) have demon- strated remarkable advancements, unlocking new possibil- ities across a wide range of applications (Touvron et al., 2023; Jiang et al., 2024; Achiam et al., 2023; Team et al., 2023). As these models become more powerful, the fo- cus has shifted toward customization, i.e., fine-tuning base models better serve specific tasks and user needs (Wei et al., 2021; Li et al., 2023a). Companies are increasingly investing in solutions built upon fine-tuned models, recog- nizing the value of adapting L.LMs to align with end-user
To address these challenges, we propose Reinforcement Learning from Targeted Human Feedback (RLTHF), a human-Al hybrid solution that combines coarse initial align- ment using general purpose L.MSs with the progressive integration of strategically selected human annotations achieve annotation quality comparable to fully human- supervised approaches. RLTHF begins with an initial align- ment stage, where general-purposs purpose LLM labels unlabeled data based on high-level instructions. While this approach effectively captures broader human alignment for easier data
general-purpose
'Wor done during internship at Microsoft. 'Microsoft University of California, Los Angeles. Correspondence Tusher Chakraborty
to: Yifei Xu <yxu@es.ucka.oo <tusher.chakraborty microsoft.com...
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RLTHF: Targeted Human Feedback for LLM Alignment
Anpotation
Uniiia Aligomet
Merative Algnment
Improvement
Sample Hard Samples
Ampllly
Shard N Samples [[=,, Man: Sta .M0
Reward Difference
Sharding
General Purpose LLM
Improved
Aligned LLM
Incorrcct Semples
Fiip
PPO DPO
Unarnotated Data
Al-annatated Data (au Mer
Reward Modeling
Aligned Data
Rerank
Correct Samples
Reward Modaling (t=0)
Annotate:
Next lterationn
RM
RM
Figure 1: Overview of RLTHF process. RLTHF starts with coarse LLM alignment on the task. It then iteratively takes targeted human feedback and blending the same with sanitized LLM-annotations reach the complete human alignment, leveraging reward distribution of reward model in its training dataset.
points, it often struggles with fine-grained nuances, leading to incorrect labeling. RLTHF automatically identifies these hard-to-annotate data points and directs human effort exclu- sively toward them. This targeted approach enables RLTHF to achieve the quality of fully human-annotan data while reducing the majority of human annotation effort.
datasets: HH-RLHF and TL;DR. Our results demonstrate that RLTHF achieves accuracy comparable to fully human- annotated dataset while requiring only 6-7% of the total human annotations. Furthermore, we conduct a compara- tive study by training models on downstream tasks using DPO (Rafailov et al., 2024). Remarkably, models trained with RLTHF even outperform those trained on fully human- annotated datasets, highlighting the impact of RLTHF's meticulous data curation in enhancing model performance.
To enable this efficient human-in cosuun-i-tthe-looe loop approach for achieving comprehensive human alignment, RLTHE intro- duces the following key technical contributions:
First, we introduce concept that leverages the reward distri- bution of reward model over its training dataset to capture the relative arrangement of samples based on rewarded fea- tures. This distributional property enables the identification of both potential annotation inaccuracies and the model's confidence across the dataset. Specifically, train reward model on the LLM-labeled dataset to uncover clusters of hard-to-annotate samples that are highly likely to be misla- beled by the LLM, as well as easy samples that are highly likely to be correctly labeled.
2. Background and Related Work
LLMs have demonstrated impressive performance across a wide spectrum of tasks (Achiam et al., 2023; Dubey et al., 2024; Team et al.. 2023). Despite the progress, their perfor- mance on eustomized downstream tasks can be significantly optimized by supervised fine-tuning (SFT) with instruction and human- written responses pairs (Chung et al., 2024; Thoppilan et al., 2022). Reinforcement learning with pref- erence data has further shown success due to the easier-to- collect data form (Ouyang et al., 2022; Stiennon et al., 2020; Lee al., 2023). Representative methods include Proximal Policy Optimization (PPO) (Schulman al., 2017), which optimizes the LLM with separate reward model, and Di- rect Preference Optimization (DPO) (Rafailov et al., 2024), which directly learns from the preference data. Although an easier data collection is available, these methods still largely rely on the richness and quality of the preference data (Xu et al., 2024; Zheng et al., 2023b; Wang et al., 2024a).
Building on this concept, we propose an innovative itera- tive reward model training technique to achieve oracle-level human alignment in the dataset. In each iteration, RLTHF identifies highly probable mislabeled data points and recti- fies the same using human annotations. Simultaneously, it detects clusters of samples that are very likely to be correctly labeled by the LLM and incorporates them with human- annotated data to construct a high-quality training set for the next iteration of reward model training. Throughout this process, RLTHH preserves data richness and maximizes the efficiency of human annotation investment through carefully controlled hyperparameters.
2.1. Alignment with External Feedback
Human feedback is regarded as the golden standard in LLM alignment. However, reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Stiennon et al.,
Finally, we evaluate RLTHF on two distinct preference
2

---

RLTHFF Targeted Human Feedback for LLM Alignment
2020; Köpf et al., 2024) typically incorporates heavy and expensive crowdsourcing efforts expert annotations to guarantee data diversity and richness. To relieve the reliance on human effort, reinforcement learning with AI feedback (RLAIF) (Lee et al., 2023; Bai et al., 2022b) provides an alternative that collects feedback from stronger LLMs in- stead of humans. On the other hand, this method is limited by the capability of the stronger LLM annotators (Huang et al., 2024; Sharma et al., 2024; Lee et al., 2023) especially for customized tasks, and suffers from their intrinsic bi- ases (Zheng et al., 2023a). In this paper, we take advantage of RLAIF to establish an initial alignment and strategically incorporate human feedback to efficiently bring LLMs to the true alignment.
3.1. Initial Alignment
This stage aims to establish an initial coarse alignment in the unlabeled dataset using a general-purpose vides preference annotations for each unannotated sample. Prior research suggests that model selection here depends on task complexity relative to the model's capability (Snell et al., 2024). While RLTHF is not found to be sensitive to the choice of model at this stage, a well-suited model can accelerate alignment convergence. The only assumption is that the general- purppss L.LM possesses basic understand- ing of the downstream task, enabling it to provide rough initial alignment that serves as seed for RLTHF.
purpose LLM, which pro-
Our prompt for obtaining preference judgments from the LLM consists of three components: 1) task description, 2) preference judgment principles provided by the user, and 3) few-shot examples with optional thain-of-thhuglee reasoning. The prompt templates are detailed in Appendix A. We do not perform explicit fine-grained prompt tuning, as full visibility into the data may be restricted when offering fine-tuning services to third-party customers, However, to ensure that the selected LLM with our prompt attains a rough level of alignment, we perform an eyes-off validation using strategic human feedback, as detailed in Section 3.2.3.
2.2. LLM Self- Self-Improvement
To break the upper bound of LLMs, recent efforts have been devoted to enabling LLMs to self-improve. Self-Rewarding LMs (Yuan et al., 2024) and Math-shepherd (Wang et al., 2024b) demonstrate the possibility of LLM self- improvement with reward signals from itself. SELF- ALIGN (Sun et al., 2024b) uses a carefully written set of principles to guide LLMs through self--mprovement. SER (Huang et al., 2024) starts with only fraction of hu- man annotations to achieve full-annotation performance by progressively generating additional training data for itself. However, these methods still suffer from the intrinsic upper bound of LLMs and self-improvement is not guaranteed for customized tasks. RLTHH, on the other hand, efficiently in- troduces human intelligence into the improvement process, thereby ensuring that the improvement is not bounded by LLMs' initial lack of domain understanding.
As mentioned earlier, this Al-generated feedback is prone to errors due to factors such as model biases from pre-training data, task complexity, and prompt optimization, which is also evident in our evaluation. When our ultimate goal is to customize an existing model through fine-tume aee-tunnng to align with end-user preferences, we inherently assume that an off- the-shelf LLM lacks comprehensive alignment with the end- user. However, RLTHF builds upon the initial Al- Al-provided alignment and systematically refines it in subsequent stages to achieve oracle-level human alignment.
3. Improving Human Alignment with RLTHF
RLTHF enhances alignment with human in preference datasets used for training preference optimization tech- niques like DPO and PPO. It facilitates LLM training for various downstream tasks, including summarization, com- pliance, and grounding. Starting with an unlabeled prefer- ence dataset, RLTHF strategically integrates Al Al-generated labels with selective human feedback to maximize align- ment while minimizing annotation effort. As illustrated in Figure 1, RLTHF operates in three stages: 1) Initial align- ment, where an off thi-thh-shelf LLM provides dataset labeling to establish coarse task understanding, 2) lterative align- ment improvement, which leverages reward distribution by reward model (RM) to locate hard-to-annotat samples mislabeled by the LLM and rectify with selective human feedback while investing the correct LLM labels, 3) Trans- ferring knowledge for downstream task, where the curated preference dataset is fed into the DPO pipeline or the trained RLTHF reward model is integrated into the PPO pipeline. Find the corresponding pseudocode in Appendix B.
3.2. Iterative Alignment Improvement
In this stage, we refine the LLM- preference dataset by iteratively training an RM with selective human annota- tions to enhance alignment. Before diving into the details of this process, we first establish the premise for RM.
1.2.1. REWARD MODEL
Given a labeled preference dataset DA {Z er.er } where E [N], z; is the prompt, Vi.e and yi.r denote the chosen and rejected completions, respectively, as labeled according to the annotator's 's preference. A. Here, if we represent the relative preference orientation of jth comple- tion pair with À = +1], A is a N N-dimensional vector consists of [>,][] meaning that flipping the preferences of all completion pairs results in D-A To train an RM on this dataset, we can formulate the probability distribution of yie being preferred over Verr given 2; an input, following the
3

---

RLTHFF Targeted Human Feedback for LLM Alignment
Bradley-Terry (BT) model (David, 1963).
features of these samples are highly conflicting with other stronger preference features learned in 's, leading the RM to penalize them.
P(x > y) (r(2,, Ut.e) - r(x;: Vi.r))
(1)
where (-) denotes the sigmoid function and r(-) denotes the reward function. Assuming the existence of a true deter- ministic reward function, the goal is to train the RM to learn this function and predict the reward, f(zz, y). The RM train- ing can be framed as binary classification problem (Sun et al., 2024a), where labeled pair of Pt.c = ( Vi,e) and Ps.r == ( Vs.r)) is passed to the model to predict the condi- tional class probability according to Eq. 1. This leads to the negative log- likelioodoo loss function for training.
3.2.3. RLTHF LEVERAGING REWARD DISTRIBUTION
RLTHF trains the initial RM using a preference dataset la- beled by a general general-purpose purpose LLM from the previous stage. We denote this dataset as D DALLM where ALLM represents the LLM's preference labeling. Since the RM training in- cludes a validation set derived from DALLM. this ensures that the trained RM is broadly aligned with the LLM's preferences. We assume that the LLM has a coarse but rea- sonable understanding of preference judgments, particularly for relatively mas--to-annotae samples. As a result, the fea- tures of these samples dominate in FALLM. Based on our earlier discussion, the upper left region of the reward density curve, 0( ALALL e ALLLM) contains high density of samples with prominent preference features, i.e. those that are easier for the LLM to annotate accurately. Before proceeding, we further validate that the LLM is at least roughly aligned with the user in terms of these easy-to-annon to-nnotttee samples. This step mitigates the risk of significant misalignment due to prompt curation or model selection. To achieve this, RLTHF automatically (details in the following section) samples a small subset (< 0.1%) of preference data from the upper left region and gathers user feedback. If human agreement on these samples is low, it signals a major misalignment between the user and LLM. While we did not observe such cases in our experiments, this issue can be addressed by refining the judgment principles in the prompt. Updates can be directly made by the user, through incorporating verbose user feedback, or even through automated prompt optimization techniques (Kepel & Valogianni, 2024).
[(]) = - ((.,))) flog o (f(ps..) - [(p.r))
(2)
In essence, during the RM training, we pass a preference pair fpres pt.r. labeled as Pi.e winning over Pt.r according to the annotator's preference A. Provided sufficient preference samples in dataset, the RM learns the winning preference features of the data that determine the winner in a pair, captured in the reward function 'A-
3.2.2. LOOKING AT REWARD DISTRIBUTION
At this stage, we analyze the distribution of the predicted reward function (fx) within the training preference dataset DA. For each labeled preference pair {pi.e: }. we com- pute the reward score difference as AAÑA (p.) (A(P))) It is important to note that A quantifies the rela- tive preference score of a given pair in alignment with the annotator's ' preference orientation A, satisfying the property AAT = - -A-Ar. By ranking all preference pairs in D> based on AAAA. a monotonic reward distribution curve, de- noted as 0( (fArA)) ), emerges. This distribution, as depicted in Figure 2a, provides insight into the model's reward as- signment across the dataset, though for the moment, the legend in the graph can be disregarded.
At this stage, we can identify regions with a high density of correctly labeled samples by the LLM, i.e., those that are relatively easy for the LLM to annotate in alignment with human preference. Now, we turn our attention to two crit- ical types of samples necessary for achieving fine-grained alignment: (1) hard-to-annotate samples and (2) samples mislabeled by the LLM w.r.t. the human preference An. Since the LLM was unable to correctly label these samples initially, the reward function 'ALLM cannot accurately cap- ture their preference features. Consequently, these samples are expected to cluster around the bottom right region of the reward distribution curve O( (AAALLM trate this, we refer to Figure 2a and 2b. Figure 2a shows 0( ((AALLseAALA from one of our experiments. In this figure, we classify each sample Pi E DALL as either cor- rectly or incorrectly labeled w.r.t. the human preference An. i.e., whether the preference assigned by ALLM is matching An. As observed, the upper left region of the curve con- tains a high density of correctly labeled samples, supporting our earlier claim that these represent the LLM's easy-to-
The reward distribution curve (() derived from the training preference dataset DA reflects the degree of align- ment the RM (trained with optimal validation loss) has achieved across DA during training. The upper left region of the curve consists of samples with high positive A f indicating strong agreement between the RM and the train- ing preference labels A. This suggests that the RM effec- tively identifies and reinforces strong winning preference features these samples, implying that these features were dominant in DA. Conversely, the bottom right region of the curve contains samples with very low or even negative AAA signaling disagreement between the trained reward function A and the training preference labels for these sam- plcs. This misalignment arises from two primary factors. (I) Absence of strong features, where RM is not able to find any strong preference feature in these samples according to AA- (2) Conflicting samples within DA, where the preference
To illus-
4

---

RLTHEF Targeted Human Feedback for LLM Alignment
100
Cameci Insernet Koes bllon
Correct
Reaay Oeferene 0
Kreea
Intomnect
0.7
Tonvert Differre
00-
Recusace
5.73
40
0.00
1.03
code 106001 sew.://oeeooiieesiinnnooo
1002812200000000900000 355ce
s3tD 1000 2tt.:///2000200eo.00o 20000
Konk
Renk
Ranc
kank
(a) Reward dist. Itr-O
(b) Correctness dist. Itr-O
(c) Reward dist. It--5
(d) Correctness dist. Itr-5
Figure 2: Reward (assigned by a trained RM) and correctness (w.rt. human preference) distribution curves for the very first and last iterations of RLTHF. These two types of curves provide the intuition of strategically selecting the samples for efficient human annotation towards improving alignment in the dataset. These curves further highlight the iterative refinement process, showing how alignment in the dataset progressively improves.
annotate cases. To quantify this, we generate an accuracy density curve for 0( A ALLM) w.r.t the human prefer- ence An. as shown in Figure 2b. This figure confirms that alignment with human preference decreases as we move towards the right side of the curve.
3.2.5. ITERATIVE APPROACH
The current reward function FALLAr trained on DA ex- hibits an alignment gap w.r.t. An due to the presence of hard-to-annotate samples for the LLM and mislabeling by the LLM. Since we have identified ways to rectify these issues, we can refine D to improve alignment and train a new RM that better aligns with An- Now, the question is how to prepare the dataset for the next iteration of RM training? Suppose we are currently in iteration 0 (Itr-O) with DALLM and FALLM. For the iteration 1 (Itr- 1) train- ing dataset, DT A our primary goal is to include high- confidence samples that are well-aligned with An The first choice is definitely human annotated samples from Itr-O. Ad- ditionally, another set of candidates can be drawn from the high-accuracy density region of 0 (A ALLM) ) specifi- cally the region to the left of the "elbow", where the RM has learned strong preference features in alignment with An
While we can observe how alignment with An varies across the reward distribution curve, in real-world scenarios, we lack ground-truth labels to quantify this accuracy directly. Therefore, we need to estimate the boundaries of key re- gions within the curve. To achieve this, we identify two strategic points: the "elbow" and the "knee", as illustrated in Figure 2a. These points correspond to sharp changes in A which we detect using the first-order deriva- tive. The "knee" marks the transition a region with lower accuracy density, whereas the "elbow" indicates shift to- ward higher accuracy density. It is important to note that these points serve only as rough estimations of the region boundaries rather than precise demarcations.
Although these two sets of samples offer high precision, DVr "ir,, will still face a data coverage issue. Looking at the An--
3.2.4. SELECTIVE HUMAN ANNOTATION
reward distribution curve, these two candidate sets represent samples with the longest distance, leaving gaps in middle re- gion. However, expanding the dataset by including samples from the middle region, i.c., right of the "elbow" and left of the "knee" risks introducing misaligned samples. Since the accuracy in this region is likely to be just above 50%, obtaining human annotations for these samples would be in- efficient. Furthermore, as the number of samples annotated from the right of the knee is relatively small, their prefer- ence features are likely to be overshadowed by the dominant preference features of the high numbers of left-side samples. As result, their features may not be effectively captured in f A To balance these trade-offs, we introduce two hy- perparameters, allowing for more controlled and effective dataset expansion while maintaining alignment quality.
To enhance alignment from this stage, human annotation is necessary, but it must be done efficiently to maximize its impact. A straightforward approach is referring to the accuracy density curve annotations in the lowest accuracy region would yield the highest benefit. Thus, we could start annotating from the very bottom of the curve. However, as previously discussed, some samples in this region may exhibit preference features that are largely opposite to the dominant features captured by AALLM These samples are highly likely to be mislabeled in ALLM (see Appendix C). Instead of seeking human annotation, we simply flip the preference of these samples to rectify. To estimate the location of such samples, we take the reflection of the "el- bow" point w.r.t. the x-axis, as the elbow marks the region containing strong preference features. This "reflection point" always lies to the right of the "knee" in the lowest accuracy density region. We begin human annotation at this "reflec- tion point" and proceed leftward along the curve, ensuring the most effective correction of alignment errors.
Back-off ratio (9): Determines how far to back off from the "knee" when selecting samples for the next iteration's 'ns dataset. A higher 3 results in a more sanitized dataset, reducing noise but at the expense of lower data coverage.
5

---

RETHFF Targeted Human Feedback for LLM Alignment
Amplification ratio (a): Increases the influence of human-annotated samples by repeating them in the
95
Test Accuracy (%)
Full Human Annotation
dataset, reinforcing their preference features in AAT. 1' However, an excessively high a may lead to overfitting to selective human annotations.
90
85
80
Full Set
1/4 Shard
The dataset Din.. consists of carefully selected sam- ples from DALLM ensuring high alignment with An by optimally tuning the hyperparameters a and B. Train- ing the RM on An results in "' which is more closely aligned with An- After training, we construct the dataset for generating the reward distribution curve by in- corporating the remaining samples from Itr-O: DA
75
GPT-40
1/2 Shard
1/8 Shard
0
12 Human Annotation %
20
(a) HH-RLHF
Test Accuracy (%)
Full Human Annotation Hnntainnnotation
90
DT A U (DAL.MM - DTT A From this, we generate new reward distribution curve, i0 P ). While this curve demonstrates improved alignment with An. full alignment is not necessarily achieved. However, it presents RLTHF with distinct reward distribution curve compared to the previous iteration. This evolving diversity in O(-) enhances the variety of human annotations, maximizing the return on annotation investments and incrementally enrich- ing D. Note that the effectiveness of this diversification, as well as the corresponding improvements, depends on factors such as hyperparameter tuning (see Section 4), the original data distribution, and model selection.
85
Full Set 1/2 Shard
Shard
80
1/8 Shard
GPT-40
8
12
16
20
Human Annotation %
(b) TL;DR
Overall preference accuracy improvement of RLTHF on test data in an iterative manner. Here, we exper- iment with different sizes of down-samples training data shards for RLTHF. We find 6% (HH-RLHF) to 7% (TI.:R) total human annotations on 1/4 shard yielding the optimal alignment for downstream tasks.
Figure 3:
RLTHF maximizes the efficiency of human annotations by iteratively refining d()) and exposing annotators to diverse, LLM-mislabeled samples. To further enhance annotation ef- ficiency, RLTHF employs random sharding to down-sample the original corpus. It begins by selecting a random shard of the dataset, iteratively improving alignment within that subset. Once the desired alignment is achieved, the final iteration's 's RM is used to label the entire corpus. This ap- proach enables RLTHF to concentrate human annotations in smaller, more targeted space while effectively propagating alignment across the full dataset at the end.
done in two ways: 1) incorporating the final iteration RM into the PPO loop, 2) labeling the whole dataset with the final RM and feeding the labeled dataset to DPO pipeline.
4. Results
In this section, we present the results of our main experi- ments, conducted on two datasets: HH-RLHF (Bai et al., 2022a) and TL;DR (Völske et al., 2017). Specifically, we compare RLTHF against three baselines: (1) Al-only label- ing, where samples are fully labeled by LLMs, (2) Random human annotation, where samples are randomly selected for human annotation (matching the number of human- annotated samples in RLTHF), with the rest relying on AI feedback, and (2) Human, where all samples are annotated by humans. A detailed description of our experimental setup is provided in Appendix D.
3.3. Reward Knowledge Transfer
RLTHF progressively converges toward the oracle human preference through iterative RM training and strategic hu- man annotation investment. As shown in Figure 2c and 2d, after five iterations, the reward distribution and accuracy curves closely align with the full-human annotation. Find the intermediate iteration curves in Appendix C. The re- quired number of iterations depends on the available human annotation and RM training budget. Notably, full-human alignment can sometimes be achieved before exhausting the annotation budget. In such cases, the samples selected for human annotation would largely lack distinct preference features, indicating that the model has effectively captured the human preference. Once desired alignment is achieved or the annotation budget is fully utilized, we proceed with fine- the-ttnnng an L.MM for the downstream task. This can be
4.1. Reward Modeling
1.1.1. OVERALL ALIGNMENT IMPROVEMENT
We here use GPT-4o for the initial alignment and evaluate RLTHF's iterative alignment improvements by measuring the preference accuracy of RMs trained with varying pro- portions of human annotations relative to the full dataset. We employ RLTHF on both the complete dataset and mul-
6

---

RETHEF Targeted Human Feedback for LLM Alignment
tiple down-sampled shards as described in 3.2.5. For given shard, we RLTHF in an iterative manner infusing targeted human annotations in each iteration. We evaluate
HH RLHF
T1;OR
95
Full Human Antotation
F-J Rluman Are station
Test Yest Accuracy (9%)
90
the trained RMs on separate test dataset and report their preference accuracy in Figure 3.
88
80
6P
In Figure 3, each data point for a shard corresponds to an iter- ation of RLTHF. The results show a consistent improvement in test preference accuracy across iterations, with significant early gains that gradually diminish as accuracy approaches the upper bound. Additionally, down-samplin enhances the efficiency of human annotations: RLTHF running on 1/2 and 1/4 shards outperforms its full-dataset counterpart when using the same number of human annotations. However, excessive down-sampling (e.g., 1/8 shard) may limit the achievable accuracy due to reduced data richness. For down- stream task fine-tuning, we identify 1/4 shard as the optimal choice. Under this setting, RLTHF enhances preference accuracy on HH-RLHF from GPT-4o's baseline of 74.7% to 89.6% with only 6% human annotations, and on TL:DR, from 78.8% to 88.0% with just 7% human annotations. We select the RMs from these iterations for labeling the full dataset, as outlined in $ 3.2.5. Evaluating the RMs trained on the fully human-labeled dataset, observe the accuracy to be 91.8% for HH-RLHF and 89.6% for TL:DR.
CIP--40
GP T-4 mini
GPT 40 min
70
10
10
Human Annotation %
RITHF (46)
Random (4o)
ALTHF (40 mini)
Figure 4: RLTHF effectively bridges the prefernce accuracy gap between LLM-generated (GPT-40/GPT-4o-ming) labels and fully beling accuracy of the LLMs. By strategically incorporating human annotations, RLTHF achieves higher accuracy gains compared to random human annotation, maximizing the impact of human effort.
data, regardless of the initial la-
4.1.3. EFFECTS OF HYPERPARAMETERS
Amplification Ratio. To investigate how the amplification ratio (discussed in Section 3.2.5) contributes to RLTHF, we fix the back-off ratio 3 at 60% and conduct a controlled study on different amplification ratios. The results for each dataset are shown in Figures 5a and 5d. We observe that both no amplification (a 1) and excessive amplifica- tion (a = 8) of human annotations lead to suboptimal RM improvements. Specifically, lower amplification results in smaller improvements in the initial iterations, while in later iterations, this trend reverses. This is expected, as no or low amplification weakens the impact of human annotations, particularly in the early iterations when the total number of annotations remains low, while over-amplification skews the training data distribution and increases the risk of overfitting, especially in later iterations when the base number of anno- tations is already large. For the best results, we start with a higher value of a and gradually reduce it (Appendix D.4),
4. 4.1.2. COMPARISON AGAINST THE BASELINES
We begin by using two different LLMs- - GPT-4o and GPT- 4o mini - for the initial AI labeling. We then employ two separate RLTHH pipelines, RLTHF (4o) and RLTHF (4o mini), to improve alignment. To evaluate their effectiveness, compare these pipelines against three aforementioned baselines (details in Appendix D.3).
The results of this experimental setup on two datasets are shown in Figure 4. RLTHF (4o) consistently outperforms Random (40), as random human annotation proves ineffec- tive in correcting Al mislabeling, resulting in only marginal improvements in test accuracy. Of particular interest is the "Return Investment (ROI)", which is measured as the increase in test accuracy per unit of human annotation. With just 6% human annotation, RLTHF (4o) achieves a 15.9x and 5.3x higher ROI compared to Random (4o) on HH-RLHF and TL;DR, respectively.
Back-off Ratio. To investigate the role of the back-off ratio i RLTHF, we conduct a controlled study where all other hyperparameters are held at their default values. The results across datasets are presented in Figures 5b and Se. We ob- serve a consistent pattern: a larger 3 yields greater improve- ment during the initial iteration, but its benefit diminishes in later iterations. Conversely, a smaller / leads to slower initial progress but accelerates in later stages. This behavior reflects a shifting trade-off between data quality and data richness. Early training, when the dataset is relatively unsanitized, quality is the primary bottleneck-makeee higher back-off ratio advantageous due to its stronger filter- ing effect. As the dataset becomes more sanitized over time, even a smaller back-off ratio can yield sufficiently clean data, allowing the benefits of increased data diversity and coverage to dominate.
Notably, RLTHF remains robust even when the initial AI labeling quality is lower. While GPT-4o mini starts with accuracy gap of 2.6% and 5.6% compared to GPT-4o, this gap shrinks to just 0.4% and -0.2 -0.2% after incorporating 10% human annotation on HHHF and TL;DR, respectively. This demonstrates that even when AI mislabeling is more prevalent, RLTHF more aggressively identifies and corrects errors, achieving a higher ROI on human annotation.
7

---

RLTHE: Targeted Human Feedback for LLM Alignment
90
90
Test Accuracy (%%
85
Test Accurary
85
85
80
80
80
Test
75
10%
82%
75
10% per 17
0
2
5
Iteration
aeration
Iteration *
(a) Amplification Ratio (HH--LHFF)
(b) Back-off Ratio (HH--LHHF)
(c) Annotation Batch Size (HI--RLHF)
85
All.ccurrcy (%6)
85
80
80
per 8% per
20%
75
p- 60%
00%
75
16% per Itr
0
2
4
5
0
3
4
Iteration
(d) Amplification Ratio (TL;DR)
(c) Back-off Ratio (TL;DR)
(f) Annotation Batch Size (TL;DR)
Figure 5: Effects of RLTHF hyperparameters HH-RLHF (top row) and TL;DR (bottom row). A larger amplification ratio (a) and back-off ratio (3) are beneficial in the initial iterations but should be gradually reduced as human annotations accumulate and dataset gets sanitized. The human annotation budget should be distributed across multiple batches, though the impact of finer granularity remains unclear.
HH-RLHF No Ampl//Backooff
TL;DR
Itr
No Annotation
Full RLTHF
No Annotation
No Ampl//Back-off
Full RLTHF
75.0
74.7
75.1
75.1
79.9
75.0
74.4
78.1
2
74.8
76.0
82.7
74.5
75.3
80.7
3
73.5
75.3
84.7
74.9
74.9
82.5
74.8
75.0
87.6
75.4
74.7
83.3
75.7
75.8
87.7
75.2
76.0
83.7
Table Ablation study of RLTHF HH-RLHF and TL,DR. All three factors human annotation, amplification of human annotations, and the back-off mechanism play crucial role in RLTHF's effectiveness.
In line with configuration settings (Appendix D.4), recommend starting with high back-off ratio and progres- sively reducing it as the dataset becomes more sanitized.
4% vs. 8% per iteration) does not provide meaningful advantage. Considering the increased GPU time and the overhead of more frequent human feedback cycles, we rec- ommend splitting the annotation budget into two iterations or using a similarly coarse granularity to strike a balance between performance gains and computational efficiency.
Annotation Batch Size. To evaluate how the number of annotated samples per iteration impacts the effectiveness of RLTHF- - and more broadly, assess whether RLTHF benefits from an iterative strategy over one-shot annotate- conduct controlled study, fixing the amplification ratio at 4 and the back-off ratio 3 at 60%.
4.1.4. ABLATION STUDY
all approach
We conduct an ablation study mechanisms in RLTHF and understand their individual contributions. All experiments in this study are performed with 4, f = 60%, and 4% human annotations per iteration, relative to 1/4 shard of the dataset. The primary objective of this study is to address two key questions.
the necessity of two
The results, shown in Figures 5c and 5f (with annotation percentages computed relative to the shard rather than the full dataset), indicate that the iterative approach yields up to 4.2% improvement by Itr-4 compared to the one-shot annotation strategy. This suggests that, across iterations, the RM not only learns from newly annotated samples but also generalizes to similar samples amplifying the benefit of each annotation round.
Does ref-immprovemet alone work? To evaluate whether RLTHF operate effectively without human annotations, set the annotation batch size to 0%, effectively reducing RLTHF to purely self- imprrvingoi framework, proposed in recent studies (Huang et al., 2024: Yu et al., 2025). The
However, our preliminary experiments show that further subdividing the annotation budget into smaller batches (c.g.,
8

---

RLTHFF Targeted Human Feedback for LLM Alignment
results, shown in Table I, reveal that -effiimprveeeentt alone faces inherent limitations and fails to exceed the baseline preference accuracy achieved by the Al (GPT in this case). Moreover, downstream task evaluations further demonstrate that the preference judgment of the AI alone is insufficient to achieve alignment with human preferences. These find- ings indicate that self-improvement based solely on Al's preference is inadequate for oracle human alignment.
Dataset Al Labeler
HH-RLHF 4o-mini
TL;DR
4o
40
4o-mini
Al-labeled
49.2
45.1
59.2
56.4
Random
52.5
46.3
59.8
57.5
RLTHF 58.1
56.1
62.3
62.4
Human
55.7
60.2
Table 2: Win rate against SFT (%). RLTHF outperforms the Al-labeled, Random, and Human baselines across both datasets. Even with weaker GPT-4o mini, RLTHF also achieves a win rate comparable to that with GPT-4o.
Are amplification and back-off necessary? To assess the importance of RLTHF's human annotation amplification and samitization back-off mechanisms, we ablate both mech- anisms simultaneously - given that the effect of removing amplification alone was already examined in $ 1.3 using a a= 1. As shown in Table I, RLTHH achieves only marginal improvement under this configuration, even as human an- notations accumulate over iterations. Without sanitization back-off, incorrect labels remain in the training data de- grading overall data quality. Without human annotation amplification, significantly more correct samples are needed to override the influence of incorrect ones, compounding the difficulty of effective alignment. Together, these results highlight that both mechanisms are essential for driving meaningful improvements in RLTHE,
are consistent with our observations in 4.1.2, further val- idating RLTHF's robustness and effectiveness, even when faced with suboptimal AI labeling due to model limitations, task complexity, or poor prompting.
Interestingly, RLTHF outperforms the fully Human- annotated baseline, despite incorporating annotations from the same dataset. We attribute this advantage to RLTHH'ss sanitized data selection for RM training, as discussed in 3.2.5. Fully human-annotated datasets inherently contain noise and biases (Wang et al., 2024a; Sun et al., 2024a; Ethayarajh et al., 2024). In $ 3.2.5, we illustrated how such samples tend to cluster around the "knee" of the re- ward distribution curve. By leveraging the back-off ratio hyperparameter, RLTHF controls noisy and biased samples, ensuring cleaner training dataset. The selected RMs from RLTHF are trained on data with a back-off ratio of 10% in the corresponding iteration, resulting in reduced bias and noise. Consequently, DPO training on data labeled by these RMs leads to better downstream performance.
4.2. Downstream Tasks
To evaluate RLTHF's effectiveness on final downstream tasks, we perform DPO training using data prepared by RLTHF and compare it against three baselines. For RLTHF, we use an RM trained on a 1/4 shard of the dataset re- fined with targeted human annotations - to label preferences across the full corpus. Specifically, for HH-RLHF, we use 6% human annotation when GPT-4o is the initial labeler and 7% when using GPT-4o mini; for TL:DR, we apply 7% human annotation in both cases. We then train an LLM, Qwen2.5-3B, with DPO using the RLTHF-refined dataset. For the Random baseline, we inject an equivalent amount of randomly selected human annotations (on top of the AI- labeled data) into the DPO training set. Model performance is measured by pairwise win rate against outputs from super- vised fine-tuned Qwen2.5-3B, using AlpacaEval (Li et al., 2023b) with Claude 3.5 Sonnet (Anthropic, 2024) the evaluation judge. Evaluations are conducted held-out test sets from both datasets (details in Appendix E).
4.3. Cost Analysis
RLTHF introduces two types of additional costs: (1) the cost of LLM annotation during initial alignment, and (2) the cost of iterative RM training. However, our case study in Appendix F demonstrates that even when accounting for these costs, RLTHF can still reduce the overall cost by 84.0-86.0% based on a conservative estimate.
5. Conclusion
In this work, we introduce RLTHF, an iterative reward model training approach that enhances alignment in prefer- ence datasets by strategically infusing human annotations, complemented by sanitized AI labeling. Through reward dis- tribution analysis, we identify key samples for targeted hu- man intervention, optimizing annotation efficiency. Our ex- periments demonstrate that RLTHH progressively improves alignment, converging toward comprehensive human align- ment. Furthermore, models trained on our refined datasets for downstream tasks even outperform the models trained on datasets with full-human annotations.
The results in Table 2 align with the observed preference accuracy trends. Across both datasets, RLTHF achieves a higher win rate than the fully Human-annotated baseline using only 6-7% of total human annotations, while signifi- cantly outperforming models trained on purely Al-labeled data as well as Random human annotations. Notably, even with weaker Al labeler (GPT-4o mini), RLTHF effectively bridges the gap within the same annotation budget, achiev- ing a win rate comparable to that of GPT-4o. These findings

---

RLTHF: Targeted Human Feedback for LLM Alignment
Acknowledgments
A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
We thank Eduardo Rodrigues, Roberto Estevao, Maria Angels de Luis Balaguer, Jessica Wolk, Rafael Padilha, Leonardo Nunes, and Shobana Balakrishnan for their valu- able support in experimenting with private data and integrat- ing RLTHF into the production pipeline.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
Huang, C., Fan, Z., Wang, L., Yang. F,, Zhao, P., Lin, Z., Lin, Q., Zhang, D., Rajmohan, S., and Zhang, Q. Self-evolved reward learning for llms. arXiv preprint 1xXi::241.000411 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language mod- els cannot self-correct reasoning yet. arXiv preprint arXiv:2310.0f 2023.
References
Jiang. A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arX00:2401.040888 2024.
Achiam, J., Adler, S., Agarwal. S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Kepel, D. and Valogianni, K. Autonomous prompt en- gineering in large language models. arXiv preprint NBKV:2407.11000 2024.
AI, F. LLM TUNING SERVICES BY FOLIO3 AI, 2024.
Amazon Web Services. Amazon sagemaker ground truth pricing, 2025. URL https://aws amazon. com/ segemaker-ail http://aaagooonntuut/////ooooooo cessed: 2025-03-31.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stan- ley, O., Nagyfi, R., et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Ac-
Anthropic. Claude 3.5 Sonnet. https: htts://wwwer //wwww
anthropic. com/ cwwude///e/e///eeeaoeee soludd/sooneen sonnet, 2024. Ac- cessed: 2025-02-18.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K. R., Bishop. C., Hall, E., Carbune, V., Rastogi, A., et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with feedback. In Forty-first International Conference on Machine Learning.
Atreya, M. Fine-Tuning Al Models with Tuning-as-a as-a- Service Platforms, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedhack. arXiv preprint 2rx4v:2204.05862. 2022a.
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
Li, X., Yu, P., Zhou, C., Schick, T., Levy, O., Zettlemoyer, L., Weston, J., and Lewis, M. Self-alignment with instruc- tion backtranslation. arXiv preprint arXiv:2308.066. 06259, 2023a.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint
2022b.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-follooiing models. https: /tta://gtie ht////ttttuu/ www.///ttttttaaaaaeeoe a @ipaca_evall 5 2023b.
Chung. H. W., Hou, L., Longpre, S., Zoph, B.. Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling iestruction-finetumme language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
David, H. A. The method of paired comparisons, volume 12. London, 1963.
Microsoft. Microsoft 365 Copilot Tuning overview (pre- view), 2024. URL https: mi cwwssoff... cann//en-uae/pppploo//nicoottte 365/ copilot-tuuning-overrie
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan,
10

---

RLTHE: Targeted Human Feedback for LLM Alignment
Microsoft Azure. Ndm a100 v4-series virtual machines, 2025. URL https::/ arnusa/aauute//iitttalaamaanniia opa-aaceelerracod//caaaaaO/// tabs=sizebase Accessed: 2025-03-31.
Sun, Z.,, Shen, Y., Zhou, Q. Zhang, H., Chen, Z.,, Cox, D., Yang. Y., and Gan, C. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024b.
com/
OpenAl. Openai api pricing, 2025. URL https:// openai.com api/pricing/ Accessed: 2025-03- 31.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23 1805, 2023.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog appli- cations, arXiv preprint arXiv:2201.08233 2022.
Pang, J.-C., Wang, Li, K., Chen, X.H.,, Xu, J., Zhang, Z., and Yu, Y. Language model seprovmmpeveeeent by reinforcement learning contemplation. arXiv preprint arXiv:2305.144883 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B.. Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv: 230,77911..1 2023.
Panickssery, A., Bowman, S. R., and Feng. Llm evalu- ators recognize and favor their generations. arXiv preprint arXiv:2404. 0100224444.377.., 13076, 2024.
Völske, M., Potthast, M., Syed, S., and Stein, B. Tl; dr: Min- ing reddit to learn automatic summarization. In Proceed- ings of the Workshop on New Frontiers in Summarization, pp. 59 59-63, 2017.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang. B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.0600
2024a.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv: 1707.06347 2017.
Wang, P., Li, L.,, Shao, 7., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2024b.
Sharma, A. Announcing fine-tuning for customization and support for new models in Azure AI, 2024.
Sharma, A., Keh, S., Mitchell, E., Finn, C., Arora, K., and Kollar, T. A critical evaluation of ai feedback for aligning large language models. arXiv preprint arXiv:2402.12366 2024.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. arXiv preprint 2021.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.0331/ 2024.
Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is dpo superior to ppo for Ilm alignment? a comprehensive study. arXiv preprint arXiv: 2404. 10719, 2024.
Stiennon, N., Ouyang, I.., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021. 2020.
Yu, P., Yuan, W., Golovneva, O., Wu, T.. Sukhbaatar, S., Weston, J., and Xu, J. Rip: Better models by survival the fittest prompts. arXiv preprint arXiv:2501. 18578, 2025.
Sun, H., Shen, Y., and Ton, J.F.. Rethinking bradley- terry models in preference- based reward modeling: Foundations, theory, and alternatives. arXiv preprint arXiv:24/1.0499, 2024a.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.. 100000 2024.

---

RLTHE: Targeted Human Feedback for LLM Alignment
Zeng, Z.,, Cheng, Q. Yin, Z., Wang, B., Li, S., Zhou, Y., Guo, Q., Huang, X., and Qiu, X. Scaling of search and learning: A roadmap to reproduce ol from reinforcement learning perspective. arXiv preprint arXiv:2412.14135, 2024.
Zheng. I.., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023a.
Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., et al. Secrets of rihf in large language models part i: Ppo. arXiv preprint arXiv:2307.0496. 30044449444..... 2023b.
12

---

RLTHE: Targeted Human Feedback for LLM Alignment
A. Prompt Templates for Initial Alignment
structure the prompt using the following components to obtain the initial preference judgments from off-the-shelf LLMs (e.g., GPT-4o, Gemini, etc.).
1. Task description: User-provider short description and instructions for the task.
2. Preference criteria: User Aserrprovided detailed criteria defining what is good/bad response.
3. Few-shot examples (optional): User-provided samples, chain-of-thought rationales and final preferences.
4. Sample annotate: The target sample be annotated.
Below provide unified system prompt and user prompts used for HH-RLHF and TL;DR. Note that these prompts not carefully tuned, as RLTHF should not assume ideal prompt engineering.
A.1. System Prompt
You are an assistant designed to compare and select the better response to given task and input based on the provided criteria.
A.2. User Prompt for HH-RLHF
## TASK
You are helpful and harmless Assistant. Given the context, provide the most helpful and harmless response.
#* CRITERIA
#*Helpfulnessee: The response should be relevant, clear, and effective in addressing the human user's task or question within the given context. **Honest***: The response should be accurate, truthful, and reliable. w.Harmlessnessse The response should be safe, considerate, and free from potential harm or negativity.
# INPUT (input)
## CANDIDATE RESPONSES
*Response A+*: (response_a]
* * *Response B**: (response.b)
## INSTRUCTIONS
Review the task, criteria, and input thoroughly.
2. Evaluate each response based on how well it aligns with the criteria.
3. Decide which response (A or B) is better aligned with the criteria.
4. IE both responses seem equal in quality, choose the one that you judge to be
the best.
5. Provide your answer in the following format:
rationale: <Your concise reasoning>
preference:
"Response or "Response
13

---

RLTHF: Targeted Human Feedback for LLM Alignment
A.3. User Prompt for TL;DR
# TASK
Summarize the given reddit post.
## CRITERIA
What makes for good summary? Roughly speaking, good summary is shorter piece of text that has the essence of the original { tries to accomplish the same purpose and conveys the same information as the original post. We would like you to consider these different dimensions of summaries:
** is the summary a good representation **Clarity:** is the summary friendly) Does it express ideas clearly?
the post?
*Accuracy:** does the summary contain the same information the longer post?
*Purpose:** does the summary serve the same purpose as the original post? *Concise:** is the summary short and to-the-point?
* **Style:** is the summary written in the same style as the original post? Generally speaking, we give higher weight to the dimensions at the top of the 1ist. Things are complicated though none of these dimensions are simple yes/ matters, and there aren't hard and fast rules for trading off different dimensions.
## INPUT {input}
#* CANDIDATE RESPONSES
**Response A+*: (response
**esponse B**: (response.b)
## INSTRUCTIONS
1. Review the task, criteria, and input thoroughly.
2. Evaluate each response based on how well it aligns with the criteria.
3. Decide which response (A or B) is better aligned with the criteria.
4. If both responses seem equal in quality, choose the one that you judge to be
the best.
5. Provide your answer in the following format:
rationale: <Your concise reasoning>
preference:
"Response A" "Response B"
B. Pseudocode for the Full RLTHF Procedure
In Algorithm 1, we present the full procedure of RLTHF in pseudocode.
C. Iterative Alignment Improvement
In Figure 6, we show all the reward distribution curves and accuracy density curves from all the iterations that we the HH-RLHF dataset.
D. Experimental Setup
D.I. Data Preparation
D.1.1. DATASETS
We use the following datasets our experiments:
HH-RLHF: We use Anthropic's helpful and harmless human preference dataset (Bai al., 2022a), which includes 161K
14

---

RLTHE: Targeted Human Feedback for LLM Alignment
Algorithm 1 Pseudocode for the Full RLTHF Procedure
Input: Unlabeled dataset D with samples (instruction, response_a, response.b). max iterations M Output: Final aligned labels L reward model R.Mimal
// Step 1: Initial Alignment on a Random Shard
So + D. D.randomSubset(()
Lo -- Sb. (b.11abeliith(LMM)))
RMM < TrainRewardModell(..
Step 2: Iterative Alignment Improvement
for i 0 to M do
A; <- EmptyL.stll
for all (a, Vor 3r) E C, do
8 - R.M (x. ye) - (M,
aappend (ir, Mer Vr. Vv. 8))
end for
Swoned f A...ortBy(())
C, f Ssoned Sannrffinndorrrretttmapllell
Hi + Suaned- f riddiiar
R2 - Suaned (
Li+1 < C. RR U Hi
RM++1 + Irainneewardoodde1([44
end for
// Step 3: Extend Full Dataset C 1.thhe R.M < TrainRewardModel1
Step 4:
Knowledge Transfer
Option Use C to directly align downstream model (e.g., DPO) Option Integrate R.Munal into RL-based optimization (e.g.. PPO)
training samples. Each sample consists of conversation context between a human and AI assistant together with preferred and non-preferred response selected based on human preferences of helpfulness and harmlessness. For SFT, following previous work (Rafailov et al., 2024), we use the chosen preferred response as the completion to train the models.
TL;DR: We use the Reddit TL:DR summarization dataset (Völske al., 2017) filtered by OpenAl along with their human preference dataset (Stiennon al., 2020), which includes 93K training samples. We use the Human-written post-summarizztion pairs for SFT, and use the human preference pairs model summarizations for RLTHF and DPO.
All test samples are completely separated from the training samples throughout the experiments.
1.1.2. FLIPPING HUMAN PREFERENCES
has been observed that both datasets contain a significant number of incorrect preferences due to human annotation noise and biases (Wang al., 2024a; Ethayarajh al., 2024). However, the reward distribution curve, these errors become intertwined with the Hard-to-annotate samples that RLTHF prioritizes for annotation. As result, incorrect human labels are more likely to propagate through subsequent iterations. This issue stems from the reliance pre-annotated public datasets, where annotation noise and biases are inevitable due to the heavy workload on human labelers. By reducing the overall human annotation burden, RLTHF helps mitigate these human errors.
To minimize this unfair penalty in our evaluation, and following prior work (Wang et al., 2024a), we first train an RM using the full set of original human annotations. We then identify and flip the labels of samples that receive negative preferences
15

---

RLTHE: Targeted Human Feedback for LLM Alignment
Correct Incomeet
Doodfty of Correct Samplos
Incoreet Hiddl
0.73
42
Middic
0.79
L Margin
Dondfty corcertr
Sicre Cot
0.90
3.50
0.02
soco 10000 15600
tank
Bank
Bank
Bank
(a) Reward dist. Itr-O
(b) Correctness dist.
(c) Reward dist. Itr-1
(d) Correctness dist. Itr-1
Cornect
1.00
Cornect
of Comeer Comerr Sample
1.00
Incomect
Margin
Decddty of cornect
0.73
L.Margn
Scow
0.25
0.25
-20
0.00
5002 1000105t0000000000000000
5000 000000000000000000000000
5000 10000 (l00000000000000000000
5000 10000 w000001 5doppo000soooooooooooooo
Rank
Rank
Rank
Rank
(c) Reward dist. Itr-2
(f) Correctness dist. Itr-2
Reward dist. Itr-3
(h) Correctness dist. Itr-3
Cornec
coroce Samgle
102
Donuty Coreet Sanolss
Incomncct
Incomect
Modea
0.75
Niddl
0.5
Margin
Margin
0.50
0.50
-20
0.02
100001 120000000222000
5050
s0to sc00zonzaasoocii
10500 scnnnostziiinniiie
Bank
Bank
Bank
Bank
(i) Reward dist. Itr-4
(i) Correctness dist. : Itr-4
(k) Reward dist. Itr-5
Correctness dist. Itr-5
Figure 6: Reward and correctness distribution curves for all the iterations on HH-RLHF dataset.
from the model-251 25% for HH-RLHF and 20% for TL:DR. These flipped labels serve as the ground truth for experiments.
To assess the effectiveness of this approach, rates against the SFT model. The results, presented in Table show that while both DPO models outperform the SFT baseline, the model trained on flipped labels achieves greater improvements across both datasets. This suggests that label flipping has a net positive impact on downstream tasks by correcting more incorrect labels than it introduces.
DPO on both the flipped and unflipped datasets and compare their win
Preference Source for DPO
HH-RLHF
TL:DR
Unflipped
51.0
59.4
Flipped
55.7
60.2
Table 3: Win rate against SFT (%)
D.2. Model Training
SFT: We perform full- pall-aeereter fine-tuning on Qwen2.5-3B base model. We use learning rate -5 warm up ratio 0.2, and batch size of 32 for training 4 epochs.
Reward Modeling: We train our reward model with -.ama-3.1-8B-IInstii This was a LoRA fine-tuning. We use learning rate le warm up ratio 0.1, LoRA rank 32, LoRA alpha 64, and batch size of 128 for training 2 epochs.
DPO: We perform DPO on the SFT model with data sanitized by RLTHF. We use learning rate 1e-6. warm up ratio 0.1, beta 0.1 and 0.5 for HH-RLHF and TL.DR datasets, respectively, and batch size of 64 for training 4 epochs.
All training is done on a node of A 100 NVIDIA GPUs with DeepSpeed.
16

---

RLTHF: Targeted Human Feedback for LLM Alignment
D.3. Baselines
We compare RLTHF with the following baselines.
GPT-4o/GPT-4o mini: This baseline involves directly using data annotated by GPT-4o/4o-mini to fine-tune a model for the downstream task, following an approach similar to RLAIF (Lee et al., 2023).
Random: This baseline combines GPTT-gnnrated preferences with randomly selected samples for human annotation varying percentages. It serves strawman approach to assess the efficiency of RLTHF's annotation strategy. Specifically, we compare RLTHF against this method at every iteration, ensuring both use the same total number of human annotations.
Human: This refers to RLHF with full human annotations. RLTHF aims to approach and even surpass this level of quality while significantly reducing annotation effort.
D.4. RLTHF-Specific Configurations
Unless stated otherwise, we use the following default configurations for RLTHE:
Sharding: RLTHF run on randomly dawnesampled 1/4 shard of the full dataset.
Amplification Ratio:
The default a values are 4, 4, 4, 2, and I for iterations 1-5, respectively, and I for all subsequent
iterations.
The default 3 values are 60%, 60%, 60%, 40%, and 20% for iterations 1-5, respectively, and 10% for all subsequent iterations.
Back-off Ratio:
Annotation Batch Size: In each iteration, human annotation is applied to 4% of the given shard.
These hyperparameters are chosen based on heuristics and limited empirical observations, which may underestimate RLTHF's full potential. However, we provide preliminary analysis of their impact on RLTHF's performance in $ 4.1.3 and ablation study of the critical components of RLTHF in $ 4.1.4. All those experiments are conducted with GPT-40 mini initial alignment to better assess RLTHF's sensitivity to different factors.
E. Obtaining Pair-wise Win Rate with AlpacaEval
To compute the pairwise win rate, we use the default annotator template a GPT-4 with Claude 3.5 Sonnet as the judge. This substitution helps mitigate self-enhhancemet bias (Zheng et al., 2023a; Panickssery al., 2024), considering that the initial alignment based on GPT-series models, All win rates are calculated using unseen test set of 4K samples for both HH-RLHF and TL:DR.
in AlpacaEval but replace
Since AlpacaExval's default prompt template does not directly apply to the HH--LLHF and TL:DR test sets, we make slight adaptations to better align with the evaluation goals of each dataset while preserving its original structure. To address the well-documented verbosity bias of LLM judges and following prior work (Zheng et al., 2023a: Rafailov et al., 2024), we explicitly request concise evaluations in the prompts to better reflect human judgment. We provide our adapted win rate prompt templates for the HH-RLHF and TL:DR datasets.
17

---

RLTHE: Targeted Human Feedback for LLM Alignment
E.I. Adapted Prompt Template for HH-RLHF
< im.start |crtt//use >user
I want you to create a Leaderboard of different large- Large-language models. To do so, will give you the instructions (prompts) given to the models, and the responses of two models. Please rank the models based on which responses would be more helpful and harmless while being as concise as possible from a human perspective. All inputs and outputs should be python dictionaries.
Here is the prompt:
{
"instruction": """[issruction)""""
}
Here are the outputs of the models:
"model": "model-1",
"answer": """ ""(output.1]"""
}
"model": "model.2", "answer": """ (outputt """
]
Now please rank the models by the quality of their answers, so that the model with rank has the most helpful and harmless output while keeping it as concise as possible. Then return list of the model names and ranks, i.e., produce the following output::
['model': <model-rame>, 'rank': <model-ranke {' ('model': <model-names 'rank'; <model-ranke
]
Your response must be a valid Python dictionary and should contain nothing else because we will directly execute it in Python. Please provide the ranking that the majority of humans would give. <lim_end|> >
18

---

RLTHE: Targeted Human Feedback for LLM Alignment
E.2. Adapted Prompt Template for TL;DR
< im >user
language models on the task I want you to create a leaderboard of different of forum post summarization. To do so, will give you the forum posts given to the models, and the summaries of two models. Please rank the models based on which does better job summarizing the most important points in the given forum post, without including unimportant or irrelevant details. Please note that the best summary should be precise while always being concise as possible. All inputs and outputs should be python dictionaries.
Here is the forum post:
{
"post": ""(instruction)"""
}
Here are the outputs of the models:
"model": "model 1",
"answer": """ """ooutput1]"""
}
"model": "model_2", "answer"; """ 2
]
Now please rank the models by the quality their summaries, so that the model with rank has the most precise summary while keeping it concise possible. Then return list of the model names and ranks, i.e., produce the following output:
['model': <model-name>, rrak'' <model-ranko), ('model': <model-name>, 'rank': <model-rark)
]
Your response must be valid Python dictionary and should contain nothing else because we will di directly execute it in Python. Please provide the ranking that the majority of humans would give.

 >
F. Cost Analysis
We take our experiments on HH-RLHF as a case study.
Dataset Size: 160,800 samples, each consisting of a prompt and two responses. The average input length is 671 tokens, and the average output length is 134 tokens.
Human Annotation Cost: Amazon Mechanical Turk (Amazon Web Services, 2025) suggested text classification pricing:
$0.012 x 3 (labelers) = $0.036 per sample
Note: Here the suggested pricing may be much lower than the actual cost. Our data samples have an average token number of 314 (prompt + responses), which is larger than most text classification units. AMT's labeling service providers typically list an hourly rate of $6-7. According to human reading speed 200-250 words per minute, the actual cost should be
19

---

RLTHE: Targeted Human Feedback for LLM Alignment
around $0.13-0. 8/sample/labeler, which is more than 10x of the suggested pricing. In the following analysis, we still the suggested pricing as lower bound provide conservative estimate of RLTHF's gain.
LLM Annotation Cost: Table 4 summarizes the estimated LLM annotation cost per sample for GPT-4o and GPT-4o mini, based on OpenAl's token-based pricing (OpenAI, 2025)'.
Table 4: GPT-4o and GPT-4o mini Annotation Cost
Input Cost
Output Cost ($ per 1M tokens)
Avg Cost ($ per sample)
Model
Input Tokens
Output Tokens
($ per 1M tokens)
GPT-4o
2.50
10.00
0.0030
671
134
GPT-4o mini
0.15
0.60
0.00018
RM Training Inference Cost: Azure ML costs $32.77 per hour for a A10 80GB node (Microsoft Azure, 2025). RLTHF RM training and inference per iteration takes less than hours on the full dataset, and less than 2 hours on the 1/4 subset. The inference time negligible compared to training time.
Comparison: For computing, we only consider RM training and inference, as the downstream LLM training is the same for both full-human annotation and RLTHF. Table 5 compares the total cost of full-human annotation against two variants of RLTHF using GPT-4o and GPT-4o mini, respectively. The RLTHF setting assumes only 6% human annotation, 1/4 dataset shard for training, and RM training iterations.
Table 5: Cost Comparison between Full-human Annotation and RLTHF
Solution
Human Annotation ($)
LLM Annotation ($)
RM Train & Infer (S)
Total ($)
Full-human
0.036 x 160,800 = 5788.8
5788.8
RLTHF (4o)
0.036 160,800 x 0.06 = 347.3 .3
0.0030 X
160,800
120.6
32.77 x 2 7 = 458.8
926.7
RLTHF (40 mini)
0.036 160,800 x 0.06 347.3
0.00018
160,800
7.2
32.77 77.77 x 2 7 458.8
813.3
Even counting the extra LLM labeling and computing overhead, RLTHF still reduce the overall cost by 84.0-86.0% Note that here the gain may be underestimated again given the rapidly developing computing infrastructure and increase of labor price.
'Pricing as of March 31, 2025.
20


Reviews:
[{"review_id": "lcwvG4bGpM", "reviewer": "Reviewer_7upR", "summary": "This paper introduces Sargy, a human-AI hybrid framework designed to improve LLM alignment with user preferences while minimizing human annotation costs. Sargy strategically combines LLM-generated labels with selective human corrections, identifying and refining mislabeled samples using a reward model’s distribution. The framework operates in three stages: (1) Initial alignment, where an LLM provides coarse labeling; (2) Iterative improvement, leveraging human feedback to correct challenging samples; and (3) Knowledge transfer, using the refined dataset for downstream preference optimization tasks like DPO and PPO. Experiments on HH-RLHF and TL;DR datasets demonstrate that Sargy achieves oracle-level alignment with just 15–20% of the human annotation effort, enabling high-quality preference learning with minimal cost.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and constructive review of our work!\n\n**Q1 - Data complexity:** The paper claims that using 15–20% of the data achieves performance comparable to using the full dataset. How does this generalize to more complex datasets?\n\n**Re:** The generalizability of Sargy across complex tasks is indeed a genuine point that we also tried to investigate in the paper. Recent research [1] suggests that the complexity of a task is a function of the model capability. To address this, we intentionally use GPT-4o mini as a representative of comparatively weaker models for initial feedback, effectively making the task harder. Our experiments in 4.1.1 show that even when starting with a weaker model (i.e., a harder task), Sargy consistently closes the gap with stronger initial models (GPT-4o) and achieves **similar preference accuracy** after 5 iterations with equal amount of data and human annotation.\n\n**Q2 - Major RM misalignment:** What happens if the reward model predicts that most of the data is misaligned with human preferences?\n\n**Re:** To address this pragmatic consideration, Sargy natively incorporates a validation process of initial alignment (Section 3.2.3 paragraph-1). If a major misalignment is found, the initial alignment prompt needs to be updated according to the misaligned samples / users / any existing techniques. We will further highlight that in our final version.\n\n**Q3 - Reward model stability:** Since reward models may rely on spurious correlations, how does the method ensure the stability and reliability of reward alignment for guiding human annotation?\n\n**Re:** Sargy provides a generic framework that can correct mistakes for any reward models. Recent evidence [2] has shown that stronger base models may better capture the real target correlations during reward modeling. In our experiments, we found Llama-3.1-8B-Instruct on par in aligning with human preference.\n\n**Q4 - Initial alignment quality:** How sensitive is the framework to the initial LLM-generated labels? Would errors in this stage propagate and affect the final performance?\n\n**Re:** Sargy’s robustness to initial alignment quality is a key strength of our approach. (Similar to Q1) Experiments in section 4.1.1 show that even when starting with a poorly aligned model (GPT-4o mini), Sargy consistently **closes the performance gap with stronger models** after the **same number of human annotations and iterations**. A key reason for this is Sargy's strategic data selection: when initial alignment is poor, the selected samples for human annotation tend to have a higher proportion of mislabeled instances. This targeted selection ensures that human feedback corrects a larger number of errors per annotation, leading to a **higher improvement-per-annotation ratio**.\n\n[1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters\n\n[2] Scaling Laws for Reward Model Overoptimization", "claims_and_evidence": "The paper claims that Sargy can achieve oracle-level alignment while reducing human annotation effort to just 15–20% of the full dataset. The experimental results on HH-RLHF and TL;DR datasets support this claim, showing that models trained on Sargy’s filtered datasets perform on par with those trained on fully annotated data. However, the effectiveness of this reduction likely depends on dataset difficulty, which is not fully explored in the paper. Additionally, since reward models may rely on spurious correlations, their alignment scores do not always guarantee correct annotations, raising concerns about robustness.", "methods_and_evaluation_criteria": "The paper presents a three-stage framework that combines LLM-generated labels with selective human feedback, guided by a reward model’s reward distribution. The evaluation is conducted on preference datasets (HH-RLHF and TL;DR) and assesses alignment quality through downstream task performance. While the method is straightforward and practical, a more detailed discussion of dataset complexity and reward model stability would strengthen the evaluation.", "theoretical_claims": "The paper does not introduce new theoretical claims.", "experimental_designs_or_analyses": "The experiments demonstrate that Sargy effectively reduces annotation costs while maintaining alignment quality. However, an ablation study on dataset difficulty and reward model stability is missing.", "relation_to_broader_scientific_literature": "The paper aligns with research on RLHF, human-AI collaboration in annotation, and active learning.", "essential_references_not_discussed": "I did not find any missing essential references.", "other_strengths_and_weaknesses": "Please see my comments for other questions.", "other_comments_or_suggestions": "Please see my comments for other questions.", "questions_for_authors": "1. The paper claims that using 15–20% of the data achieves performance comparable to using the full dataset. How does this generalize to more complex datasets?\n\n2. What happens if the reward model predicts that most of the data is misaligned with human preferences?\n\n3. Since reward models may rely on spurious correlations, how does the method ensure the stability and reliability of reward alignment for guiding human annotation?\n\n4. How sensitive is the framework to the initial LLM-generated labels? Would errors in this stage propagate and affect the final performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9r4nJrC3OA", "reviewer": "Reviewer_fzYM", "summary": "The paper introduces Sargy, an iterative human-AI hybrid framework for aligning large language models (LLMs). The core idea is to leverage a reward model to identify data points that are difficult for an AI to label consistently with human preferences and then to selectively solicit human feedback on these challenging instances. By focusing human annotation efforts on these \"hard-to-label\" samples, the authors demonstrate that their approach achieves performance comparable to models trained with full human annotation, while using only 15-20% of the human annotation effort. This significantly reduces the cost associated with aligning LLMs.\n\n## update after rebuttal\nI have decided to maintain my original score. My assessment remains that the novelty of the proposed technique is borderline, as it bears significant resemblance to existing filtering methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and constructive review of our work!\n\n**Q1:** Comparison with a simple filtering strategy\n\n**Re:** Thanks for the relevant reference! The RIP paper was first published right around the submission deadline (1/30), and we will cite it in our final version. After careful reading, we want to highlight that:\n- RIP focuses on selecting more effective data while Sargy focuses on improving alignment with human preference. They can work together.\n- The improvement from human feedback is validated by our experiments: the ablation study (section 4.1.5) with \"No Annotation\" is exactly a representative of \"filtering the bottom percentile of reward gap distribution\". Sargy gets solid gain compared to this.\n\n**Q2:** Analysis of the accuracy of the reward model itself\n\n**Re:** The primary metric for all results in Section 4.1 is the preference accuracy of the RM itself. We will explicitly mention that in our final version.\n\n**Q3:** Quantify the training overhead\n\n**Re:** We take our experiments on HH-RLHF as a case study\n- **Dataset Size:** 160,800 samples, each with a prompt + 2 responses\n- **Human Annotation Cost:** Amazon Mechanical Turk [1] suggested text classification pricing: 0.012 * 3 (labelers) = \\\\$0.036 per sample\n\n  Note: Here the suggested pricing may be much lower than the actual cost. Our data samples have an average token number of 314 (prompt + 2 responses), which is larger than most text classification units. AMT's labeling service providers typically list an hourly rate of \\\\$6-7. According to human reading speed of 200-250 words per minute, the actual cost should be around \\\\$0.13-0.18/sample/labeler, which is more than 10x of the suggested pricing. In the following analysis, we still use the suggested pricing as a lower bound to provide a conservative estimate of Sargy's gain.\n\n- **LLM Annotation Cost:**\n  - Average input length (template + prompt + 2 responses): 671 tokens\n  - Average output length (rational + judgement): 134 tokens\n  - OpenAI API cost (per 1M tokens)\n    \n    **GPT-4o:** \\\\$2.5 for input; \\\\$10 for output\n    - 671 * 0.0000025 + 134 * 0.000010 = \\\\$0.0030 per sample\n    \n    **GPT-4o mini:** \\\\$0.15 for input; \\\\$0.6 for output\n    - 671 * 0.00000015 + 134 * 0.0000006 = \\\\$0.00018 per sample\n- **RM Training & Inference Cost:** Azure ML costs \\\\$32.77 per hour for a 8xA100 80GB node [2]. A Sargy RM training + inference per iteration takes less than 8 hours on the full dataset, and less than 2 hours on the 1/4 subset. The inference time is negligible compared to training time.\n- **Comparison:** (For computing, we only consider RM training + inference, as the downstream LLM training is the same for both full-human annotation and Sargy)\n  - **a. Full-human annotation:** 0.036 * 160800 (human annotation cost) = \\\\$5788.8\n  - **b. Sargy (full set + GPT-4o + 0-5 iterations) + 20% human annotation:** 0.0030 * 160800 (LLM annotation) + 0.036 * 160800 * 20% (human annotation) + 32.77 * 8 * 6 (training + inference) = \\\\$3213.1\n  - **c. Sargy (full set + GPT-4o mini + 0-5 iterations) + 20% human annotation:** 0.00018 * 160800 (LLM annotation) + 0.036 * 160800 * 20% (human annotation) + 32.77 * 8 * 6 (training + inference) = \\\\$2759.7\n\n  Our additional experiments on both TL;DR and HH-RLHF show that by down-sampling the full dataset to a 1/4 shard for Sargy’s processing and conducting inference (scoring) on the full dataset **only at the end** using the reward model from the final iteration, we achieve accuracy comparable to using the full dataset throughout Sargy's process. This approach not only reduces computational costs but also decreases the required human annotations **(6-7% instead of 15-20%)** w.r.t. to the full dataset. We will include these results in the final version. With such approach, the cost of Sargy can be further reduced:\n  - **d. Sargy (1/4 shard + GPT-4o + 0-6 iterations) + 6% human annotation:** 0.0030 * 160800 * 1/4 (LLM annotation) + 0.036 * 160800 * 6% (human annotation) + 32.77 * 2 * 7 (training + inference) = \\\\$926.7\n  - **e. Sargy (1/4 shard + GPT-4o mini + 0-6 iterations) + 6% human annotation:** 0.00018 * 160800 * 1/4 (LLM annotation) + 0.036 * 160800 * 6% (human annotation) + 32.77 * 2 * 7 (training + inference) = \\\\$813.3\n\n  Even counting the extra LLM labeling and computing overhead, Sargy can still reduce the overall cost by **44.5-86.0%**. Note that here the gain may be underestimated again given the rapidly developing computing infrastructure and increase of labor price.\n  \n  We will include this analysis in our final version.\n\n[1] https://aws.amazon.com/sagemaker-ai/groundtruth/pricing/\n\n[2] https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/ndma100v4-series?tabs=sizebasic", "claims_and_evidence": "In general, yes.", "methods_and_evaluation_criteria": "The paper's approach of using the reward gap to identify and relabel hard-to-annotate preference pairs with human is interesting. The comparison with a random selection baseline effectively highlights the benefits of their targeted annotation strategy.\n\nHowever, there are potential confounding factors to consider. A recent study [1] suggests that simply filtering out preference pairs with a large reward gap, regardless of the direction of the gap, can lead to significant performance improvements. This raises the question of whether the performance gains observed in this paper are solely due to the relabeling of the bottom percentile of the reward gap distribution, or if a similar improvement could be achieved by just filtering these instances without human intervention. The authors might consider exploring the impact of filtering and compare with relabeling to demonstrate the necessarily of human annotation.\n\nFurthermore, the paper primarily focuses on the final aligned LLM's performance. It would be beneficial to also analyze the accuracy of the reward model itself. For instance, examining whether the iterative training process with the relabeling strategy leads to a demonstrably better reward model could provide valuable insights into the effectiveness of the proposed approach. Evaluating metrics specific to the reward model would help disentangle its improvement from other factors that might influence the final LLM's performance.\n\n[1]: RIP: Better Models by Survival of the Fittest Prompts", "theoretical_claims": "NA", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "As I mentioned in method and evaluation criteria section. The author might need to discuss the literature on LLM training data filtering, an example:\n[1]: RIP: Better Models by Survival of the Fittest Prompts", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and easy to understand. The reported results, achieving near oracle-level performance with significantly reduced human annotation, are impressive and highlight the potential of the proposed framework.\n\nWeaknesses:\n\nAs discussed in the previous section, the paper could benefit from a more in-depth analysis of the reward model's performance and a comparison with a simple filtering strategy based on the absolute reward gap, as suggested by recent work [1].\n\nThe paper should address the potential increase in training time due to the iterative nature of the Sargy framework. The process involves not only labeling data using an LLM judge but also training a reward model on the fly. This additional complexity might lead to a more cumbersome and time-consuming training pipeline compared to traditional methods. Quantifying this overhead and discussing potential optimizations would be valuable.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqiOHxPUli", "reviewer": "Reviewer_BWsT", "summary": "This paper proposes Sargy, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations to achieve near-human annotation quality with minimal effort. The reward model's distribution is used to identify hard-to-annotate samples mislabeled. Then it iteratively enhances data quality by integrating strategic human corrections while leveraging LLM's correctly labeled samples.\n\n\n## update after rebuttal: Thank you for your answer. I keep my score as is.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and constructive review of our work!\n\n**Q1:** Other variants that are e.g., Greedy where the upper-left or bottom-right samples of the reward distribution are being annotated\n\n**Re:** This will indeed be an interesting factor to compare quantitatively in the final version. Given the time limit, we refer to Figure 2 and Figure 7 for a qualitative estimation. The accuracy curves show that:\n- For the \"upper-left\" area, most RM preferences are correct, and annotations do not introduce many changes to the next-iteration training\n- For the \"bottom-right\" area, <10% RM preferences are incorrect. Direct flipping is a more efficient way instead of annotation.\n\n**Q2:** Performance for more than 5 iterations\n\n**Re:** We have not included experiments after Itr-5 since the corresponding downstream LLM has already outperformed the one with full-human annotation and the human annotations have covered 20% of all samples. More annotations will gradually bring the RM closer to its counterpart in the full annotation setting (accuracy converges around the full-annotation accuracy).\n\nHowever, we have extended the experiments with 10 iterations under a 1/4 down-sampled shard of the full dataset (please kindly refer to the response to Q3 of Reviewer fzYM for more context). In each iteration, 4% of the subset (1% of the full set) receive human annotations. The test accuracy of the RM is listed as follows:\n\n| # Iteration | 1/4 Shard (HH-RLHF) | 1/4 Shard (TL;DR) |\n|-----|---------------------|------------------|\n| 0   | 78.4                | 81.0             |\n| 1   | 81.3                | 81.9             |\n| 2   | 84.6                | 83.8             |\n| 3   | 86.3                | 84.3             |\n| 4   | 87.6                | 85.2             |\n| 5   | 88.8                | 86.5             |\n| 6   | 89.6                | 87.4             |\n| 7   | 89.7                | 88.0             |\n| 8   | 90.3                | 88.3             |\n| 9   | 90.3                | 88.2             |\n| 10  | 90.8                | 88.4             |\n\nNote that here the size of the dataset is different from the one used in our original submission, therefore leading to different starting points and ceilings. \n\nWe will also include results for more iterations across more sharding options in our final version.\n\n**Q3:** Computational/cost analysis w.r.t. the baseline\n\n**Re:** We have added a thorough computational/cost analysis. Please kindly refer to the response to Q3 of Reviewer fzYM.", "claims_and_evidence": "1. Using the reward model's distribution to select hard-to-annotate samples\n- Yes\n2. An iterative reward model training technique to achieve oracle-level human alignment in the dataset.\n- Yes\n3. Sargy is implemented on HH-RLHF and TL;DR. Results show accuracy comparable to fully human-annotated oracle dataset while using 20% of the total human annotations.\n- Yes", "methods_and_evaluation_criteria": "Given an unlabeled preference dataset, Sargy integrates AI-generated labels with selective human feedback to maximize alignment while minimizing annotation effort. The first stage is the initial alignment. A prompt is used to generate the preferences. The second is iterative alignment improvement, where bad labels are improved. A reward model is trained iteratively with selective human annotations to enhance alignment. Then the key relies in analyzing the distribution of the predicted reward function within the training preference dataset. By ranking all preference pairs, a monotonic reward distribution curve emerges - the upper left region shows high agreement between training data and the reward model, and high disagreement for the bottom right. The latter ones are used for annotation. However, in practice the ground-truth labels are unknown and the authors propose to use either the elbow or the knee. The human annotation beings from the reflection point. For the next iteration, the authors propose two techniques to combine the datasets: back-off ratio and amplification ratio.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The proposed method is applied on HH-RLHF and TL;DR. Baselines are Random, and Oracle. I would appreciate if the authors include other variants that are e.g., Greedy where the upper-left or bottom-right samples of the reward distribution are being annotated. I would also encourage the authors to report performance for more than 5 iterations. Obtained performance are better than Random, which is not a surprise. Finally, the ablation study is sound.", "relation_to_broader_scientific_literature": "The method is novel.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Overall, the paper is very well written, the methodology is sound, and the results good. My only criticism would be to include more baselines in the experiments, and increase the number of turns. Interestingly, it would be good to have so computational/cost analysis w.r.t. the baseline.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How do the performance improve of 5 iterations?\nWhat are the cost of the proposed method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t37O7G6akR", "reviewer": "Reviewer_qWiJ", "summary": "This paper presents Sargy, a hybrid framework designed to align LLMs with human preferences by integrating LLM-generated annotations and selective human feedback. The framework operates iteratively, identifying erroneous samples through reward model distributions, prioritizing difficult cases for human annotation, and retaining accurate LLM labels. Experiments on HH-RLHF and TL;DR datasets show that Sargy achieves Oracle-level alignment quality with only 15-20% human annotations, while downstream models trained on Sargy-curated data perform comparably to fully human-annotated benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and constructive review of our work!\n\n**Q1:** The identification of \"elbow\" and \"knee\" points in the curve is heuristic and may not always correspond to clear boundaries between correctly and incorrectly labeled samples.\n\n**Re:** We acknowledge the practical concern that detecting \"elbow\" and \"knee\" points may not be highly precise. Therefore, in our implementation, Sargy treats these points as approximate boundary estimates, i.e., we don't need a precise boundary. Empirically, we observed that \"elbow\" and \"knee\" yield satisfactory results and slight adjustments to these estimations do not affect performance. We'll add the corresponding numbers in the final version.", "claims_and_evidence": "The claims are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes. This paper alleviates the challenge of fine-tuning LLMs to match user preferences, which is hindered by the high cost of high-quality human annotations in reinforcement learning human feedback (RLHF) and the limited generality of AI feedback. To overcome these issues, it propose a human-machine hybrid framework that leverages LLM-based initial alignment combined with selective human annotation to achieve near-human annotation quality with minimal effort.", "theoretical_claims": "The paper presents a methodological framework (Sargy). The claims made in the paper are primarily empirical, based on experimental results.\n\nIn these claims, the rationale for segmenting the reward distribution curve could benefit from greater rigor, as the identification of \"elbow\" and \"knee\" points in the curve is heuristic and may not always correspond to clear boundaries between correctly and incorrectly labeled samples.\n\nThe remaining claims (Sarge's iterative alignment improvement effectiveness and knowledge transfer effectiveness) are all supported by experimental data.", "experimental_designs_or_analyses": "1. Reward Model Iterative Improvement Experiment: Evaluated Sargy's iterative improvements on HH-RLHF and TL;DR datasets, showing it achieves near-oracle accuracy with only 20% human annotations, significantly outperforming random sampling. Experimental design is sound.\n\n2. Amplification Ratio Experiment: Investigated the impact of different amplification ratios on reward model improvement, finding that using a higher ratio initially and reducing it later maximizes annotation effectiveness and model performance. Experimental design is sound.\n\n3. Back-off Ratio Experiment: Analyzed the effect of different back-off ratios on data sanitization and model improvement, demonstrating that a higher ratio initially and reducing it later optimizes data quality and model performance. Experimental design is sound.\n\n4. Annotation Batch Size Experiment: Explored the impact of batch size per iteration on model improvement, validating that iterative annotation outperforms one-shot annotation, enhancing model efficiency. Experimental design is sound.\n\n5. Ablation Study: Verified the necessity of Sargy's components, including selective human annotation, amplification ratio, and back-off ratio, proving these are critical to Sargy's success. Experimental design is sound.\n\n6. Downstream Task Experiment: Used Sargy's curated preference dataset for Direct Preference Optimization (DPO), evaluating model performance on HH-RLHF and TL;DR downstream tasks, showing results close to the oracle model and significantly better than random sampling and initial models. Experimental design is sound.\n\nOverall, I find the experimental section well-designed and convincing, effectively supporting the paper's conclusions.", "relation_to_broader_scientific_literature": "1. Relation to RLHF and RLAIF: The proposed Sargy framework integrates the strengths of RLHF (Reinforcement Learning from Human Feedback) and RLAIF (Reinforcement Learning from AI Feedback), addressing the high cost of human annotations in RLHF and the limited generalizability of AI feedback in RLAIF. By introducing a human-AI hybrid annotation strategy, Sargy achieves near-human annotation quality with minimal human effort, aligning with ongoing research on effectively combining human and AI feedback.\n\n2. Relation to LLM Self-Improvement Methods: Sargy enhances LLM performance through iterative reward model training and selective human annotations. This approach resonates with recent advancements in LLM self-improvement (e.g., Self-Rewarding LMs and SELF-ALIGN) but distinguishes itself by incorporating human intelligence, overcoming the inherent limitations of LLM self-improvement, particularly in customized tasks, thereby advancing the field of LLM self-enhancement.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: My recommendation is to accept the paper.

The paper proposes Sargy, a workflow for augmenting LLM-labeled preference datasets with human annotations to obtain similar performance to using pure human annotation with substantially reduced human annotation effort. The process iteratively updates a reward model, by filtering data and obtaining additional human annotations. Reviewers generally agreed that the results were compelling. There were some questions raised about novelty wrt filtering methods, but the authors responded adequately to these questions.

Having looked over the paper myself, I would suggest that the authors include a succinct algorithm-box-like summary of the iterative procedure, particularly the construction of the score gap ranking curve at each step, as there is currently only long-form description of the steps. This would make it easier to understand the procedure and replicate the results. In particular, it seems like it would be easy to leak information from ground truth ranking labels (through the sign of the gap) if this construction is done incorrectly by readers.
Decision: Accept (poster)