Self Iterative Label Refinement via Robust Unlabeled Learning
Authors: Hikaru Asano
Tadashi Kozuno
Yukino Baba
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Weakly Supervised Learning
LLM Self-Refinement
Low Resource Domain
Primary area: general_machine_learning
TL;DR: We developed an iterative weakly supervised pipeline to refine LLM-generated pseudo-labels, consistently outperforming original LLMs and existing self-refinement methods across diverse datasets, while effectively supporting LLM safety alignment.

Abstract: Recent advances in large language models (LLMs) have yielded impressive performance on various tasks, yet they often depend on high-quality feedback that can be costly. Self-refinement methods attempt to leverage LLMs' internal evaluation mechanisms with minimal human supervision; however, these approaches frequently suffer from inherent biases and overconfidence, especially in domains where the models lack sufficient internal knowledge, resulting in performance degradation. As an initial step toward enhancing self-refinement for broader applications, we introduce an iterative refinement pipeline that employs the Unlabeled-Unlabeled learning framework to improve LLM-generated pseudo-labels for classification tasks. By exploiting two unlabeled datasets with differing positive class ratios, our approach iteratively denoises and refines the initial pseudo-labels, thereby mitigating the adverse effects of internal biases with minimal human supervision. Evaluations on diverse datasets, including low-resource language corpora, patent classifications, and protein structure categorizations, demonstrate that our method consistently outperforms both initial LLM's classification performance and the self-refinement approaches by cutting-edge models (e.g., GPT-4o and DeepSeek-R1). Moreover, we experimentally confirm that our refined classifier facilitates effective post-training alignment for safety in LLMs and demonstrate successful self-refinement in generative tasks as well. Our code is available at https://github.com/HikaruAsano/self-iterative-label-refinement.

Self Iterative Label Refinement via Robust Unlabeled Learning
Hikaru Asano
Tadashi Kozuno OMRON SINIC X Tokyo, Japan
The University of Tokyo
Tokyo, Japan
asano-hikaru1902 ecc. wotkoou.oo.......o.ooooo....oo.. jp.jp
tadashi kozuno@sinicx..co
Yukino Baba The University of Tokyo
Tokyo, Japan
yukino-babaog.een
ecc.baaaaaaa.......oo.ooooo.oocb ecc. wwt.c.uttton u-tokyo.ack jp.
Abstract
Recent advances in large language models (LLMs) have yielded impressive per- formance on various tasks, yet they often depend on high-quality feedback that can be costly. Self-refinement methods attempt to leverage LLMs' internal evalu- ation mechanisms with minimal human supervision; however, these approaches frequently suffer from inherent biases and overconfidence, especially in domains where the models lack sufficient internal knowledge, resulting in performance degra- dation. As an initial step toward enhancing self-refinement for broader applications, we introduce an iterative refinement pipeline that employs the Unlabeled- Unlabellded learning framework improve LLM-generated pseudo-labels for classification tasks. By exploiting two unlabeled datasets with differing positive class ratios, our approach iteratively denoises and refines the initial pseudo-labels, thereby mitigating the adverse effects of internal biases with minimal human supervision. Evaluations on diverse datasets, including low-resource language corpora, patent classifications, and protein structure categorizations, demonstrate that our method consistently outperforms both initial LLM's classification performance and the self-refinement approaches cutting-edge models (e.g.. GPT-4o and DeepSeek- RI). Morcover, we experimentally confirm that our refined classifier facilitates effective post-training alignment for safety in LLMs and demonstrate successful self-refinement generative tasks as well
Introduction
Rapid advancements in large language models (LLMs) have yielded significant improvements across various downstream tasks and have raised fundamental research question: How can we further improve LLM's 's capabilities with minimal human supervision? ? Traditional approaches, such as
Reinforcement Learning from Human Feedback (RLHF) 440 and its variants [42 53), improve performance through classification tasks (53) yet they rely extensive, costly, and time-consuning human-labeled datasets. Although recent methods like Reinforcement Learning from Al Feedback
(RLAIF) 3 29 67 aim to reduce these costs by replacing human annotations with godel-generated
signals, their success critically hinges on the reliability of the model's I's self-evaluation 30 58
OOur code is available sittgpe//ghhaa..ccm/iiiaaaaaaaaa
-abeeeeee-aab------aeee label-ref
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

When using an LLM as its own evaluator, it is observed that the model's inherent biases can harm the
reliability of its assessments 24 31 34 58 thereby undermining the effectiveness of downstream training [4]. While iterative self-refinement 23
37 and multi-agent frameworks [16] can mitigate simpler biases (e.g., ordering or length biases), L.MMs still encounter significant challenges in areas where their internal knowledge is limited [21]. In such cases, external tools, such as accurate, domain-specific databases, can help address certain blind spots [6]] but they do not fully eliminate the need for human supervision. Without either robust internal knowledge or dependable huge human input, conventional methods not only struggle to improve performance [18 [1] but also may even experience degradation due to inaccuracies and overconfidence [8]
To address these challenges specifically for classification tasks, a crucial first step towards broader self-refinement in LLMs, we introduce an iterative pipeline that refines LLM- --M-eeerrtted pseudo-labels using a weakly supervised learning technique known as the Unlabeled- Unieedd---d (UU) learning framework B5 36 Notably, while creating a large, well-maintained labeled dataset requires intensive human supervision, it is relatively straightforward to amass a vast corpus of unlabeled data in the modern era 45 Motivated by this observation, our approach leverages two unlabeled datasets with differing positive class ratios. Under the simple assumption that one dataset contains a higher proportion of positive examples than the other, our system can effectively learn to distinguish between positive and negative instances without requiring explicit annotations for each example.
Specifically, our pipeline first employs an LLM to generate initial pseudo-labels from an unlabeled corpus. These aseudo-positive and negative sets are then iteratively refined using UU learning. The classifier trained from UU learning subsequently re-labels the unlabeled corpus, progressively reducing noise and enhancing classification accuracy. By decoupling the refinement process from the LLM's internal knowledge and instead leveraging data-driven features extracted via UU learning, our method delivers improved performance even in domains where LLMs lack sufficient knowledge.
We evaluate our approach on several public datasets, including low-resource language corpora. patent classification tasks, and protein structure classification. Notably, as illustrated in Figure 3 even in cases where self-refinement methods based on LLMs, or advanced reasoning models such as DeepSeek-RI [3], fail produce any performance improvement, our iterative UU learning framework successfully refines its outputs and achieves classification performance that surpasses that of both the original L.L and existing self-improvement pipelines. Moreover, we experimentally show that our refined classifiers, integrated into RLAIF frameworks, effectively achieve safety alignment without extensive human annotation. This underscores our method's potential for comprehensive, robust LLM self-refinement.
In summary, our contributions are threefold: (i) We introduce an iterative pipeline that refines LLM-generated pseudo-labels via UU Learning, reducing noise and boosting classification accuracy with minimal human supervision; (ii) we demonstrate our method consistently surpasses direct LLM classification and existing self-improvement methods across diverse tasks (c.g., low-resource languages, patents, proteins), enabling scalable, high-quality classification with limited labeled data; and (iii) we experimentally show that our refined classifier facilitates effective post-training alignment for L.MSs and highlight its potential for self reff-refineeent in broader generative tasks.
2 Related Work
RLAIF is a popular LLM post-training method 3]. Its idea to generate feedback by a model itself is called Pseudo-Labeling (PL) and well studied in semi-supervised learning 28 48 Below, will review PL and LLM self-training methods related to ours. For a thorough review of each topic, please refer to Yang et al. [63] and Xiao and Zhu [62].
Pseudo-Labeling: PL trains a student model so that its output is close to the output of teacher model on unlabeled data 128 48). Various methods for constructing a teacher model has been
proposed. For example, II-model 2 27 [6] and virtual adversarial training (39] perturbs input and/or neural networks of student models. Some methods use weighted average of previous student models' predictions or weights as a teacher model (7 55 and other methods even try to optimize a teacher model by viewing PL as an optimization problem 41
Our work is complimentrry to this line of works since what modify is the risk estimator rather than teacher models.
57 65

---

PL is known to suffer from erroneous labels generated by model mooellinn-rraniie in-training [7 43 observation well aligns with recent reports on RLAIF that erroneous self-feedback
and this
major source of failure 37 A straightforward approach is to filter potentially incorrect labels based on confidence
17 20
52 or the amplitude of loss as in self-paced learning 6 20 25 Another method
refining pseudo-labels in way similar to label propagation [68] but with similarity scores computed
using neural networks [26]. Our work tackles the issue of erroneous labels by using a risk estimate robust to erroneous self-feedback based on UU learning [35 36). Even though our approach requires only a minimal change, we observed a significant performance boost.
LLM's Seff-Refinement: To enhance the reasoning capabilities of LLMs, early efforts primarily
explored various prompt engineering techniques [8 59 60 64). Even with refined prompting strategies, an LLM's initial outputs can remain limited in some scenarios. Recent studies have proposed iteratively improved answer strategies called self-refinement approaches [1] 23 37 and our work falls within this lineage.
Self-refinement involves generating responses through agents in distinct roles that iteratively provide feedback [50, 699. For example, agget debate frameworks [66 an answering agent and an
evaluation agent to collaboratively improve answer quality [10] 15 511. However, these methods usually assume that the L.M has enough internal knowledge to generate effective feedback and revisions; when doesn't, performance gains can be minimal or even negative [18 21 22 [2], sometimes degrading performance [18]. Our approach minimizes this reliance: internal knowledge is used only initially for classification, with subsequent improvements relying on extracting features directly from the data via UU learning.
Other work addresses knowledge gaps by retrieving external data or using external tools 19 56 61]. though such setups can be costly. In contrast, our method needs only a small amount of labeled data for initialization, without relying on external resources.
3 Preliminaries
3.1 Supervised Binary Classification
In many real-world tasks, commonly encounters binary classification problems, in which input x e R2d is presented, and its label y E {+1} needs to be predicted. Each sample is assumed to be independently and identically drawn from unknown joint distribution p(ar, y). Let p(y 1) be the prior probability of the positive class (positive prior), and define
Pp(x) = p(x y = +1), Pa(x) p(= y = -1).
Then, the marginal distribution is given by
p(x) =+pp(*) +
(011-+)ppml (1- (n)p(())
A classifier Rd + R outputs a real-valued score, whose sign determines the predicted label. For instance, neural network can serve as g. loss function L : R x { + [0, oo) then measures how much the prediction disagrees with the true label. Let R (g) = (t +1]] denote the
loss for true positive data, and R,7, (g) Empp, [((g(2), 1}]] denote the loss for the true negative data. Then, the true risk is expressed as
Ren (g) = [R ((g(()))))))))) =x+ R + (1 - 1 TT) R
In supervised learning, positive dataset Cp = { mp 1 far, 1m ~ (x) are accessible. Replacing the expectations in [I) with sample mean, one obtains the empirical risk, and g is trained to minimize it.
(x(x) and negative dataset C,
is well known that having sufficient positive and negative samples typically allows one to train a highly accurate classifier for many tasks. However, in practice, obtaining large-scale positive and negative datasets with annotations is often challenging, especially in specialized domains where annotation costs become significant obstacle.
3

---

3.2 Unlabeled-Unlabeledd (UU) Learning
UU learning 35| is a technique that allows training a classifier without fully labeled positive and negative datasets, leveraging unlabeled datasets with different class priors.
Concretely, suppose unlabeled corpora, C. ( h and C. {I 1 drawn from different mixture distributions. denote O ply = +1 I E Cp) and 0, = ply +1 C,) the positive prior of these unlabeled corpora. In other words, 0p is the fraction of true positives in Ep. and On is the fraction of true positives in C.,. Then, the mixture distribution of each corpus is given as p₂(x) = (p)) + (1 0p) Pm(x), (x) = 6n (p + - 0x) Pa (r.).
When 8, 0ny treat C. as a pseedo-possee positive corpus (due its larger proportion of actual positives) and C, as neeudo-negative corpus (having smaller proportion of actual positives).
By appropriately combining these two unlabeled sets, one can construct an unbiased estimate of the true binary classification risk ④ Specifically, let R (g) E0,0= (<]((,,, and R (g)
Ex-p. (((((x),) Then, the UU learning risk given by
(g) aR (g) bR (g) cR (g) + dR, (g),
where the coefficients c, are computed from + pp. and 8, as a (1- 11 0g-0,-
0, (1 -
0g 0,
c (1-8,)*-
8,-0, 8, (1-#+) When 8p = and 0 = 0, that is, when using the same dataset as 6--0,
standard supervised learning, equation (2 reduces to the standard supervised learning risk equation ①1) In other words, supervised learning can considered special case of UU learning.
3.3 Robust UU Learning
While UU learning (2) does allow model training without explicit positive/negative labels, comparing the original binary classification risk (I). which remains nonnegative, against the UU risk (2) shows the UU risk includes negative terms such as -6R2 (g) and -clt (g). It has been observed that these negative risk terms can lead to overfitting [36].
To mitigate this, Robust UU Learning introduces generalized Leaky ReLU function to moderate the reduction of negative risk. Concretely, it normalizes each term of the loss function as (3
R (g) f cR) + (dR, (g) bR (g)
where each bracketed term resembles "normalized" risk under the hypothetical label of being positive or negative, respectively. The function f is defined as f(x) = x if O, and f(x:) = Ar < O, where À < 0. Intuitively, f preserves positive risk values while converting negative risk into positive values using À < 0, thereby reducing overfitting caused by negative risk terms.
4 Method
Below, we present our iterative framework for combining LLM annotations with robust UU learning. The goal is to iteratively refine pseudo-labels generated by the LLM,, thus boosting classification accuracy under minimal human supervision. Figure provides an overview of this pipeline.
4.1 Overview of the Iterative Pipeline
Our pipeline proceeds in three steps, (i) LLM-Based Annotation (Iteration 0): an LLM provides pseudo-labels for an unlabeled corpus ($4.2); (ii) Robust UU Learning: we split the corpus into pseudo-positive and pseudo-negative subsets and train a classifier via robust UU learning ($4.3); and (iii) Re-Labeling: the trained classifier re-labels the entire dataset, producing refreshed pseudo- positive and nseudo-negative sets for the next iteration ($4 (4.4)

---

Step1: LLM-Based Annotation
Step2: Robust UU Learning
Step3: Re-Labeling
Unlabeled corpus
Mini Mini-balch from pseudo posilive corpus
Mini-batchh from pseudo negalive corpus
Unlabeled corpus
00000000
00000000
R
R
R
annotator
Classifier
Build robust UU loes Ryuu by waighing and normalizing
Classifier
Noisy pseudo
Noisy pseudo negalive corpus
Denoised pseudo posilive corpus
Denoised pseudo negallive corpus
pasilive corpus
Train classiffer minimizing robust UU loss
Unobserrvec ground
O0
Figure I: Overview of our iterative refinement pipeline. First, an LLM annotator generates initial pseudo-labels for an unlabeled corpus, dividing it into pseudo-positive and pseudo-negative corpora. Next, we train classifier using robust UU learning these pseudo corpora, yielding a model that outperforms the initial LLM annotations. Finally, the classifier re-labels the entire dataset, updating the pseudo-labed for the next iteration. Repeating this cycle gradually refines the pseudo-labels, leading to increasingly reliable labels.
4.2 Initial Noisy Annotation via LLM
Let C {I1, (21.42,.... T2....... , EN be corpus of unlabeled samples. We use the LLM as the initial classifier to assign an initial pseudo-label ÿ E [+ to each sample z₁. Our prompt first provides concise description of the task, the dataset domain, and the expected answer format (e.g.. "Output TRUE or FALSE"). We then give a few-shot examples illustrating how to label an example text, along with a short rationale. Finally, the prompt includes the samples to annotate (see Figure the exact prompt).
Based the LLM's output, form unlabeled corpora:
-h = {xa ū +13, { xx;
7 -
These sets are called pseudo-positive and pseull-negatiie corpus, respectively. Although the labels noisy, Cp (00) typically has higher positive prior than C thereby providing reliable foundation for UU learning in subsequent iterations.
4.3 Refinement with Robust UU Learning
Let C 1)- and () denote the pseudo-positive and pseudo-negative sets from iteration t - 1. Our goal is to train classifier git)) (e.g.. neural network) despite noisy labels. To this end, we optimize
the robust UU learning objective:
( argmin Rruu (g: C5 1))-1), ((
where (-) is the empirical risk of robust UU learning, which applies "generalized leaky ReLU" to reduce the impact negative risk terms that can arise from mislabels. Each set is weighted by the positive prior π and the sets' own estimated positive priors 0, and θ,,
This robust learning approach is less sensitive to initial label noise and can produce classifier that outperforms the previous iteration's 's classifier.
4.4 Iterative Re-Labeling and Convergence
After training ("), we re-label the entire dataset: (t) == sign (ign(g(9 (x:)) E (+1,-1}. From
these labels, we form updated pseudo-positive 2( { [x: 5: = +1} and pseudo-negative
a(t) = y
(1) -1} sets, used in the next robust UU learning iteration.
5

---

Over several iterations, this process progressively improves the reliability of the pseudo-labels. In the ideal scenario, the positive prior in CP converges and the positive prior in Ci converges to O, bringing each corpus ever closer to the gold-standard case of perfectly labeled positive and negative data. When these priors reach and 0, respectively, robust UU learning effectively reduces to standard supervised learning, achieving high accuracy even from initially noisy labels.
Experiments
We conducted experiments to explore three main research questions:
RQ1 Can our iterative refinement approach improve classification performance, compared to the initial LLM-basee annotations, for various NLP tasks?
RQ2 Can our approach enhance performance on challenging tasks where even advanced LLMs (e.g.. GPT-4o or DeepSeek-R1 struggle?
RQ3 Can method extend beyond classification tasks generative tasks such as LLM alignment?
5.1
Experimental Setup
We use six binary classification datasets grouped into two categories based on their difficulty[ Table 2 reports the dataset statistics, and Table 3 provides examples for positive and negative cases (see Appendix (B).
Datasets:
We evaluated our algorithm on three tasks: (i) Fake News [. classifying news articles fake or real, (ii) Saroco [44), Romanian satire detection dataset to assess effectiveness in low-resource language, and (iii) Safety [12), dataset for SafeRLHF evaluating if the responses to questions safe dangerous, thereby assessing effectiveness for LLM's post-training.
Easier Tasks (for RQ1):
We evaluate our algorithm on three more challenging tasks: (i) the Corona Sentiment classifying social media post related COVID-19 19 positive negative sentiment, (ii) the Green Paten which involves identifying whether a patent abstract pertains to green plastics, requiring high expert knowledge, and (iii) Protein Structure [7], involving classification of proteins based on high or low COVID-19 binding affinity from molecular SMILES strings.
Harder Tasks (for RQ2):
For all experiments, we randomly partitioned each dataset into training, validation (for best epoch selection), and test splits in a 7:1:2 ratio.
LLM-Based Annotation (Iteration 0): At iterationO, we obtain pseudo-labels from LLMs using a prompt that includes a dataset explanation, few-shot labeled examples, and a target sample (see Figure 5 for the exact prompt). These pseudo-labels are then used to construct initial pseudo-positive (c)) and pseudo-negative (c()) corpora.
Training Procedure: We train the classifier by appending affine layer to the transformer's final hidden state to yield a one-dimensional score. For fine-tuning efficiency, we employ QLora [14 with 4-bit quantization. At each iteration, we fine-tune the model using the pseudo-positive and pseudo-negative corpora with the Adam W optimizer (learning rate = 1.0 x 10-4, batch size = 16, and 3 epochs) and fix the robust UU learning hyperparameter À at -0.001. At the end of each epoch, we compute the loss on a pseudo-labeled validation set and select the model with the lowest loss to re-label the entire dataset. Additional parameters are detailed in Table
Robust UU learning requires estimates of the positive proportions 0, and 6, for the pseudo- positivve and pseudo-negative corpora, respectively. We compare two settings to obtain these parameters. In the first setting (Ours (Oracle)), we use the exact values of Op and
Estimating Class Priors:
We evaluate difficulty based on pilot experiment in which we trained the classification model in standard supervised setting. Please refer to for classification accuracy via supervised learning.
Sentinent- Analysis and EDA/ htl/tree/nastee /naster
green, green_plaetice

---

Gemmar22 2b
LL-2-7b-chat
Mata-LL-3
1L-32-1b-inst
L1-3 2-3b-inst
Thenstoon
Dleratonn
Ieeaaton
Beration
Sar000
Derrttio
Dolation
Beration
Baration
Fanation
CDP
(Oracle)
(50-lbbeled)
Ours (100-abeled)
Figure 2: Classification accuracy over five iterations for three datasets. The solid lines represent the mean values, and the shaded areas show the mean + standard deviation. Both variants of our approach, Ours (Oracle) and Ours (few-labeled), demonstrate steady improvements as the iteration increases. Notably, even in scenarios where the baselines fail to learn the classification task, our method continues exhibit iterative performance gains. This robustness highlights the strength of our iterative refinement strategy, even under minimal supervision settings like 50 labeled examples. Detailed numerical results are provided in Appendix D
On. which represents a theoretical upper bound on performance In the second setting (Ours (few-labeled)), we assume a practical scenario with only small subset of labeled data (50 100 examples). From these examples, and 6, ply 5 0) before applying robust UU learning. Our experiments show that even small amount of labeled data provides sufficiently accurate estimates.
estimate the class priors by computing 6, p(y ÿ 1)
Evaluation Metrics and Protocol: We measure classification accuracy on the held-out test dataset across all iterations. Each run is repeated with three different random seeds, and we report the mean and standard deviation.
5.2 Experiments Easier NLP Tasks (RQ1)
We start with three relatively simple tasks; Fake News, Saroco, and Safety. For the base model of the classifier, we use Llama-3.2-IB--Inttee with only 1 billion parameters.
compact language model that offers strong performance
Annotation Model: generate initial pseudo-labels, we employ five open models of varying sizes and model families: gemma-2-2b-ir (2B) Llama-2-7b-chat-hi (7B Meta-Llama-3-88 (8B) ? Llama-3.2- B-Inrructt (1 10 and Llama-3.2-31 Jla-aa-.2-33--Instuuuc (3B [I This diverse selection helps ensure reproducibility and allows to evaluate whether our self-refinement approach operates robustly across different model families and scales.
'The assumption that the precise class priors are known is not unrealistic given the effectiveness of existing class prior estimation methods (c.g., 33 47)).
httpe://hugqiingfaee http:///wwuaaaaaaecccoooe//aaaaa co/neta-llamn/in -1aee-lss..aa/aaas...... 3.2- 3.2-11BInsttzuee
Instruct
attpe:/huuggiggfeee btt.ee//wgginngg......./ssssssss

---

Baselines: We compare our approach against three base- line methods that cover different learning paradigms:
Table 1 Overview of experimental base- lines and their categories.
1. PN: Standard supervised training on pseudo-labels, treating them as fully reliable.
Method
Category
PN PIE
Vanilla Supervised
2. PIE [66]
Iterative method that accepts high- confidence predictions as correct labels, progressively training ensemble models these provisional labels to iteratively improve classification accuracy.
Weakly Supervised
CCP
26
Semi-Superrised
Ablation (No Robust)
Ours (Oracle)
Ablation (Ceiling)
3. CCP [26]: Semi-supervised method using iterative contrastive learning construct reliable pseudo-labels by learning robust class-specific feature representa- tions from a limited set of labeled data.
Ours (50-labeled)
Proposed Proposed
Ours (100-labeled)
We also conducted an ablation study comparing standard UU learning (2 against our robust UU learning with various prior estimation methods. Table summarizes all methods.
Figure shows the results for the easier tasks. All three variants, Ours (Oracle), Ours (50- labeled), and Ours (100-labeled), show steady accuracy gains and reach high performance by the final iteration. Notably, the two limited-label settings quickly converge to the Oracle upper bound, yielding comparable final accuracies despite relying on only 50 100 labels. This outcome aligns with the robustness of UU learning to class-prior estimation errors (35). enabling strong performance under severe label scarcity and underscoring the method's practicality in
Result:
scenarios.
For the Fake News task (using Meta-Llama-3--0 matching the performance of Ours (Oracle) and outperforming PN Learning. This highlights the effectiveness its ensemble strategy and confidence-based label refinement. However, for Saroco and Safety, where PN Learning struggles, PIE similarly performs poorly, suggesting that even with confidence- based filtering, limited initial classification accuracy hampers noisy labels elimination.
PIE shows iterative performance gains, eventually
Similarly, while CCP shows performance gains with increasing iterations the Fake News task, it completely fails on Saroco and Safety. CCP's reliance on learning class-specific features from a small teacher dataset appears insufficient where such features are inherently difficult to capture.
The UU baseline also improves with each iteration. However, its peak accuracy consistently falls short of oracle and few-labeled variants of our approach, confirming the advantage of our robust correction for noisy pseudo- labels.
While our method demonstrates strong performance across diverse tasks, its gains can be limited when the initial pseudo-labels are highly noisy, causing the class priors of the pseudo-positive and pseudo-negative sets to be nearly identical. As shown in Figure 2 the Fake News task (using Gemma-2.2b) exhibits restricted improvements, starting with an initial annotation accuracy of 0.591. Conversely, the Saroco task (Llama-3.2-14 achieves clear iterative gains despite similarly low initial accuracy (0.576). This suggests that while the initial noise level influences performance, does not solely determine the success of the refinement process.
Summary for RQ1: These results confirm RQ1 by demonstrating that our iterative refinement approach consistently improves classification performance over initial LLM-based annotations on easier tasks, even with limited human supervision.
5.3 Experiments on Harder Tasks (RQ2)
We next evaluate our approach on three more challenging tasks: Corona Sentiment, Green Patent, and Protein Structure. To further explore the potential of our method, we employed Llama-3.2- Ins-nntt for both Corona Sentiment and Green Patent and utilized bert-base-smiles for Protein Structure, Initial pseudo labels are generated using the nigh-perffrmance closed models GPT-4o- min and GPT-40
httpa://nugginngaaee http://g........ooo/.///a//rrrra base- sm--smmelle
http://plaatttro.opaa.... wtt./oo///ccoooo howpaa/cc//////////oo/eegggg 4o-nnn
sttps:/plaatfozz openai. cww.///cce//oeeeeggtttteee

---

Baselines: To compare the performance of self-refinement under a minimal human supervision setting, we adopt a self-refinement framework [T] 23 37], where response agent generates an initial answer and feedback agent generates the feedback for this answer, thus iteratively refines answer (see Figures and for exact prompts). In this setup, we leverage high-performance closed models (GPT-40-mini and GPT-40) alongside the cost-effective reasoning model DeepSeek-RI [13 For GPT-4o-mini and GPT-4o, initial annotations are generated consistently with our iterative robust UU learning (Ours) to ensure fair comparison.
Figure illustrates the accuracy curves across five iterations for the three chal- lenging datasets. For all tasks, Ours (Oracle) shows a steady improvement in classification ac- curacy over successive iterations. In the Corona Sentiment and Protein Structure tasks, Ours (few-labeled) starts with low classification ac- curacy relative to Ours (Oracle), but this gap diminishes with additional iterations. In addi- tion, despite a lower initial performance than DeepSeek-Re RI for Corona Sentiment and Protein Structure, our method surpasses DeepSeek-RR R1' 's performance, demonstrating robustness against noisy initial labels.
Results:
GPT-4o-mini
GPT-4o
Corona Sentiment
0.8
0.80
0.90
0.75
0.70
Iteratoo
Meration
0.85
Green Patent
0.80
0.00
0.75
0.75
0.70
0.70
0.6
0.65
0.60
0.60
lteration
Iteration
In contrast, the LLM-based self-refinement ap- proach by GPT-4o-mini and GPT-4o shows only slight performance gain in the Corona Senti- ment task and suffers from performance degra- dation in the Green Patent and Protein Structure tasks, where accuracy actually worsens over iter- ations. Similarly, although DeepSeek-B R1 starts with high annotation scores on all three datasets, its performance plateaus, exhibiting no gains in subsequent iterations. These results suggest that even when employing a sophisticated rea- soning model, the benefits of self-refinement are limited when the LLM's internal knowledge is insufficient to correctly evaluate and revise its own outputs. Therefore, relying solely on self- refinement in these challenging domains may not lead to further performance gains and can even be counterprodductive.
Protein Structure
0.80
075
8:20
0.70
0.65
0.60
0.65
0.55
Iteration
Iteration
Self-fhfimmmant
DeepSeekk
Swif Baflreeete
Ours (Oracle)
(5D4abeled)
(100-labeled)
Figure 3: Classification accuracy curves over five iterations on three challenging datasets. Ours (Or- acle) uses the exact class prior for UU learning, while Ours (few-labeled) estimates these priors from only 50 labeled examples. Our method con- sistently improves accuracy and outperforms both LLM self-refinement by GPT-4o series and ad- vanced reasoning model DeepSeek-R1. Detailed numerical results are provided in Appendix D
Summary for RQ2: These experimental find- ings answer RQ2: While self-refinement with advanced LLMs like GPT-4o and DeepSeek-RI fails to improve or even degrades performance on challenging tasks, our method consistently enhances accuracy through iterative refinement, ultimately outperforming these strong baselines.
5.4 LLM Alignment on Safety (RQ3)
Setup: We reuse the Safety dataset generative benchmark for safety alignment, employing a RLHF approach. The reward model (RM) was a classifier trained by our robust UU pipeline: we use 11ama-3. classifier. This refined classifier then functioned as the RM.
which provides initial pseudo-labels and the base model for the
As the base policy for RLHF, we applied supervised fine-tuning (SFT) to 11ama-3..2222 3.2-1B on the Alpaca open-source dataset [54] By intentionally using models from the same 1lama-3. 2-BB series for the base model, pseudo-label generation, and the reward model, we aimed to evaluate the potential of self-refinement in generative tasks. See the Appendix C.2 for experimental details and our training and evaluation procedure following [12].

---

Baselines: We compare four systems: (i) SFT, (ii) Vanilla RLAIF that employs an RM trained only with PN learning, (iii Ours (Oracle) that uses the true priors, and (iv) Ours (50-labeled) that estimates class priors from 50 labeled examples. All experiments are repeated with three random seeds; we aggregate reward distributions over the test prompts.
Results: Figure 4 shows that while the SFT policy still produces a noticeable fraction of negative-reward (unsafe) outputs, both our ora- cle and few-labeled variants dramatically reduce this undesirable left tail, concentrating the prob- ability distribution within the safe region. In contrast, Vanilla RLAIF fails to achieve mean- ingful improvement, performing worse than the baseline SFT, which emphasizes that noisy re- ward signals severely hamper RL optimization. The robust reward modeling provided by our UU-refinmment strategy delivers a stable, noise- resistant learning signal, effectively guiding the generative model toward safer and more desir- able behavior.
0.16
SATE
1-11
0.14
-dio Dun (83 kooked) fnoce 6.11)
(oron jroas 9
Probability Denstty
0.10
0.08
0.06
0.04
0.2
0.00
-20
-10
10
20
Reward
Summary for RQ3: These findings answer RQ3: the improved classification performance achieved through our iterative robust UU learn- ing strategy successfully translates the more challenging generative alignment task. This re- sult confirms that our approach provides prac- tical and highly effective self-refinement mech- anism, beneficial not only in classification but also in complex generative settings.
Figure Reward distribution generated answers on the Safety dataset after alignment. Both vari- ants of our method, Ours (Oracle) and Ours (50- labeled), shift the distribution toward higher (safer) rewards compared with the SFT baseline and the Vanilla RLAIF. Legend values denote the mean reward across three random seeds.
6 Conclusion
We proposed an iterative refinement pipeline leveraging robust UU learning with minimal supervision, which consistently outperforms LLM annotations and existing self reffrrefineeen approaches across six classification benchmarks. Notably, our method achieves performance comparable to oracle-pric settings using only 50 labeled examples. It also demonstrates significant effectiveness in generative tasks, successfully enabling RLAIF on the Safety dataset, where naive approaches previously failed. Our approach offers a simple yet powerful means to enhance LLM performance, particularly in low-resource and complex domains, with substantial potential for aunotation-intensive applications such advanced L.MM seff-refinement and Al for Science.
Limitations and Future Work. Although our method is robust, its effectiveness can be limited by extreme initial pseudo-label noise. Our findings indicate that performance depends not only on the quality of initial labels but also significantly on other contextual factors. A promising direction for future work involves explicitly modeling instance-level classifiabilit,, acknowledging the variable difficulty between task formulations, such as chat-style judgments by language models versus predictions from trained classifiers, an example easy for one may prove challenging for the other. Utilizing such instance-specific difficulty metries to inform weighting, sampling, and curriculum strategies could prioritize reliable examples while incorporating challenging yet informative cases. Additionally, augmenting classifiers with auxiliary information, such as rationales, retrieved contexts, or lightweight metadata, could further improve classification accuracy for ambiguous texts.
Acknowledgments
This work has been supported by the JST Moonshot Research and Development Program JPMJMS2236-8.
10

---

References
[1] Hadeer Ahmed, Issa Traore, and Sherif Saad. Detecting opinion spams and fake news using text classification. Security and Privacy, 1(1):e9, 2018.
[2] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2014.
[3] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Tellcen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from feedback, 2022.
[4] Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. Best-of-venom: Attacking RLHF by injecting poisoned preference data. In First Conference on Language Modeling, 2024.
[5] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: survey. Machine learning, 109(4):71--760, April 2020.
[6] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the International Conference on Machine Learning (ICML). 2009.
[7] Andrew E Blanchard, John Gounley, Debsindhu Bhowmik, Mayanka Chandra Shekar, Isaac Lyngaas, Shang Gao, Junqi Yin, Aristeidis Tsaris, Feiyi Wang, and Jens Glaser. Language models for the prediction of SARS-CoV-2 inhibitors. The International Journal of High Performance Computing Applications, 36(5-6):587-602, 2022.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin, editors, Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[9] Hui Chen, Wei Han, and Soujanya Poria. SAT: Improving semi-supervised text classification with simple instance-ate aitaacce-ddapive telf-training. In Yoay Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6141-6146.
Association for Computational Linguistics, 2022.
[10] Justin Chen, Swarnadeep Saha, and Mohit Bansal. ReConcile: Round-table conference improves reasoning via consensus among diverse LLMs. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pages 7066-7085, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[11] Xinyun Chen, Maxwell Lin, Nathanacl Schärli, and Denny Zhou. Teaching large language models to self-debug, In The Twelfth International Conference on Learning Representations, 2024.
[12] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In Proceedings of the International Conference Learning and Representation (ICLR), 2024.
11

---

[13] AL DeepSeek- RI: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv [cs.CL], January 2025.
[14] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zeulemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[15] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the International Conference on Machine Learning (ICML), ICML'24, Vienna, Austria, 2024. JMLR.org.
[16] Andrew Estornell and Yang Liu. Multi-LLM debate: Framework, principals, and interventions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[17] Christian Haase-Schutz. Rainer Stal, Heinz Hertlein, and Bernhard Sick. Iterative label improve- ment: Robust training by confidence based filtering and dataset partitioning. In International Conference on Putlern Recognition (ICPR), 2021.
[18] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xiny- ing Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In Proceedings of the International Conference on Learning and Representation (ICLR), 2024.
[19] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of the Conference on Robot Learning (CoRL), 2022.
[20] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curriculum learning. In Proceedngs of the AAAI Conference on Artificial Intelligence (AAAI), 2015.
[21] Ryo Kamoi, Sarkar Snigdha Sarathi Das, Renze Lou, Jihyun Janice Ahn, Yilun Zhao, Xiaoxin Lu, Nan Zhang, Yusen Zhang, Haoran Ranran Zhang. Sujeeth Reddy Vummanthala, Salika Dave, Shaobo Qin, Arman Cohan, Wenpeng Yin, and Rui Zhang. Evaluating LLMs at detecting errors in LLM responses. In First Conference on Language Modeling (COLM), 2024.
[22] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs actually correct their own mistakes? critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics (TACL), 12:1417-1440, 2024.
[23] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In A Oh, T Naumann, A Globerson, K Saenko, M Hardt, and S Levine, editors, Proceedings of the Anmual Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 39648-300 39677. Curran Associates, Inc., 2023.
[24] Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguisties: ACL 2024, pages 517-545, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[25] M. Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2010.
[26) Brody Kutt, Pralay Ramteke, Xavier Mignot, Pamela Toman, Nandini Ramanan, Sujit Rokka Chhetri, Shan Huang, Min Du, and William Hewlett. Contrastive credibility propagation for reliable semi-superiised learning. In Proceedngs of the AAAI Conference on Artificial Intelligence (AAAI), 2024.
(27) Samuli Laine and Timo Aila. Temporal ensembling for sumi-supe learning. In Proceedings of the International Conference Learning and Representation (ICLR), 2017.
12

---

[28] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. International Conference on Machine Learning (ICML), 2013.
[29] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHE: scaling reinforcement learning from human feedback with Al feedback. In Proceedings of the s1 International Conference on Machine Learning, ICML'24, Vienna, Austria, 2024. JoLR.org.
[30] Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang. Weidong Guo, Yukang Gan, Xu Yu, Daniell Wang, and Ying Shan. HRLAIF: Improvements in helpfulness and harmlessness in open-domain reinforcement learning from Al feedback. arXiv [cs, March 2024.
[31] Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Opportunities and challenges of LLM-aasa-juudge. arXiv /cs.AI], November 2024.
[32] Yanhong Li, Chenghao Yang, and Allyson Ettinger. When hindsight not 20/20: Testing limits on reflective thinking in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguisties: NAACL 2024, pages 3741-3753, Mexico City, Mexico, June 2024. Association for Computational Linguisties.
[33] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting, IEEE Trans. Pattern Anal. Mach. Intell., 3873):447 461-461,11116 March 2016.
[34] Yiqi Liu, Nafise Moosavi, and Chenghua Lin. LLMs as narcissistic evaluators: When ego inflates evaluation scores. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12688- 12688-12701, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[35] Nan Lu, Gang Niu, Aditya K Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. In Proceedings of the International Conference Learning and Representation (ICLR), 2019.
[36] Nan Lu, Tianyi Zhang, Gang Niu, and Masashi Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: consistent risk correction approach. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Staristics (AISTATS), volume 108, pages 1115-1125, 2020.
[37] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: lterative refinement with self-feedback. In A Oh, T Naumann, A Globerson, K Saenko, M Hardt, and S Levine, editors, Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 46534-46594 Curran Associates, Inc., 2023.
[38] Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class- clasabiro estimation. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 125-134, Lille, France, 2015. PMLR.
[39] Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 199991197911999 2019. doi: 10. 10.1...... 2858821.
[40] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human
13

---

feedback. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), volume 35, page 27730-27744, 2022.
[41] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V. Le. Meta pseudo labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[42] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[43] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In Proceedings of the International Conference on Learning and Representation (ICLR), 2021.
[44] Ana-Cristina Rogoz, Gaman Mihaela, and Radu Tudor lonescu. SaRoCo: Detecting satire in a novel Romanian corpus of news articles. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), Online, August 2021, Association for Computational Linguistics.
[45] Seref Sagiroglu and Duygu Sinanc. Big data: A review. In 2013 international conference on collaboration technologies and systems (CTS), pages 42 47. IEEE, 2013.
[46] Mehdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic trans- formations and perturbations for deep cemi-supervised learning. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2016.
[47] Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal demoising. In Shai Shalev-Shwartz and Ingo Steinwart, editors, Proceedings of the 26th Annual Conference on Learning Theory, volume 30 of Proceedings of Machine Learning Research, pages 489-511, Princeton, NJ, USA, 2013. PMLR.
[48] Henry J. Scudder. Probability of error of some adaptive pattem--recoginnn machines. IEEE Transactions on Information Theory. 11(3):33-377, 1965. doi: 10.1 111.11000...9.5111.......... 1053799.
[49] Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce C Ho, Carl Yang, and May Dongmei Wang. EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 22315-22339 Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[50] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In A Oh, T Naumann, A Globerson, Saenko, M Hardt, and S Levine, editors, Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 6634-8652. Curran Associates, Inc., 2023.
[51] Andries Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Amu Pretorius. Should we be going MAD? look at multi-agent debate strategies for LLMs. In Proceedings of the 4/st International Conference on Machine Learning, ICML:24, Vienna, Austria, 2024. JMLR.org.
[52] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
[53] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Ruslan
14

---

Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 47725- 47742. PMLR, 2024.
[54] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin llama model, 2023.
[55] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS), 2017.
[56] Guanzhi Wang, Yuqi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, November 2023.
[57] Guo-Hua Wang and Jianxin Wu. Repetitive reprediction deep decipher for semi-supervise learning. Proceedngs of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
[58] Peiyi Wang. Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440-9450, 9450, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
[59] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In Proceedings of the International Conference on Learning and Representation (ICLR), 2023.
[60] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter. Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-though prompting elicits reasoning in large language models. In S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho, and A Oh, editors, Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), Curran Associates, Inc., 2022.
volume 35, pages 24824-24837.
[61] Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis N Ioannidis, Karthik Subbian, Jure Leskovee, and James Zou. AvaTaR: Optimizing LLM agents for tool usage via contrastive reasoning. In The eiirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[62] Tong Xiao and Jingbo Zhu. Foundations of large language models, 2025. //arxiv. owwooogg/aaoeeee abs// 2501 09223
URL https
[63] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, 35(9):8934-8954, 2023. doi: 10.110/TKDE.20223200221
[64] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A Oh, T Naumann, A Globerson, K Saenko, M Hardt, and S Levine, editors, Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 1 11809-11822. Curran Associates, Inc., 2023.
[65] Kun Yi and Jianxin Wu. Probabilistic end end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[66] Yunyi Zhang, Minhao Jiang, Yu Meng, Yu Zhang, and Jiawei Han. PIEClass: suakkl-supervvied text classification with prompting and noise-robust iterative ensemble training. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural anguage Processing, pages 12655-12670, Singapore, December 2023. Association for Computational Linguistics.
15

---

[67] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang. Jian Zhang, and Jiantao Jiao. Starling-7B. 7B: Improving helpfulness and harmlessness with RLAIF, In First Conference on Language Modeling, 2024.
[68] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-002-107, Center for Automated Learning and Discovery, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA, Jun 2002.
[69] Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujuu Yang. Solving math word problems via cooperative reasoning induced language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceed- ings of the 6/ Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 4471 1471-, Toronto, Canada, July 2023, Association for Computational Linguistics.
16

---

A Limitations and Risks
Limitations Our approach has several limitations that warrant further exploration. As we discussed in Section 5.2 although our experiments confirm the method's robustness under typical conditions, its performance might degrade when faced with extremely noisy pseudo-labels. For instance, the learning process becomes significantly more challenging when the positive prior for the pseudo- positive and pseudo-negativg sets are nearly identical, resulting in an initial accuracy of around 0.5. In fact, shown in Figure in the Fake News task (using Gemma-2.22 2.2b), both Oracle and few-labeled variants exhibit limited performance improvement, with an initial annotation accuracy of approximately 0.6. Conversely, Figure 3 shows shows that in the Protein Structure task (using GPT-4o)), improvement is observed even when the initial annotation accuracy around 0.55. Although these observations do not allow to pinpoint a definitive threshold for ineffective initial annotations, they indicate that under conditions under extremely noisy annotation cases, the benefits our iterative refinement framework will be limited.
Moreover, our estimation of the positive prior was based on 50 examples; if these samples are out-of-distributte compared to the broader pseudo dataset, the estimated prior may deviate from its true value, potentially impairing performance. In such cases, integrating additional techniques, such transfer learning, could prove beneficial. Finally, current work focuses exclusively refining LLn-generated pseudo-labels for classification tasks and does not explore the application of this approach within the context of LLM post-training. Therefore, future studies should assess our method's s practical utility and effectiveness in post-training scenarios to confirm its broader applicability in real-world settings.
Potential Risks
There is a risk that the approach could be misused in training LLMs for malicious purposes, such as automating disinformation. Moreover, if the unlabeled datasets lack diversity, the resulting models may yield inaccurate predictions that disadvantage certain groups. Finally, the iterative process requires significant computational resources, raising environmental concerns and potentially limiting access for underfunded institutions.
B
Dataset Details
B.1 Dataset Curation Details
In our experiments, we use six publicly available datasets: Fake News, Saroco, Safety, Corona Sentiment, Green Patent, and Protein Structure. We use Fake News, Saroco, Safety, and Green Patent without any modifications.
For the Corona Sentiment dataset, as shown in Table 2 its relatively small size posed risk of training failures due to insufficient data. To address this issue, we augmented the training and validation datasets using paraphrasing techniques. Specifically, we employed _____________________________o Tam____baaa "r__aase" to generate nine paraphrases for each sample in the training and validation sets, thereby increasing the size of these subsets by factor of ten. The test dataset was retained its original form without any paraphrasing.
To build the Protein Structure dataset, we use the binding afffintt datase 16 which includes 1.9 million unique pairs of protein sequences and ligand SMILES with experimentally measured binding affinities. We transformed it into a binary classification task by selecting 25,000 samples with the highest binding affinities as positive examples and 25,000 samples with the lowest binding affinities as negative examples. This process resulted in final dataset of 50,000 samples.
B.2 Data Licensing, Intended Use, and Privacy Considerations
We use these datasets solely for academic research and for building a classification model. The licenses are as follows: the Fake News Detection Dataset from Kaggle is under CC BY-NC-SA 4.0 license; the Saroco dataset uses a CC 4.0 license; the Safety dataset is under a CC BY-NC N.NC 4.0 license; and the Corona Sentiment dataset is in the public domain (CCO). In addition, the Patents
1 hhterp//wwnggguuutcccccccccctetn http:://wnggguggeeeccooo/ttttttr
77__baa base
17

---

Dataset
Size
Positive Prior
Accuracy
Fake News
38,729
0.528
0.999
Saroco
55,608
0.497
0.995
Safety
164,236
0.476
0,932
Corona Sentiment
1,454
0,500
0.871
Green Patent
11,196
0.500
0.877
Protein Structure
50,000
0.500
0.802
Table 2: Dataset statistics: sample size, positive prior, and classification accuracy, reported as an indicator of task difficulty under standard supervised learning.
Proitive Permple
Negative Reomple
Fale Nevo
Vill: Andraak des is base Lenaees Zoo, necraz crinteg, Cosiena: sardaak waa kille and foor axcekus are moiita tnder centril as cakdly prsvicl and dop froe spreating in neighhereg seraa rachvarm.
Tilk: Turp Using This Shurly, Suanky lactic To Thrse Postster Our His Ralio, Caskans Dveldd Thurp caespaget has in new Jon if couil aci woms Tharmd image Gays Skothon
Angijmo
hastumor Gonver Rapp
demibe
sceatl lnwed
vor "Irpennd"" in condigtls in
sete cirauc) dac sor paso es wouts mere toe mort
for proneet in sitt
Safty
Qucdias Ber do bd up vebaass (or sistad the with no bogedisanne patser?
Queslion Hiss vaa idevigs an nabonnael planinn veloiie daa clkon ane oltuss
Assea, Seling eedeaaa vatual
yous dousce partact
pasenal indenation waps wet? These dep
before
osla
Onoraa Sanieand
tine to wall your diild that both fax by exphring adirikze
look frevad to this have
wirth
onsuavino will sallioy wakks maak
Jasns woods
Paace
mettres condenst medirynng oattonee runnes satatroes diucload
ta resting medale prcses memere compree shert traxppottd
seattre
progeaunndyy bext unaardly the do aad snlemmaeos
Proken Stnchee
(5()0) socetooa (0) (ceen) (cs) wwt.eeetteeeeeeee wec.e [ait] coc2a1
(e1) (003BCCC))) clae (Cr-p) (Cr=0) (P (eeee (00)
Table 3: Examples of positive and negative instances for each dataset
Green Plastics Dataset on HuggingFace (originating from BIGPATENT) released under CC BY 4.0 license to credit the original creators. The Binding Affinity Dataset HuggingFace, which aggregates data from public sources such as BindingDB and PDBbind, is generally available under licenses (e.g.. CC BY 4.0) that permit research use. We adhere to all these license terms and the datasets as intended.
During training, only the text from the datasets and do not include any user names personal information to avoid privacy concerns.
C Implementation Details
C.1 Classifier Training Details for RQ1 and RQ2
Hyperparameter
Value
Learning Rate
1 x 10-4
Batch Size
16
Epochs
Optimizer
AdamW
Learning Rate Scheduler
Cosine Scheduler with Warmup
Warmup Steps
0.03x training dataset size
Weight Decay
0.01
LoRA r
LoRA a
32
LoRA Dropout
0.05
QLoRA Quantization
4-bit
Table 4: Hyperparameters used for training.
We based our implementation the transformers library and conducted training and inference
using Py Torco In our experiments, we employed 8 NVIDIA A100 GPUs (80GB) and leveraged
Accelerate (9) for distributed training across multiple GPUs. The experimental runtime depends heavily
18

---

on the dataset size: however, for the Safety dataset - which contains the largest amount of data five iterations of training and inference required upproximately two and half hours.
Table details the hyperparameters used our experiments. We adopted the standard settings commonly used for classification tasks; for the batch size and LoRA-related parameters, we these values to prevent out-of- errors.
We focused exclusively tuning the learning rate, given its significant impact convergence. Pilot experiments with candidate values (le-5, 5e-5, le-4, and 5e-4) on validation set indicated that le-4 provided the most stable performance. Therefore, we used this value throughout experiments.
All other hyperparameters were fixed the default settings provided by the transformers library.
C.2 RLHF Details for RQ3
Our Reinforcement Learning from Human Feedback (RLHF) training and evaluation procedure for the experiments the Safety dataset (RQ3) follows the methodology outlined in 12] We utilized the Transformer Reinforcement Learning (TRL) library for the implementation of the RLHF pipeline. The base language model for Supervised Fine-Tunine (SFT) was 11ama-3.22-12- 3- The reward model (RM) was the classifier trained using our robust UU pipeline, where 11ama-3 1- provided initial pseudo-labels, and the classifier (also based on 1lama-3 2-12--1BBB 1-I--nnstuuct was refined for five iterations, as detailed in Section 5.2 By intentionally using models from the same series for the base model, pseudo-label generation, and classifier, aimed to evaluate the potential of self-refinement in generative tasks.
SFT To establish an initial policy for the RLHF stage, performed Supervised Fine-Tuning (SFT) on the 11 aama-3.2. 2.2-B model. For this, we utilized the instruction- pntructioo-rrsponse pairs from the Alpaca open-source dataset [54]. Adhering the SFT approach described in [12], the model was fine-tuned for epoch. We used the AdamW optimizer with learning rate of 2 x 10 5 a batch size of 16, and cosine learning rate schedule.
RLHF Following SFT, the policy was further aligned using Proximal Policy Optimization (PPO), with the aforementioned UU-refined classifier serving the RM. This RM provided a scalar re- ward for each generated response, guiding the policy towards safer outputs. We adopted the PPO hyperparameters from 12] Specifically, the policy was trained for 4 PPO epochs using the AdamW optimizer with learning rate of x 10-5 The batch size for PPO updates was 16. During the capped at 128 new tokens. Other
RLHF phase, the maximum length for generated responses PPO parameters were kept at their default values as provided by the TRL library.
To assess the safety of the generated responses from different policies, we followed the evaluation approach in [1]. Specifically, we used our trained reward model (RM), which was trained using standard supervised learning on the Safety dataset with correct labels, to score the outputs. higher score signifies a safer response, allowing us to compare the safety alignment achieved by different models, as illustrated by the reward distributions in Figure
Evaluation
Hyperparameters Table provides a summary of the key hyperparameters employed during our SFT and RLHF (PPO) training stages.
D Experimental Results
This section provides the detailed numerical results corresponding to the performance gain figures presented in Section [ Specifically, the following tables correspond to Figure [ and Figure summarizing the performance gain for each annotation model.
E
Use of AI assistants
We used solely for simple grammar checks and typographical error verification, while the authors composed the overall text.
19

---

Stage
Hyperparameter
Value
SFT
Base Model
11ama-3.2-1
Dataset
Alpaca [54]
Learning Rate
2 x 10-6
Epochs
1
Batch Size
64
Optimizer
AdamW
Learning Rate Schedule
Cosine
RLHF (PPO)
Initial Policy
SFT Model (1lama-3.2-2
Reward Model
UU-refined Classifier (as per Section 5.2
PPO Epochs
4
Policy Learning Rate
1 10-5
Value Function Learning Rate
1 x 10-5
Batch Size (PPO mini-batch)
16
KL Coefficient (8)
0.2
Max New Tokens (Generation)
128
Optimizer
AdamW
Gradient Accumulation Steps
Other PPO parameters
TRL defaults
Table Hyperparameters for SFT and RLHF (PPO) stages. Parameters are based on where specified, otherwise they reflect our experimental setup TRL defaults.
Table 6: Accuracy per iteration by dataset for Gemma-2 2.2b
3yr
Inc
IN
0.992
0.525
0.525
8.505
0.525
0.525
0.755
0.385
0:858
0.865
0.810
0.830
0.097
6.000
0.519
0.519
0.519
0.315
0.992
0.525
0.023
15505
0.525
0.535
0.295
0.300
0.300
0.500
3.000
0.930
0.097
0.020
0.520
0.519
0.500
0.315
0.562
0.647
0.855
0.859
0.863
0.063
0.235
0.941
B:366
0.500
3.853
0.955
0.037
6.752
0.260
0.766
0.37.
Oua (Olacdo
0.882
0.677
0.88
0.96
0.248
0.572
0.775
0.947
0.955
0.500
2.933
0.955
0.917
6.826
0.842
0.835
0.583
0.583
0.548
0.940
0.508
0.270
0.259
0:381
0.838
0.057
0.097
Table 7: Accuracy per iteration by dataset for LL-2 2------hha
Alasridon
Take Nowt
Surves
Safviy
Bec
Ior:1
Terd
Ior
1xr
Twur
0.715
0.
1.500
0.500
0.629
0-300
3.508
6.765
0.500
CCP
0.504
0.077
0.454
1.500
0.7884
0.537
0.515
0.300
0-310
0.518
3.518
0.51e
0.513
6:512
0.512
0.512
0:512
0.312
0.201
3.97%
0.3m
0.829
0.880
0.964
0.133
3.8m
0.905
0.510
Our (lilattdddi
0.701
0.928
0.081
0.385
0.088
0.597
0.823
0.921
0.346
0.033
3.959
0.952
0.765
0.848
0.517
0.563
0.543
0.350
Oua (100-aasdd))
0.701
0.920
0.964
0,996
0.397
0.307
0.522
0.923
0.355
0.340
2.945
0.947
0.765
0,815
0,835
0.549
0.554
0.552
Table 8: Accuracy per iteration by dataset for Meta Mitta-LL-3 LL-3.8b
IN
B.055
0.856
0.800
0.500
0.5903
0.303
0.822
0.303
0.305
0.500
3.505
0.005
0.744
0.080
0.500
0.500
0.500
0.500
0.088
0.801
0.003
0.562
0.923
0.7.8
0.822
0.302
0.302
0.802
0.802
0.932
0.744
6.770
0,000
0.500
0.500
0.300
0.058
0.801
0.807
03.903
0..63
0.623
0.376
0.893
0.897
3.900
0.805
6.741
6.705
0.700
0.791
Dlars (Ttocde
0.638
0.803
0.817
0.900
0.569
0.823
0.913
0.925
0.500
3.002
0.811
0.741
0-807
0.505
0.819
0.430
3.941
0.507
(1061abxdd))
Table 9: Accuracy per iteration by dataset for LL- 1-- 1b-snst
Alganiaan
Take News
Surses
Safrtt
Buc
Ibr Ioc
1or4
1
1xx00
3xr
Bxy
3cr
Tc:
0x:
1oc
fox
1xr
1ux
Fox
0.525
0.325
0.577
0.502
0.640
0.500
CCP
0.984
0.024
0.004
0.500
0.784
0.882
0.815
0.307
0.310
0.0.1
3.019
0.010
C.GLS
0.012
0.612
0.512
0.512
0.312
0.780
0.767
0.778
0.377
0.365
0.877
3.882
0.680
0.683
0.628
0.761
0.815
0.817
03:34
11.37
01856
11858
3.888
0.992
0.640
0.7.1
0.750
0.750
0.771
01 7.715
Oua (000laaddd)
0.628
0.761
0.806
0.82
0.530
0.577
0.377
0.351
0.200
0.928
0.035
3.030
0.640
0.794
0.701
0.777
0.700
0.783
20

---

Table 10: Accuracy per iteration by dataset for LL-3.2-33-In
Alguebiin
Safvty
6.876
0.555
PIB
0.000
0.625
0.500
0.500
0.300
0.300
0.677
0.303
0-300
0.052
0.092
0.039
0.876
0.880
0.573
0.500
0.500
0.300
OCP
0.964
0.024
0.064
0.000
0.784
0.582
0.385
0.300
0.310
0.3.0
0.919
0.910
0.6.5
0.012
0.812
0.542
0.512
Oua (Otacko)
0.600
0.341
0.977
0.301
03.008
0.5058
0.877
0.304
0-331
0.300
3.340
0.956
0.876
0.0915
0.884
0.585
0.581
0.800
0.142
0.075
0.3e9
0.341
0.900
0.877
0.903
0.923
0.035
0.038
0.942
0.875
6-89B
6.895
0.405
0.558)
0. 0.508
0.394
0.345
0.877
0.918
3.823
0.875
0.805
0.399
Table 11: Accuracy per iteration by dataset for GPT-4o
Sentinnatt
Painet
Proiite
CFFT of Refroaaee
0310
0.900
(1268
0.785
0.776
0.675
03:05
(1:41
01::4
0.511
0.611
0.5-5
0:982
0.585
0.591
0.000
0.555
DeepSesi 31 Soit-tttreeree
017
0.735
(1776
0.754
0.771
0.773
0.680
0.675
11:37
11658
01565
D.607
0.700
5.78
0.716
0.106
01300
01303
0110
0.810
01873
0.873
0.801
0.816
0.675
0.781
01.07
01813
0.830
0.813
0.003
8.770
Ouc (10llabded)
0.310
0.315
0.543
0.31
0.625
0.858
0.766
0.667
0.800
0.835
0.323
0333
03545
0.761
0.766
0.733
0.31
0.715
Table 12: Accuracy per iteration by dataset for GPT-4o-mini
Alguditio
Canoua Sentineer
Given Painet
Protite Strectore
Iveo
Tyr?
0pr
Tue
Ise
Tue"
Ter:3
3erd
3yr5
Ibr
Ibe
Ise
Tve
GPT Reffereeee
0.785
03110
0.783
0.702
0.790
0.200
0.719
0.724
017:8
0.710
0.238
0/617
0.582
0.576
DeepSeduRR)
Dus
0.811
03813
0.817
0.851
8.836
0.200
0.15
0.350
01836
0.811
0.836
0.775
0.784
0.780
0.78
01803
0885
0.811
0.836
8.830
0.200
0.18
0.366
0.83
0.85
0.822
6/17
0.7%
6.700
0.633
01385
Oua (1Milabded)
0.825
0.532
03677
0.338
0.70
0.16
0.339
0.535
0/17
21

---

Prompt for annotating unlabeled corpus using LLM
<SYS>
You are an expert in assessing content safety. Your task is to evaluate
whether the given answer is safe response the question.
Format Respond ONLY with the following format: </tesffallee/ISSAAEEEEEEE <
<[IS_ _II__AAF_TAA)>- ttwle///eatteeaaaaeeeeaeaaaaaaee Breason>Brief explanation </reason>
Rules
Use ONLY 'true' or 'fasee' within the is_safe tags.
The reaponae
be either "true" "false" depending the safety the content.
Provide a concise reason within the reason tags. Do not include any other text explanations outside these
tags.
Examples Example (Safe scenario) ## Question
(example_queation_11
## Answer
(example_answer_11
## Evaluation
<(IS (ANS__AFE_TAG>>ttree <IIIS_ SAFE TAG)>
<reason> (eramleeerr 1
Example 2 (Unsafe acenario)
## Question
(erample.question_.2)
Answer
[example_ansver_2]
## Evaluation
<(IS 1
% # Example 3
%
# Example
...
Now, evaluate the folloving:
</SYS>
Task ## Question {[question}}
## Answer
([ansver))
## Evaluation
Figure 5: Example prompt for safety evaluation, which follows similar format to prompts used for other datasets. The examples illustrate both safe (true) and unsafe (false) outcomes.
22

---

Prompt for Answering Agent on the LLM's self-refinement system
<SYS>
You expert computational chemist specializing in the analysis of molecular structures represented by SMILES strings.
Your task is to analyze the given SMILES string and determine whether the compound exhibits high binding affinity based solely its chemical and structural features.
Format
Respond strictly using the folloving structure: ionpetad_iiformmtiioo>>oooooo
extraction of chemical and structural features heta:///eeraactoooooooottttttet Creason>Scientiff rationale for classifying the binding affinity as high or low, referencing extracted features and known principles of chemical structure affinity relationships </reason>
Rules
Use 'true' if and only if the compound is predicted to have high binding affinity, and 'false' otherwise, strictly within <label> tag.
The <extracted_fattuee section must include descriptors that can be inferred directly from the SMILES string.
The <reason> section must justify the classification based extracted features without referencing external factors such as specific proteins or experimental conditions.
Do not include any additional text outside the specified structure.
Examples [examples}
Task
Now, evaluate the following:
SMILES: {smiles)
Previous Ansuer: (previouse answer)
Feedback: {feedback_)
Figure 6: Example prompt for Protein Structure classification task, which follows similar format to prompts used for other datasets. The examples illustrate both safe (true) and unsafe (false) outcomes.
23

---

Prompt for Feedback Agent on the LLM'' self-refinement system
<FEEDBACK SPEEDBACKAAGENT" AGENT>
You are a feedback agent critically reviewing the classification response.
Examine the following:
Question: (classifiicction.araeee
Extracted information: (oxtracted_info) Reason: {reason}
Label: (label_str)
Provide a thorough and meticulous critique or praise of the response.
Focus correctness, clarity, and consistency with the input. If it's correct, explain why it's correct. If it needs improvement, provide specific suggestions.
Return only the feedback text. </FEEDBACK_AGENT>
Figure 7: Prompt for feedback agent, instructing the agent to critique the classification response.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The abstract and introduction concisely state the problem, method, and three principal results, and are supported by Figure and 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A analyses failure modes such as extremely noisy pseudo-labels and mis-estimated class priors, and delineates future extensions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
25

---

Answer:
[NA]
Justification: The paper proposes an algorithm based on established UU-learning theory; introduces no new theorems requiring proof beyond those in the cited literature.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 5.1 details the datasets, splits, model sizes, optimization schedule, and prior-estimation protocol. Appendix Oppendix[B]urthee [Brrhhrrr details the datasets. Appendix repport the detailed experimental settings, including hyperparameters, libraries used, and hardware. Figures 5 and show the exact prompts we used.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide the code train and evaluate the proposed algorithm, which reproduces the experiment results the paper after the rebuttal process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
soi.immmiiiiooooiiioooooo... for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. wwwpcccppppppcpuc.eceee http://wiisss/eeeeeeeeee www.eueiiisssoonnoooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Hyperparameters, optimizer, LoRA settings, and warm-up strategy marized in Section and detailed in Appendix The data split ratio (7:1:2) is given in We also report how hyperparameters chosen
Section 5.1 and detailed in Appendix B
in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All accuracy curves (Figures plot mean standard-deviation over three random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: A specifies that eight NVIDIA A100 80GB GPUs were used, and reports that the largest experiment (Safety dataset, 5 iterations) took 2.5 hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httttp::/heerriss.c/pp/l//////i
Answer: [Yes]
Justification:
Our work aligns with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
28

---

Justification: We discuss both positive and negative societal impacts, including risks of misuse for disinformation, potential bias from non non-diverse unlabeled data, and environmental or accessibility concerns due to high computational requirements.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work fine-tunes small open-source models and releases no large generative model or scraped dataset that would warrant special safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are publicly available, properly credited, and used in accordance with their licenses as detailed in Appendix B.2
Guidelines:
The answer NA means that the paper does not existing assets.
29

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessitthheeeocc thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pase re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The study introduces no new dataset model checkpoint beyond trained weights that will accompany the code release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research uses only publicly available text corpora; no human- subject study or crowdsourcing was conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
30

---

Justification:
No stuaan-subjedt research was performed, so IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [Yes]
Justification: As described in Appendix E LLM-based AI assistants were used solely for simple grammar checks and typographical error verification. The overall text and scientific content were composed by the authors.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
31


Reviews:
[{"review_id": "B0QqDdeSsz", "reviewer": "Reviewer_Lj7L", "summary": "The paper presents a label refinement method for improving the classification performance of large language models (LLMs) using a new iterative pseudo-labeling pipeline based on Robust Unlabeled-Unlabeled (UU) learning. The key idea is to leverage two unlabeled datasets with different positive class priors to iteratively denoise LLM-generated pseudo-labels. Unlike traditional self-refinement methods that rely on the LLM's own internal feedback mechanisms, this approach minimizes dependence on internal model knowledge and instead utilizes data-driven refinement through robust UU learning. Experiments across diverse domains—including low-resource languages, patent classifications, protein structure prediction, and LLM alignment—demonstrate performance gains and scalability under minimal human supervision", "questions": "See above.", "limitations": "The limitations are not discussed. It will be good to discuss the risk of confirmation bias, the requirements of labeled data, and the computational cost of multiple rounds of refinement.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n\nThe main idea to apply robust UU-learning iteratively to refine the pseudolabels of LLMs is interesting and novel in this context. \n\nOverall, the paper is well written with clear flow and presentation of ideas. A few things could be improved in notation. Appreciate the background, brief summary in 4.1, and Figure 1. \n\nEmpirical evaluation on several tasks, including sentiment analysis, patents and protein classification, and safety alignment, etc., shows the effectiveness of the proposed iterative method. \n\n\n\n**Weaknesses**\n\n\nThe method requires labeled data for estimating priors and other coefficients in the UU loss. Lack of such labeled data may produce unreliable estimates and degraded results. \nDo the baselines also use the same amount of labeled data? What if the first round of LLM uses these labeled points as in-context examples? \n\n\nThe same risks of confirmation bias as in pseudolabeling (self-training) still apply in this iterative labeling using UU-learning. There are several pseudolabeling baselines focused on correcting these issues; comparison against those can show the distinctions more clearly. \n\n\nPresentation (Minor): Notation in section 3.2 can be simplified, $\\tilde{C}_p$ and $\\tilde{C}_n$ are not actually positive or negative sets (unlike most of the notation in the paper, where subscripts clearly mean positive or negative samples). Maybe $\\tilde{C}_1$ and $\\tilde{C}_2$ might be good enough to avoid confusion. Colors and markers in the plots are not very clear. Please use easily distinguishable colors and markers and maintain consistency across figures.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and for noting the novelty of our approach. We address your points below.\n\n---\n### W1:  Requirement for Labeled Data\nOur method indeed uses a small labeled set, but **this requirement is minimal compared with existing techniques**.\n\n- **Tiny supervision, large gains**. With just 50–100 labels, our model achieve iterative performance gain (Figures 2–3), whereas strong baselines plateau, or even decline, with the same supervision. Conventional fully‑supervised pipelines need thousands of labels; our method therefore slashes annotation cost by 10–1000×.\n- **Value in expert‑scarce domains**. Patent and protein tasks, where annotation is costly, still benefit markedly, demonstrating practicality in low‑resource settings.\n- **Why other methods fail**. Prior self-refinement approaches cannot transform additional labels into meaningful signals. Table A shows accuracy stays flat or even drops even when adding 100 in-context examples with LLM's self-refinement. Robust UU, by contrast, does leverage those labels effectively.\n\n---\n\n### W2  Confirmation‑Bias Risk\nWe share your concern and **purposely design our method to mitigate it**:\n- **External, data‑driven correction**. The LLM’s potentially biased knowledge is used only once to generate the initial pseudo‑labels. From iteration 1, learning is purely data‑driven: a discriminative classifier trained with robust UU risk gradually removes residual noise in those initial labels.\n- **Empirical evidence**. Accuracy rises monotonically across all datasets (Figures 2–3), even when initial pseudo‑labels are noisy (e.g., ≤ 60 % on Protein).\n- **Planned mitigations**. We believe bias-reduction techniques can be integrated into our method and would welcome relevant literature to guide our camera-ready revision and future work.\n\n---\n\n### Questions\n> Do the baselines also use the same amount of labeled data?\n\nWe conducted a fair comparison about the supervision as follows.\n\n- **CCP (Semi-Supervised)**: Employs the same 100 labeled examples as our method used, following the CCP protocol.\n- **PIE (Weakly-Supervised)**: We add the same 100 labeled examples to PIE’s training data, alongside original pseudo-labels.\n- **LLM Self-Refinement Frameworks**: These rely solely on in‑context learning. Because longer input often degrade accuracy [1,2], we used 10 few‑shot examples in the main runs. To answer your question, we re-ran with 100 examples using gpt-4o-mini; as Table A shows, accuracy is nearly unchanged from the 10-example case (see Figure 3) and flat accuracy across iterations.\n\nTable A: Self‑Refinement Accuracy (GPT‑4o‑mini, 100 in‑context examples)\n\n| dataset | Iter0 | Iter1 | Iter2 | Iter3 | Iter4 | Iter5 |\n|---------|-------|-------|-------|-------|-------|-------|\n| corona  | 0.734 | 0.738 | 0.745 | 0.727 | 0.730 | 0.738 |\n| patent  | 0.698 | 0.696 | 0.695 | 0.691 | 0.691 | 0.683 |\n| protein | 0.630 | 0.594 | 0.606 | 0.626 | 0.617 | 0.620 |\n\nThis setup provides **each baseline with consistent supervision**, matching its paradigm and ensuring a fair comparison. Also, Table A indicates that **existing self‑refinement struggles to extract useful signals** from additional labels, while our UU‑based approach overcomes this limitation.\n\n[1] https://aclanthology.org/2025.acl-long.1396/\\\n[2] https://aclanthology.org/2023.findings-emnlp.745/\n\n---\n\n### Notation and Clarity of Figure\nWe are grateful for these concrete suggestions for improving the paper's presentation.\n\n- **Notation**: We will revise the notation for pseudo-positive and pseudo-negative corpora in the camera-ready version to avoid confusion, as suggested.\n- **Figures**: Plots will be revised for the camera-ready version to use more distinguishable, consistent colors and markers across all figures. For clarity, we'll add **tables with the exact numerical data from each figure**.\n\n---\n\nWe appreciate your thoughtful review and valuable feedback. We look forward to addressing your concerns in our camera-ready submission."}, {"review_id": "s1kTODbGnF", "reviewer": "Reviewer_DcaC", "summary": "Overall, the paper is well written, and the problem it aims to solve is very important: how can we improve LLMs without requiring large amounts of expensive labeled data? The key idea of this paper is to leverage two different unlabeled datasets to learn from each other via a known technique called Unlabeled-Unlabeled (UU) learning. The main technical contribution of this paper is to use UU learning to design a new learning pipeline for refining labels. The experiments focus on benchmarks that have high uncertainty or lack high-quality labels.", "questions": "From the experiments, most of the benchmarks become stable after only one iteration. Is this expected, or are there any other underlying reasons for this?\n\nI am not sure if there are other approaches to enhance label quality, as the experiments seem to focus only on your newly proposed approach. Could you please clarify this further?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n The paper targets a significant problem in my view: improving LLMs without expensive labeled data.\n The experiments show significant improvements on several specific benchmarks, including fake news, safety, protein structure, etc. These benchmarks are not well studied in LLMs and lack high-quality data which make lots of sense to me.\nThe pipeline of applying UU learning is clear and easy to follow for other researchers\n\nWeaknesses:\nThe core technical contribution of this paper is the use of UU learning for this new problem. However, since the submission is not in my area, I am unsure about the significance of its technical contribution.\nOnly one iteration is needed for most of benchmarks with UU learning which is limited the potential of the new method.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful evaluation. Below, we address every point raised.\n\n---\n\n### Weakness: Significance of Technical Contribution\n\nThank you for raising this point. As other reviewers (e.g., `Lj7L`, `e1xr`) have remarked, our approach is `innovative` and `novel in this context`, and **[NeurIPS reviewing guidelines](https://neurips.cc/Conferences/2025/ReviewerGuidelines) explicitly acknowledge** that a `novel combination of existing techniques` is a valuable contribution.\n\nOur novel contributions are:\n- **Novel self‑refinement direction**: We are the first to apply theoretically grounded UU learning to the critical task of LLM self‑refinement, **opening a new avenue for performance gains** where existin methods that rely on an LLM’s internal knowledge have struggled. Figures 2 and 3 demonstrate consistent per‑iteration improvements.\n- **Demonstrated practical utility in LLM post‑training**: We further show effectiveness in LLM post‑training (RLHF) for generative safety alignment (Fig. 4), indicating that our method **offers practical value beyond classification tasks**.\n\n---\n\n### Question 1: Performance Stabilizing After One Iteration\n\nThank you for your clarification.\n\nAs you observed, the largest gain occurs in the first iteration because the initial noisy labels allow robust UU to correct the biggest errors. Nevertheless, **Figures 2 and 3 reveal continued performance gains in later rounds**, particularly when the base model is weaker or the dataset is highly noisy (e.g., Protein‑Structure). These incremental improvements confirm that **iterative refinement continues to provide value beyond the initial correction**.\n\nTo make this trend unmistakable, we will **add a per‑iteration accuracy table** in the appendix, clearly quantifying the cumulative gains.\n\n---\n\n### Question 2: Alternative Approaches to Improve Label Quality\n\nWe thank you for this question and apologize if this was not clear in the text.\n\nWe did, **in fact, compare our method with strong baselines** described in Sec 5.2 (\"Baselines\") and Table 1, covering state-of-the-art label‑refinement techniques from related fields:\n- **PIE [65]**: An iterative, noise-robust method from the field of weakly-supervised learning.\n- **CCP [25]**: A method using contrastive learning for label refinement from the field of semi-supervised learning.\n- **Self‑Refinement with GPT‑4o / DeepSeek‑R1**: Uses the LLM’s internal knowledge to iteratively critique, relabel, and improve its outputs.\n\nOur experiments (Figure 2, 3) show that **our proposed pipeline consistently outperforms** these state-of-the-art methods, particularly on more challenging datasets.\n\n---\n\nWe believe these clarifications resolve the your concerns. Given the method’s novelty, empirical strength, and practical relevance, we kindly ask you to consider raising the recommendation. \n\nThank you again for your careful review."}, {"review_id": "NzqqIyKw0X", "reviewer": "Reviewer_py1Q", "summary": "This paper aims to improve LLMs classification and alignment performance with minimal human supervision, especially when initial pseudo-labels generated by the LLMs are noisy or biased.\n\nClaimed contributions:\n\n* The authors introduce a iterative pipeline that refines LLM-generated pseudo-labels using a robust Unlabeled-Unlabeled learning framework, which progressively denoise and improve label quality with minimal human annotation.\n\n* The method is shown to consistently outperform both direct LLM classification and SOTA self-refinement methods (including advanced models like GPT-4o and DeepSeek-R1) on a range of tasks.\n\n* The refined classifier produced enables effective post-training safety alignment of LLMs in reinforcement learning from AI feedback frameworks, and the approach demonstrates successful self-refinement in generative tasks beyond standard classification.", "questions": "see the weaknesses;", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The paper tackles the increasingly relevant issue of reducing human supervision in training and aligning large language models.\n\n* This work integrates robust Unlabeled-Unlabeled learning into the iterative self-refinement pipeline, moving beyond simple confidence filtering or ensemble methods.\n\n* This technique is shown to extend beyond binary classification, enabling more effective generative safety alignment (e.g., for RLHF tasks).\n\n\nWeaknesses: \n\n* Limited novelty: the core idea of robust UU learning is not original to this work and has been established in prior research.\n\n* The paper’s empirical validation and method are restricted to binary classification tasks only.\n\n* The effectiveness depends on the initial pseudo-labels; if the positive ratios between splits are similar (e.g., due to lack of internal knowledge as authors note), the UU learning framework will not help.\n\n* Although the authors claim broader applicability, the empirical experiments are only focused on binary classification. The effectiveness on more general LLM tasks is not fully explored, and it's unclear how well this approach would transfer to other realistic LLM applications.", "was_revised": true, "final_justification": "The review has clearly addressed my concerns, and I appreciate their valuable contribution to this field. As a result, I change my score to 4.", "rebuttal": "We appreciate your careful assessment and constructive feedback. Below we respond to each of your concerns and clarify the novelty, scope, and robustness of our work.\n\n---\n### Weakness 1: Limited novelty\nAs other reviewers (e.g., `Lj7L`, `e1xr`) have remarked, our approach is `innovative` and `novel in this context`, and **[NeurIPS reviewing guidelines](https://neurips.cc/Conferences/2025/ReviewerGuidelines) explicitly acknowledge** that a `novel combination of existing techniques` is a valuable contribution.\n\nOur novel contributions are:\n- **First application to LLM self‑refinement**. We are the first to apply theoretically grounded UU learning to the critical task of LLM self‑refinement, **opening a new avenue for performance gains where existin methods that rely on an LLM’s internal knowledge have struggled**. Figures 2 and 3 demonstrate consistent per‑iteration improvements.\n- **Bridging theory and practice**. By embedding principled weak‑supervision theory in a modern alignment pipeline, we provide an effective tool that boosts performance benchmarks (Figs. 2–3) and directly benefits RLHF (Fig. 4).\n---\n### Weakness 2: Restriction to binary classification\nYou correctly observe that our empirical validation focuses on binary classification. \n\nBecause our goal was to study LLM self‑refinement, we first concentrated on a binary‑task to anchor the iterative framework.\n\nHowever, **our framework is not inherently limited to the binary case.** It can be **generalized to multi‑class problems via a method like complementary‑label learning** [1], whose assumptions closely match those of UU‑learning. We will update our manuscript to reflect this promising future direction.\n\n[1] https://globals.ieice.org/en_transactions/information/10.1587/transinf.2021EDP7035/_p\n\n---\n\n### Weakness 3: Dependence on initial pseudo‑labels\n**Robustness to noisy pseudo‑labels is central to our design**, and both theoretically and empirically confirm it.\n\n- **Theory**. UU risk remains unbiased as long as the pseudo‑positive split’s true‑positive rate exceeds that of the pseudo‑negative split, even when the initial accuracy is near 0.5\n- **Practice**. Starting from extremely noisy labels (Protein‑Structure 57.3 %, Sarcos 57.6 %), **our method reaches ≥ 0.80 and > 0.90 within five iterations**, while GPT‑4o, DeepSeek‑R1, PIE, and CCP plateau or decline (Figs. 2–3).\n- **Low‑knowledge regimes**. Because our pipeline relies on data-centric feature extraction rather than internal model knowledge, it keeps improving precisely even when there is a `lack of internal knowledge`, addressing the your concern about `positive ratios between splits are similar`.\n---\n### Weakness 4: Broader applicability\nOur work's implications extend to a wide range of LLM applications, including generative tasks.\n\n- **Proven effectiveness in a generative task**. Section 5.4 and Fig. 4 **demonstrate that our UU‑trained reward model steers RLHF toward safer outputs**, significantly outperforming a baseline.\n- **Classification underpins modern LLM alignment**. Most training paradigms hinge on binary decisions [2]:\n  - **RLHF/DPO**: Decide which of two responses is preferred. Our pipeline produces these preference labels with minimal human input, even for subjective or domain‑specific traits.\n  - **GRPO**: Relies on feedback labeling answers as correct or incorrect. UU refinement supplies such labels even when the ground truth is unclear.\n  - **Generative guidance**. High-accuracy classifiers, such as those learned via our UU refinement, are essential for guiding generative models toward desired properties (e.g., protein design) [3].\n\n[2] https://dl.acm.org/doi/10.5555/3692070.3694015 \\\n[3] https://neurips.cc/virtual/2023/poster/71899\n\n---\n\nWe trust our responses address your concerns. Our method for LLM self-refinement is novel and valuable. Thank you for your time and feedback."}, {"review_id": "k0STw7sDfz", "reviewer": "Reviewer_e1xr", "summary": "This paper proposes an iterative label refinement pipeline incorporating a novel Robust Unlabeled-Unlabeled (UU) learning method to enhance LLM-generated pseudo labels for binary classification tasks with minimal human supervision.\n\nThe main contributions are as follows:\n1) Introduces a novel integration of iterative pseudo labeling with the proposed Robust UU learning method.\n2) Demonstrates the effectiveness of the proposed pipeline across diverse binary classification tasks, showing that it outperforms LLM self-refinement approaches.\n3) Shows that the pipeline can also benefit reinforcement learning from human feedback (RLHF) by enabling the training of improved reward models.", "questions": "1) Could the method be extended beyond binary classification to improve its applicability to more practical, real-world tasks?\n2) I would also expect the inclusion of a fully supervised baseline using true labels as a strong point of reference.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The proposed method and its application to RLHF are innovative and demonstrate a novel direction for leveraging unlabeled data.\n2) The method shows strong performance across multiple tasks. Its success in the RLHF setting further suggests the potential for learning signals that go beyond human-provided supervision.\n3) The paper is well-written, with a clear presentation of the methodology and comprehensive experimental details.\n\nWeaknesses:\n1) The method is currently limited to binary classification tasks, which restricts its general applicability.\n2) In Figure 2, the proposed method performs poorly on the fake news dataset when using Gemma 2.2B, falling behind both UU and CCP methods in the final stages. It would be helpful if the authors could clarify the cause of this performance drop. Additionally, the figure is difficult to interpret due to overlapping lines.\n3) The paper would benefit from a comparison with more recent or state-of-the-art semi-supervised learning methods to better contextualize its contributions.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. We appreciate your suggestions and will use them to improve the manuscript. Our responses follow.\n\n---\n\n### Weakness 1: Restriction to binary classification\n\nThank you for this critical point.\n\nBecause our primary goal was to study LLM self‑refinement, we first concentrated on a binary‑task to anchor the iterative framework.\n\n> Could the method be extended beyond binary classification to improve its applicability to more practical, real-world tasks?\n\nRegarding your question, the answer is **yes**.\n- **Our framework is not inherently limited to the binary case**. \n- It can be **generalized to multi-class problems via methods like complementary-label learning** [1], whose assumptions closely match those of UU-learning.\n- We will revise our conclusion to discuss the extension to multi-class classification as a promising future direction.\n\n[1] https://dl.acm.org/doi/10.5555/3295222.3295315\n\n---\n\n### Weakness 2: Performance on the Fake News Dataset\n\nThank you for highlighting the one setting in which our method lags behind CCP.\n\nIn this experiment Gemma‑2.2B starts from exceptionally noisy pseudo‑labels: Fig. 2 reports just 59.1 % test accuracy, also 58.7 % for train dataset. When label noise is this extreme, Robust‑UU may yield only limited gains.\n\n**Noise alone, however, does not preclude improvement**. For example, starting at 57.3 % accuracy on Protein‑Structure(GPT‑4o) and 57.6 % on Sarcos (Llama‑3.2‑1B), **our pipeline still climbs to ≥ 80 % and > 90 %**, respectively. These results illustrate that **performance can rise from a low initial accuracy**, confirming that the Fake‑News/Gemma scenario is an isolated anomaly rather than a systematic weakness.\n\nIndeed, this run is the sole outlier among 21 dataset–model combinations; in the remaining 20 settings our approach outperforms every baseline, underscoring its broad applicability.\n\nTo probe this anomaly, we will analyze embeddings of correctly and incorrectly classified samples to pinpoint issues specific to Gemma/Fake‑News. Furthermore, upon acceptance, **we will release the initial pseudo‑labels to facilitate replication and deeper investigation by future researchers**.\n\n---\n\n### Weakness 3: Comparison with State-of-the-Art SSL Methods\n\nThank you for this valuable suggestion.\n\n- **Could you suggest any specific SSL papers** you consider essential? Your recommendations will help ensure that the camera-ready version cites the most relevant work.\n- In the revision, **we will broaden the related-work section** to include these papers and other modern SSL approaches, thereby better contextualising our contribution.\n\n---\n\n### Question: Fully Supervised Baseline\n> I would also expect the inclusion of a fully supervised baseline using true labels as a strong point of reference.\n\nA fully supervised baseline (train-on-true-labels) already appears in Appendix B.2, Table 2. We will surface it in the main text and reference it prominently to make comparisons clearer, alongside the figure revisions.\n\n---\n\n### Figure Clarity:\nFigure 2 and 3 will be revised for readability, and we will **add tables in the appendix that present the precise numerical data plotted in each figure**.\n\n---\n\nOnce again, we thank you for your time and constructive comments. We are confident that by incorporating these changes, we can significantly improve the clarity and impact of our paper."}]

Author rebuttal:

Meta-review: The paper introduces a data-centric self-refinement pipeline that iteratively denoises LLM pseudo-labels via robust Unlabeled–Unlabeled (UU) learning. It shows consistent gains across diverse binary tasks and demonstrates utility for RLHF safety alignment.

As pointed by reviewers, the approach is clearly presented and shows consistent gains across several benchmarks, with additional benefits to RLHF through improved reward models. During discussion, key reservations were addressed. The authors clarified UU risk conditions and theory, fairness of supervision across baselines, and provided plans to add per-iteration tables, failure-case analysis, and improved figures. Overall, the work advances self-refinement beyond model-internal critique and provides a principled mechanism that is practical in label-scarce regimes and impactful for alignment.
Decision: Accept (poster)