Title: Self Iterative Label Refinement via Robust Unlabeled Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Weakly Supervised Learning

LLM Self-Refinement

Low Resource Domain
Abstract: Recent advances in large language models (LLMs) have yielded impressive performance on various tasks, yet they often depend on high-quality feedback that can be costly. Self-refinement methods attempt to leverage LLMs' internal evaluation mechanisms with minimal human supervision; however, these approaches frequently suffer from inherent biases and overconfidence, especially in domains where the models lack sufficient internal knowledge, resulting in performance degradation. As an initial step toward enhancing self-refinement for broader applications, we introduce an iterative refinement pipeline that employs the Unlabeled-Unlabeled learning framework to improve LLM-generated pseudo-labels for classification tasks. By exploiting two unlabeled datasets with differing positive class ratios, our approach iteratively denoises and refines the initial pseudo-labels, thereby mitigating the adverse effects of internal biases with minimal human supervision. Evaluations on diverse datasets, including low-resource language corpora, patent classifications, and protein structure categorizations, demonstrate that our method consistently outperforms both initial LLM's classification performance and the self-refinement approaches by cutting-edge models (e.g., GPT-4o and DeepSeek-R1). Moreover, we experimentally confirm that our refined classifier facilitates effective post-training alignment for safety in LLMs and demonstrate successful self-refinement in generative tasks as well. Our code is available at https://github.com/HikaruAsano/self-iterative-label-refinement.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an iterative label refinement pipeline incorporating a novel Robust Unlabeled-Unlabeled (UU) learning method to enhance LLM-generated pseudo labels for binary classification tasks with minimal human supervision.\n\nThe main contributions are as follows:\n1) Introduces a novel integration of iterative pseudo labeling with the proposed Robust UU learning method.\n2) Demonstrates the effectiveness of the proposed pipeline across diverse binary classification tasks, showing that it outperforms LLM self-refinement approaches.\n3) Shows that the pipeline can also benefit reinforcement learning from human feedback (RLHF) by enabling the training of improved reward models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The proposed method and its application to RLHF are innovative and demonstrate a novel direction for leveraging unlabeled data.\n2) The method shows strong performance across multiple tasks. Its success in the RLHF setting further suggests the potential for learning signals that go beyond human-provided supervision.\n3) The paper is well-written, with a clear presentation of the methodology and comprehensive experimental details.\n\nWeaknesses:\n1) The method is currently limited to binary classification tasks, which restricts its general applicability.\n2) In Figure 2, the proposed method performs poorly on the fake news dataset when using Gemma 2.2B, falling behind both UU and CCP methods in the final stages. It would be helpful if the authors could clarify the cause of this performance drop. Additionally, the figure is difficult to interpret due to overlapping lines.\n3) The paper would benefit from a comparison with more recent or state-of-the-art semi-supervised learning methods to better contextualize its contributions.\n\nQUESTIONS:\n1) Could the method be extended beyond binary classification to improve its applicability to more practical, real-world tasks?\n2) I would also expect the inclusion of a fully supervised baseline using true labels as a strong point of reference.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve LLMs classification and alignment performance with minimal human supervision, especially when initial pseudo-labels generated by the LLMs are noisy or biased.\n\nClaimed contributions:\n\n* The authors introduce a iterative pipeline that refines LLM-generated pseudo-labels using a robust Unlabeled-Unlabeled learning framework, which progressively denoise and improve label quality with minimal human annotation.\n\n* The method is shown to consistently outperform both direct LLM classification and SOTA self-refinement methods (including advanced models like GPT-4o and DeepSeek-R1) on a range of tasks.\n\n* The refined classifier produced enables effective post-training safety alignment of LLMs in reinforcement learning from AI feedback frameworks, and the approach demonstrates successful self-refinement in generative tasks beyond standard classification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The paper tackles the increasingly relevant issue of reducing human supervision in training and aligning large language models.\n\n* This work integrates robust Unlabeled-Unlabeled learning into the iterative self-refinement pipeline, moving beyond simple confidence filtering or ensemble methods.\n\n* This technique is shown to extend beyond binary classification, enabling more effective generative safety alignment (e.g., for RLHF tasks).\n\n\nWeaknesses: \n\n* Limited novelty: the core idea of robust UU learning is not original to this work and has been established in prior research.\n\n* The paper’s empirical validation and method are restricted to binary classification tasks only.\n\n* The effectiveness depends on the initial pseudo-labels; if the positive ratios between splits are similar (e.g., due to lack of internal knowledge as authors note), the UU learning framework will not help.\n\n* Although the authors claim broader applicability, the empirical experiments are only focused on binary classification. The effectiveness on more general LLM tasks is not fully explored, and it's unclear how well this approach would transfer to other realistic LLM applications.\n\nQUESTIONS:\nsee the weaknesses;\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
Overall, the paper is well written, and the problem it aims to solve is very important: how can we improve LLMs without requiring large amounts of expensive labeled data? The key idea of this paper is to leverage two different unlabeled datasets to learn from each other via a known technique called Unlabeled-Unlabeled (UU) learning. The main technical contribution of this paper is to use UU learning to design a new learning pipeline for refining labels. The experiments focus on benchmarks that have high uncertainty or lack high-quality labels.

STRENGTHS AND WEAKNESSES:
Strengths:

 The paper targets a significant problem in my view: improving LLMs without expensive labeled data.
 The experiments show significant improvements on several specific benchmarks, including fake news, safety, protein structure, etc. These benchmarks are not well studied in LLMs and lack high-quality data which make lots of sense to me.
The pipeline of applying UU learning is clear and easy to follow for other researchers

Weaknesses:
The core technical contribution of this paper is the use of UU learning for this new problem. However, since the submission is not in my area, I am unsure about the significance of its technical contribution.
Only one iteration is needed for most of benchmarks with UU learning which is limited the potential of the new method.

QUESTIONS:
From the experiments, most of the benchmarks become stable after only one iteration. Is this expected, or are there any other underlying reasons for this?

I am not sure if there are other approaches to enhance label quality, as the experiments seem to focus only on your newly proposed approach. Could you please clarify this further?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a label refinement method for improving the classification performance of large language models (LLMs) using a new iterative pseudo-labeling pipeline based on Robust Unlabeled-Unlabeled (UU) learning. The key idea is to leverage two unlabeled datasets with different positive class priors to iteratively denoise LLM-generated pseudo-labels. Unlike traditional self-refinement methods that rely on the LLM's own internal feedback mechanisms, this approach minimizes dependence on internal model knowledge and instead utilizes data-driven refinement through robust UU learning. Experiments across diverse domains—including low-resource languages, patent classifications, protein structure prediction, and LLM alignment—demonstrate performance gains and scalability under minimal human supervision\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n\nThe main idea to apply robust UU-learning iteratively to refine the pseudolabels of LLMs is interesting and novel in this context. \n\nOverall, the paper is well written with clear flow and presentation of ideas. A few things could be improved in notation. Appreciate the background, brief summary in 4.1, and Figure 1. \n\nEmpirical evaluation on several tasks, including sentiment analysis, patents and protein classification, and safety alignment, etc., shows the effectiveness of the proposed iterative method. \n\n\n\n**Weaknesses**\n\n\nThe method requires labeled data for estimating priors and other coefficients in the UU loss. Lack of such labeled data may produce unreliable estimates and degraded results. \nDo the baselines also use the same amount of labeled data? What if the first round of LLM uses these labeled points as in-context examples? \n\n\nThe same risks of confirmation bias as in pseudolabeling (self-training) still apply in this iterative labeling using UU-learning. There are several pseudolabeling baselines focused on correcting these issues; comparison against those can show the distinctions more clearly. \n\n\nPresentation (Minor): Notation in section 3.2 can be simplified, $\\tilde{C}_p$ and $\\tilde{C}_n$ are not actually positive or negative sets (unlike most of the notation in the paper, where subscripts clearly mean positive or negative samples). Maybe $\\tilde{C}_1$ and $\\tilde{C}_2$ might be good enough to avoid confusion. Colors and markers in the plots are not very clear. Please use easily distinguishable colors and markers and maintain consistency across figures.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe limitations are not discussed. It will be good to discuss the risk of confirmation bias, the requirements of labeled data, and the computational cost of multiple rounds of refinement.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}