Title: A Closer Look to Positive-Unlabeled Learning from Fine-grained Perspectives: An Empirical Study
Venue: NeurIPS 2025
Accepted: yes
Keywords: positive and unlabeled learning
Abstract: Positive-Unlabeled (PU) learning refers to a specific weakly-supervised learning paradigm that induces a binary classifier with a few positive labeled instances and massive unlabeled instances. To handle this task, the community has proposed dozens of PU learning methods with various techniques, demonstrating strong potential. In this paper, we conduct a comprehensive study to investigate the basic characteristics of current PU learning methods. We organize them into two fundamental families of PU learning, including *disambiguation-free empirical risks*, which approximate the expected risk of supervised learning, and *pseudo-labeling methods*, which estimate pseudo-labels for unlabeled instances. First, we make an empirical analysis on disambiguation-free empirical risks such as uPU, nnPU, and DistPU, and suggest a novel risk-consistent set-aware empirical risk from the perspective of aggregate supervision. Second, we make an empirical analysis of pseudo-labeling methods to evaluate the potential of pseudo-label estimation techniques and widely applied generic tricks in PU learning. Finally, based on those empirical findings, we propose a general framework of PU learning by integrating the set-aware empirical risk with pseudo-labeling. Compared with existing PU learning methods, the proposed framework can be a practical benchmark in PU learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a classic problem: PU Learning. It analyzes and compares two common types of algorithms for this problem (based on ERM and Pseudo-label), and proposes a new algorithm (GPU) by integrating the two categories based on empirical studies. The proposed method is evaluated through experiments on three typical datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper is well-structured and written in a fluent manner, making it easy to follow and understand.\n* It provides a comprehensive review of PU Learning, demonstrating a solid grasp of the research landscape in this field.\n\n**Weaknesses**\n\nThe core research objective of this work remains ambiguous: \n* If the study aims to establish a benchmark, expanding the experimental scope with more diverse datasets, detailed parameter configurations (e.g., learning rates) would significantly enhance reproducibility; \n* If the study intends to propose a novel algorithm, the performance needs substantial enhancement, as the method currently fails to outperform state-of-the-art approaches under most experimental settings. \n\nAddressing this ambiguity and reinforcing either the benchmark rigor or algorithmic superiority would strengthen the paper's scientific contribution.\n\nQUESTIONS:\n**1. Problem**\n\nIn real-world applications, one of the key challenges in PU Learning is the unknown positive sample distribution $\\pi$. Although techniques like MPE [1]/BBE [2] can estimate it, their accuracy in high-dimensional data remains uncertain. Could the author please discuss whether this issue was considered in the research?\n\n**2.Related Work**\n\nWhile the review of PU Learning is comprehensive, several representative studies have not been discussed or compared. For example: [2, 3, 4]. Could the author please include a discussion on how these works relate to the research? Comparing the method with these studies would help clarify the unique contributions and positioning of the approach within the field.\n\n**3. Method**\n* What is the specific motivation for combining SAPU and pseudo-labeling in the method? Which particular challenges in PU Learning does this combination address?\n* Similar algorithmic designs (ERM && pseudo-labeling) exist commonly in other weakly supervised learning problems [5] (e.g., UU Learning [6], CL Learning [7, 8], Learning with New Class [9, 10]). Would the method be applicable to these scenarios as well? Elaborating on the generalizability of the approach across related learning problems would add depth to the discussion.\n\n**4. Experiments**\n\nTo enhance the reproducibility and interpretability of the results, could the author please provide the following details:\n* Why were different backbones (LeNet-5 && CNN) used for different datasets instead of a unified one?\n* What was the rationale behind setting different epoch numbers for each dataset?\n* How were the positive and negative categories defined for each dataset? Different classification methods can lead to drastically different problem difficulties, as the difficulty of distinguishing between some classes is low while that of others is high.\n* Could the author elaborate on the process of setting and tuning hyperparameters (e.g., learning rates, weight decay)?\n\nProviding these details would greatly assist readers in understanding the experimental design and reproducing the results, thereby strengthening the credibility of the findings.\n\n[1] Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure\n\n[2] Mixture proportion estimation and PU learning: A modern approach\n\n[3] Positive-unlabeled learning with label distribution alignment\n\n[4] Positive-unlabeled learning using random forests via recursive greedy risk minimization\n\n[5] Machine Learning from Weak Supervision: An Empirical Risk Minimization Approach\n\n[6] On the minimal supervision for training any binary classifier from only unlabeled data\n\n[7] Learning from complementary labels\n\n[8] Unbiased risk estimators can mislead: A case study of learning with complementary labels.\n\n[9] Handling New Class in Online Label Shift\n\n[10] An Unbiased Risk Estimator for Learning with Augmented Classes\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-06-02"}

comments: SUMMARY:
This paper categorizes existing PU learning methods into two types and summarizes or improves them, respectively. For the first type of methods, this paper proposes SPU based on aggregate supervision and proves that it is risk-consistent. It also achieves good performance. For the second type of methods, this paper summarizes their techniques and evaluates the tricks used in these methods. Finally, this paper combines these two types of methods and proposes GPU, which demonstrates excellent performance on multiple datasets.

STRENGTHS AND WEAKNESSES:
Strengths:

1. This article provides a thorough discussion of many existing works and offers many valuable tricks.
2. The performance of GPU in most scenarios is promising.
3. The article is clearly written.

Weaknesses:

1. The novelty is limited. SAPU is simply a straightforward application of aggregate supervision in PU learning.
2. This article seems to be merely a summary of existing methods. Although it indicates that certain tricks are effective, it does not explain why.
3. This article lacks a detailed introduction to aggregate supervision.
4. The datasets used in this paper are too small and too simple. The authors should conduct experiments on larger and more realistic datasets.

QUESTIONS:
1. Why does the choice of loss function in Table 4 have such a significant impact? How should we choose an appropriate loss function?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes systematic analysis and practical improvement methodologies to the PU learning through fine-grained empirical investigation. The authors systematically organize PU learning methods into two fundamental classes: unbiased empirical risk methods and pseudo-labeling methods. For unbiased empirical risk methods, the authors propose a novel set-aware empirical risk (SAPU) from the perspective of aggregate supervision. For pseudo-labeling methods, systematic ablation analyses are conducted. Finally, the authors present GPU framework, achieved by unifying SAPU with pseudo-labeling strategies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis paper not only categorizes PU learning into unbiased empirical risk methods and pseudo-labeling methods from a novel perspective, but also provides fine-grained analysis of specific techniques within each class, encompassing multiple dimensions including loss functions, pseudo-labeling techniques, threshold strategies, and general tricks. \n\n2.\tThis paper introduces the concept of aggregate supervision into PU learning, proposing the SAPU empirical risk framework, and subsequently presents the GPU framework by integrating it with pseudo-labeling related strategies. \n\n3.\tThe analysis is thorough and the experiments are comprehensive.\n\nWeaknesses: \n\n1.\tThis paper fails to provide a clear classification standard for the distinction between \"soft pseudo-labeling\" and \"hard pseudo-labeling.\" Additionally, this paper lacks clear interpretation of results in the loss function section or general selection guidelines.\n\n2.\tThe technical classification and methodological descriptions are inadequately detailed. Key technical descriptions lack clarity, particularly the specific computational process for class-specific adaptive thresholding, which requires more explicit explanation.\n\n3.\tThis paper exhibits issues with inconsistent symbolic representation (e.g., SAPU vs saPU) and formula writing errors throughout the manuscript.\n\nQUESTIONS:\n1.\tIn Table 1, why is the P³Mix classified as a \"hard pseudo-labeling\" method? Should not the mixup technique naturally produce \"soft pseudo-labels\"?\n2.\tRegarding Table 4: a) Why do other methods generally demonstrate superior performance with double-hinge loss, while SAPU consistently shows relatively poorer results with double-hinge loss? b) Different loss functions exhibit varying performance across different datasets. How can this phenomenon be explained? Are there general selection guidelines available?\n3.\tThe description about class-specific adaptive thresholding is unclear in lines 210-214. In FlexMatch, $\\tau_p$ is fixed at 0.95, but here it appears to continuously change based on the previous epoch's threshold. Additionally, how is $ \\mathcal{C} $ computed?\n\nMinor Issues:\n\na) Unify notation: SAPU or saPU. \n\nb) Eq.10: $ j=1 $ should be $ i=1 $. \n\nc) Line 145: $ \\mathcal{S}_ i=\\{\\mathbf{x}_ {ij}\\}_ {i=1}^{S} $ should be $ \\mathcal{S}_ i=\\{\\mathbf{x}_ {ij}\\}_ {j=1}^{S} $.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This work conducts empirical studies on PU learning methods and proposes a novel PU framework, which combines set-aware empirical risk with pseudo-labeling techniques. Extensive experimental analysis demonstrates the effectiveness of the PU framework.

STRENGTHS AND WEAKNESSES:
Strengths:
This work innovatively proposes the GPU framework that combines set-aware empirical risk and pseudo-labeling techniques, featuring a modular structure for easy portability and extension. The work provides theoretical proof of risk consistency for the SAPU method with detailed derivations, and is well-written with clear organization.
Weaknesses:
1.	When the SCAR assumption is violated, are the methods in this work still applicable?
2.	While Theorem 3.2 provides convergence guarantees, this work lacks deeper theoretical analysis, such as convergence rates and other fundamental properties.
3.	Why does the incorporation of moving average techniques lead to performance degradation? This phenomenon appears counterintuitive and requires explanation.
4.	The GPU framework's performance improvements over existing methods remain within statistical error margins in most cases, and even fails to achieve optimal performance in some scenarios. What accounts for this limitation?

QUESTIONS:
Please refer to the “Weaknesses”.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02