PU-BENCH: A UNIFIED BENCHMARK FOR RIGOROUS AND REPRODUCIBLE PU LEARNING
Authors: Qiuyi Chen
Haiyang Zhang
Leqi Zhang
Changchun Li
Jia Wang
Wei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: PU learning
semi-supervised leaning
benchmark
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Positive-Unlabeled (PU) learning, a challenging paradigm for training binary classifiers from only positive and unlabeled samples, is fundamental to many applications. While numerous PU learning methods have been proposed, the research is systematically hindered by the lack of a standardized and comprehensive benchmark for rigorous evaluation. Inconsistent data generation, disparate experimental settings, and divergent metrics have led to irreproducible findings and unsubstantiated performance claims. To address this foundational challenge, we introduce **PU-Bench**, the first unified open-source benchmark for PU learning. PU-Bench provides: 1) a unified data generation pipeline to ensure consistent input across configurable sampling schemes, label ratios and labeling mechanisms; 2) an integrated framework of 18 state-of-the-art PU methods; and 3) standardized protocols for reproducible assessment. Through a large-scale empirical study on 8 diverse datasets (**2880** evaluations in total), PU-Bench reveals a complex yet intuitive performance landscape, uncovering critical trade-offs between effectiveness and efficiency, and systematically mapping method robustness against variations in label frequency and selection bias. It is anticipated to serve as a foundational resource to catalyze reproducible, rigorous, and impactful research in the PU learning community. The source code is publicly available at <https://github.com/XiXiphus/PU-Bench>.

Published as a conference paper at ICLR 2026
PU-BENCH: A UNIFIED BENCHMARK FOR RIGOROUS AND REPRODUCIBLE PU LEARNING
Qiuyi Chen', ; Haiyang Zhang 1*] Leqi Zhang', Changehun L Jia Wang', Wei Wang'
1 School of Advanced Technology, Xi' an Jiaotong-Liverpool University, China
2 Cooleee of Computer Science and Technology. Jilin University, China
Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University, China chnnyi,,che2002,,, 1 Leqi. z (erpg2233@ssuddere @student. xjtiu. edu. en (haiyang. zhang, , j aaa wang02, wei wang03 @xjtlu. edu.cn cn changchun 930 gma com
ABSTRACT
Dositive-Unlabeiie (PU) learning, a challenging paradigm for training binary clas- sifiers from only positive and unlabeled samples, fundamental many ap- plications. While numerous PU learning methods have been proposed, the re- search is systematically hindered by the lack of standardized and comprehen- sive benchmark for rigorous evaluation. Inconsistent data generation, disparate experimental settings, and divergent metrics have led to irreproducible findings and unsubstantiated performance claims. To address this foundational challenge, we introduce PU-Bench, the first unified open-source benchmark for PU learn- ing. PU-Bench provides: 1) unified data generation pipeline to ensure consis- tent input across configurable sampling schemes, label ratios and labeling mech- anisms; 2) an integrated framework of 18 state-of-the-arr PU methods; and 3) standardized protocols for reproducible assessment. Through a large-scale em- pirical study 8 diverse datasets (2880 evaluations in total), PU-Bench reveals a complex yet intuitive performance landscape, uncovering critical trade-offs be- tween effectiveness and efficiency, and systematically mapping method robustness against variations in label frequency and selection bias. It is anticipated to serve as foundational resource to catalyze reproducible, rigorous, and impactful re- search in the PU learning community. The source code is publicly available at htttps://gittuu..omm com/
Bench
INTRODUCTION
DesitiveUnnlabele (PU) learning tackles common classification problem where only some pos- itive examples are labeled and the unlabeled rest are a mixture of unidentified positives and true negatives (Elkan & Noto 2008 Bekker & Davis 2020). This setting arises frequently in real-world applications where negative examples are difficult or costly to annotate. For example, in recom- mender systems, it is known which items users like, but not which they dislike (Zhou et al. 1021 Zhang et al. 2021). This characteristic makes PU learning an essential technique across diverse hields, and the past few years have a rapid development in algorithm design for PU learning, such as disease-related gene identification (Yang et al.) 2012 Molaci & Jalili, 2025), drug-drug in- teraction prediction (Zheng al., 2019), document retrieval (Wang et 2024 Zhang et al. 2024)
and medical image classification Nagayy & Ukita 2021).
Despite these advances, a significant challenge remains due to the lack of a standardized, unified, and comprehensive benchmark for fair comparison among different algorithms. This issue causes two major obstacles to further development. First, there is a remarkable inconsistency in experi- mental setups and evaluation protocols across studies. Researchers often employ different datasets, varying data sampling strategies (e.g., case-control vs. single-training-sel (Bekker & Davis [020] Mielniczuk & Wawrzenczyk 2024), and divergent labeling assumptions (e.g., selected completely
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
at random vs. selected random) (Gong et al. 2021: 2025). which inevitably leads to inconsistent and incomparable results. As indicated in Appendix Table C.I these disparate settings are rather common, preventing fair and holistic understanding of algorithmic performance. Second, the lidity of performance claims is often undermined by the high sensitivity of PU methods to empirical factors. Our empirical results confirm that variations in the label ratio labeling assumption can be significant enough to alter relative performance rankings of state-of-the- algorithms. Given that these factors are not uniformly controlled in prior work, many published comparisons may not reflect the true capabilities of individual methods, leading to potentially unreliable conclusions.
To address these challenges and establish a solid foundation for future research, we propose PU- Bench, to the best of our knowledge, the first comprehensive, open- soence, and unified benchmark for PU learning. This work makes three primary contributions: 1) Unified Open-Source Bench- marking Framework: We design and release an open-source, modular framework for the rigorous and reproducible evaluation of PU learning algorithms. It features a configurable PU generator, a unified training pipeline, and a comprehensive evaluation suite to ensure fair and consistent compar- isons. 2) Comprehensive Empirical Study; We conduct the most comprehensive empirical study in PU learning date, benchmarking 18 representative methods across diverse datasets. Our perimental design covers 20 configurations per method-datee -atasst pair, including 14 label ratios under SCAR and 3 SAR mechanisms evaluated representative label frequencies. This protocol yields a total of 2,880 controlled evaluations, enabling rigorous assessment of both peak performance and robustness across the full spectrum of PU learning scenarios. 3) In-Depth Analysis and Action- able Guidelines: We provide extensive evaluation and analysis from various perspectives, including effectiveness, efficiency and complexity, as well as robustness to varying label ratios and selection bias. Our findings reveal the strengths and limitations of the current PU methods, and propose set of practical, data-driven guidelines for algorithm selection and design.
2 PRELIMINARIES
Problem Setup. In PU learning, an observed training sample consists of a of labeled positives
CP and an unlabeled set U that mixes positives and negatives. The standard one-sided labeling
assumption posits that only positives can be labeled, i.e., p(Y 1|5 1) 1 (equivalently,
p(S = 1 0) 0) (Elkan & Noto 2008 Bekker & Davis 2020). Let π = p(Y 1)
denote the class prior. The class-conditional densities are given by (+)) = p(r Y = 1) and
(=) p Y = 0). The marginal is the mixture f(x) =
(z) =f
10-mm1 (x). The objective
is to learn a decision function hg : X - [0, 1] that estimates the posterior p(Y = 1|X = x). Key
notations used throughout this paper are summarized in Appendix Table A.I
Data Sampling Scheme. Training data can be typically generated under two sampling schemes: the single-training-se (ss) scenario and the case-control (cc) scenario (Bekker & Davis 2020). In the ss scenario, the training set is created by drawing samples i.i.d. from the population distribution, and only the positive examples within this set have chance of being labeled. The scenario sumes that the sabeled-positive set is drawn i.i.d. from p(r Y = 1) and the unlabeled set is drawn i..dd from the population distribution p(x). The primary distinction between the two scenarios is the composition of the unlabeled sample. Under cc, the unlabeled data follow the population mixture of the class-conditional distributions p( Y 1) and p(x | Y = 0) with mixing proportion m. whereas the ss scenario uses a mixture with a different effective proportion because labeled positives are excluded from 24 (Mielniczuk & Wawrzenczyk, (204)). Consequently, PU methods should al- ways state and consider the sampling scheme when conducting experiments and interpreting results. A detailed illustration of the differences between ss and ce can be found in Appendix
Labeling Mechanisms. Under both sampling schemes, there is one set of samples drawn i.i.d. from the population distribution p(z) and one drawn i.i.d. from the positive class p(x Y = 1), governed
by labeling mechanism with propensity e(x)) = p(S 1 Y X, x)) (Bekker & Davis, 2020). In PU learning, the most widely used assumption is Selected Completely At Random (SCAR), which posits constant propensity, i.e., e(x) = (Bekker & Davis 2020). A more general assumption is Selected At Random (SAR). where the propensity depends on features, i.c., e(x) # c
(Bekker & Davis 2018
2

---

Published as a conference paper at ICLR 2026
Image Datusels
Minimization Estimatoos
enPIT 15 PUSB
MPF-PIT LBE-PU
Dis-PET PUET 201
MNIISTIE
P.MNIST
CIEAR 10 ADNI
Robust-PE 1 Holiate-PPI
LIGAN-PU PUI-CCPFF
D. Text Dattorts IMDe
Disamiiguaion
Supervised ERM
Seif-PU
PMM
2X
20News
Tabelar Datasets
Connect-4
Generative
Spambase
VAE-U 25
CGePPU
Matching
2017-2019
2020
2021
2022
2023
2024
Figure Overview the collected methods and datasets.
3 PU-BENCH
In this section, we introduce PU-Bench, a unified benchmark designed for PU learning. We first detail the scope of our study, covering the datasets and collected algorithms (Section 3.1), followed by a detailed explanation of the system architecture that ensures reproducible data generation, model training, and evaluation (Section 3.2)
3.1 DATASETS AND METHODS
Datasets. PU-Bench evaluates 8 widely used datasets spanning three different modalities: text
IMDb (movie reviews for sentiment classification) (Maas et al. (011)) and 20News (text classifica-
tion) Lang 1995); image- MNIST (handwritten digit classification) (LeCun et al. 1998). Fashion-
MNIST (F-MNIST) (Xiao et al. 2017), CIFAR-10 (natural image classification) (Krizhevsky &
Hinton
2009), and ADNI (structural MRI for Alzheimer's disease diagnosis) (Jack Jr et al.] (008); tabular Spambase (email spam detection) (Dua & Graff] (219), and Connect-4 (board game outcome prediction) (Dua Graff 2019). This diverse selection of tasks and domains ensures comprehensive assessment of model robustness and generalizability. Detailed descriptions of each dataset are provided Appendix B.1
Although these datasets have been widely adopted in PU learning, existing works often construct PU data in disparate ways, such as adopting different label ratios, making varied assumptions about the unlabeled data distribution (e.g., SCAR or SAR), and employing distinct sampling designs (e.g., ss vs. cc). These inconsistencies result in heterogeneous experimental settings and limit the repro- ducibility of reported results. To address this issue, PU-Bench provides systematically organized collection of datasets and standardized PU data generation pipeline, ensuring consistent input across methods and allowing researchers to focus on methodological development rather than data preparation.
Methods. PU-Bench has implemented total number of 18 established PU learning algorithms. The selection is based on methodological relevance, reproducibility, and general applicability. Specifically, prioritize influential and recent PU methods from top-tier venues, focusing ex- clusively on domain-agnostic methods with publicly available implementations or author-provided implementations. Methods with unavailable code and non-reproducible pipelines are excluded. A detailed summary of the selected methods is provided in Appendix Table B.3 and Fig.
We categorize the collected methods into three groups based on their algorithmic strategies, as illustrated in Fig. []] 1) Risk-Minimization Estimators: methods that directly minimize empiri- cal risk or its variants under PU constraints, including nnPU (Kiryo et al.) (017). PUSB (Kato) et al.] 2019). VPU (Chen et al.) 2020a). LBE-PU (Gong et al. (021). MPE-PU (Garg et al.] 2021 PUET (Wilton et al. 2022). Dist-PU (Zhao et al. 2022), PULDA (Jiang et al. 2023); 2) Disambiguation- Guidedd Supervised ERM: methods that first resolve label ambiguity in the unla- beled pool by constructing aseudo-labels or selecting proxy pegatives/posstiivss (often under class- prior constraints with group/meta signals), and then a standard supervised ERM model is trained on CP U This training phase often leverages mixup, consistency regularization, or iterative teacher-student self-training. This group includes Self-PU (Chen et al. 2020b), P3Mix (Li al.
3

---

Published as a conference paper at ICLR 2026
Stage3: Performance Evaluation Suite
Effectivenees
Efficiency
Matrics: AUG, F1 score, Accuracy, Precision, Recall
Training Duration
Memory Usage
Stage2: Unified Training Pipeline
Mokk-Mnimization Estimators
Disamblguatio-Calle
Generative Distribution Matching
Supervised ERM
cmpirical risk minimizaiion
pseado-label
generator
disctminato
Stage1: PU Data Generator
Split
Binary Cullagae
Daia Sampling
Labed Cantrol
Selection Simulate
p= Pr(x= 1y=
Ixit Confis
Single Training Set
(SCAR)
Dataset
several classes
S2 (LBB-1)
YAML
collapsing
Case- Control
(LDG-22
Chaton
Traing
Valddea
Teal
54 (PUSB)
Figure 2: The modular framework for PU-Bench.
2022
Robust-PU Zhu et al.
2023). Holistic-PU (Xinrui al. 2023 LaGAM-PU (Long al.,
2024. PUL-CPBF (Li et al.)
2024): 3) Generative Distribution Matching: methods that align pos-
itive and unlabeled distributions with model-induced predictions via generative adversarial mod- eling, including PAN (Hu et al., 2021), VAE-PU (Na et al. (2020) and CGenPU (Papić et al. 2023) This taxonomy provides a structured framework for analyzing methodological differences. Detailed descriptions of each method are provided in Appendix B.2
3.2 SYSTEM DESIGN
Despite the extensive research on PU learning, the field currently lacks standardized framework for empirical evaluation, posing a challenge to experimental reproducibility and fair comparison. This methodological inconsistency manifests across the entire experimental pipeline. Studies vary widely in their data generation processes: employing different sampling schemes, c.g. ss or ce (Miclniczuki & Wawrzenczyk 2024h, varying the size of the positive set, and operating under distinct labeling assumptions, e.g. SCAR or SAR (Gong et al.] (021). as detailed in Appendix C.T Furthermore, disparities in training protocols, hyperparameter optimization, and the choice of evaluation metrics (shown in Appendix C.4) make it difficult to aggregate findings or reliably observe the true state- of-the-art. To address these critical challenges, PU-Bench aims facilitate rigorous, reproducible, and transparent evaluation of PU learning algorithms. It is built upon three core, interoperable components that standardize the experimental workflow: 1) PU Data Generator for generating experimental conditions, 2) Unified Training Pipeline for consistent model execution, and 3) Performance Evaluation Suite for comprehensive analysis and comparison, as shown Fig.
PU Data Generator. The PU Data Generator responsible for systematically transforming stan- dard classification datasets into diverse and reproducible PU learning scenarios. The generation process follows structured, multi-stage pipeline. It begins with a binarization module that converts multi-class datasets into binary Sastive-Neeative (PN) format, which is then split into training, validation, and test sets, with the total number of samples in the training set and class prior π fixed. Then, the generator simulates the data collection environment by choosing data sampling scheme to define where the labeled samples LP are from; details of the implementation data sampling schemes by specifying the value label ratio c, which controls how many LP are sampled, and the labeling mechanism, which defines the selection strategy for LP. Our framework supports multiple mech-
the two shown in Appendix C.2 Finally, the labeling simulation is performed

---

Published as a conference paper at ICLR 2026
anisms to emulate real-world complexities, including (S1) the common SCAR assumption, where every positive has a uniform labeling probability: (S2) the destndeedeependent sampling strategy where the propensities are based the auxiliary posterior ((z) to favor high-posterior positives (Gong et al. (021);; (S3) another instance-dependent sampling strategy where the propensities em- phasize ambiguous or boundary positives (Gong al.] [021]; (S4) the posterior sharpening strategy where top-scoring positives are deterministically selected under a sharpened posterior (Kato et al., 2019). Details of the propensity functions and scoring rules are given in Appendix
Unified Training Pipeline. This module eurrently supports 18 state-of-thh-ar PU methods, de- tailed in Sec. [.1 To ensure rigorous reproducibility and eliminate confounding variables, PU-Bench employs a fully modular, configuration-drive tiated from external YAML descriptors. These descriptors specify model backbones, PU loss func- tions, and shared hyper- parameerr,,eters, including optimizer type, learning-rate schedules, and weight initialization. The framework accommodates multiple data modalities via specialized encoders for text, image, and tabular data. Training begins by loading the configuration and PU-formatted datasets, after which the corresponding PU learning criterion and learner are instantiated. A unified trainer automates forward/backward passes, loss computation, metric logging, and checkpointing. By allowing parameter adjustments through simple YAML edits, the framework ensures stream- lined, reproducible training, evaluation, and visualization across all PU methods.
training framework where all algorithms are instan-
Performance Evaluation Suite. The Performance Evaluation Suite is designed to provide holistic and statistically robust assessment of each algorithm. measuring performance across two primary di- mensions: predictive effectiveness and computational effiritiienaefiiicineeee.. This dual focus addresses a critical gap in the literature, where inconsistent reporting undermines the fairness of cross-method com- parisons and obscures the practical trade-offs of different approaches. To ensure a standardized evaluation, we implement a unified protocol where all metrics are computed on a held-out, ground- truth test set. For effectiveness, we record widely used evaluation metrics including accuracy (Acc), precision, recall, macro-F1, area under the ROC curve (AUC). To measure efficiency, the framework logs wall-clock time and peak GPU memory per epoch; a checkpoint is written when- ever validation-set macro-F1 hits new best, and the full config, seeds, metric traces, and hardware stats are archived for full reproducibility.
4 EXPERIMENTAL RESULTS
This section presents the primary empirical results of our study, where we benchmark 18 represen- tative PU learning methods across datasets. To establish a controlled and comparable baseline that aligns with established practices, all experiments are conducted under conventional PU setting. This protocol, the most widely adopted configuration in prior studies, simulates the case-control sampling scenario, assumes the SCAR labeling mechanism, and utilizes fixed label frequency of c 0.1. Detailed statistics of the simulated PU datasets, specific implementation settings such as the backbone models employed, and expanded set of performance metrics provided in Ap- pendix [ for full reproducibility and deeper analysis. To assess whether the observed accuracy gaps are robust random seed variability rather than artifacts of sampling noise, we additionally perform two-sided paired t-tests with Holm-Bonferrni correction between each PU method and nnPU (Kiryo et al.) (017) across all datasets; the detailed procedure and corrected p-values are summarized in Appendix D.2.2
4.1 EFFECTIVENESS COMPARISON
To establish practical performance ceiling and contextualize the results, we include fully super- vised PN classifier trained with complete label information as an oracle reference. According to the results presented in Table I Fig. and Appendix D.2.1 we have the following observations:
Risk- Minimiziinz Estimators perform strongly but with higher variance. As shown in Ta- ble I while this family often achieves the best results on specific datasets (e.g., Dist-PU on ADNI and CIFAR-10, LBE-PU MNIST and F-MNIST), it suffers from significantly higher variance and threshold sensitivity than disambiguation- counterparts, exemplified by LBE-PU's col- lapse ADNI under extreme lasitive-label imbalance (Appendix (.2.3). D.2.33 Within this group, nnPU
5

---

Published as a conference paper at ICLR 2026
Table I: Accuracy score of all PU methods across all datasets under the conventional setting. The best results are shown in bold and the second best are underlined. Entries marked with are signif- icantly better according to the two-sided paired t-tests in Appendix D.2.2
Type
Method
IMDb
20News
MNIST
F-MNIST
CIFAR-10
ADNI
Connect-4
Spambase
nnPU
[5]
24.37
88.81
94.85.0.00
96.6721.41
85, 65.30
65.75
a4
21.66.2.7
PUSB
P4
27.86:332
87.49
95.19s1.36
95.76s144
87.68"22.5
65.85:4.355
35,07a2.5
81.56=1.0
VPU
28.08 14:13
87.76
96.46" s3.ce
97.50g1.43
83.7613.1
66.31
85.62 18:0
79.25
Risk-
MPE-PU
77.65 17.65
86.68:455
95.90
97.79:203
83.41
68,47
04
81.11:3.45
Minimization
Estimatoes
L8E-PU
0
74.25
45.74
97.23" 50:00
98.42' 10:00
83.98 12.00
65
83.9410.1
6..53
PUFT
05.
16
84-2349
14.47
Dist.PU 36
77.88 am
88 a5.00
95. 25.70
95.51 95.31
88.09"
75.02" am
73.85
85.71" 1a:50
PULDA
75.30
26.67.000
96.60.00
96.92.0.42
87.02
67.9165
84.80
Self-PU S
74.05:0m
86.9355 1
90.86g1 on
93.2 93.22g1 an
76.73
66.41g:1
75.67
72, 12:30:
P3Mix-E
78.27 ass
84.42as
94.2621
a6
87.364210
69.21
84.14:3.0
Disambiguaton
P3Mix-C
77.48"
88 02
95.23s1.52
96.53
22.65
67.69
85. 65.10
$1.46:1.42
-Guided
Robust-PU
77.37aa.12
88 1
95.05
84.30as.19
67
81.66-3.0
Supervised
Holissic-UU PU
32
61.40s4.4:
75,20 75.20
92.75s1.0
62.16:2.43
$7.08
65.07
79 19.50ss.as
$6.59:1..5
LaGAM-PU
76.81a
14
26.22
85,08"
PUL PUL-CPBF
20
75.266s.m
84.92
88.31:1192
98.07"a0.a
80.27
66.11
88.12" 14
85.91
Generative
66.18eamz
17.51
76.5621.51
61.29
19.21
50.38a1.12
79.39
59.15-1.11
Distributiion
PAN 11
70.261n.4:
78.20a4
87.45.1.m
93.81at.
68.70am.12
61.99a4.3
78.37a300
38.71
Matching
CGeaP
70
46
863661
93.35
47.79
60.455..
79.42aa
31.66....2
PN
79.89.0.1
92.32
96.54.0.
98.94.0.22
94.8
82.01.0.1
92.3810.0
91.03_6.0
Risk-Minimization Estimators
Disambiquation-Guidee ERM AUC
Generative Distribution Matching
AUC
AUC
Recall
Recall
Recall
75 100
0,72 ad .00
Accuracy
Accuracy
Accuracy
Precision
Precision
Precision
PN
PUSB
WPU
PN
SeF-Pu
-- -PMMss PINI
PN
WA-PU
PAN - CGenPP
MPE: PU
LBE-PU PU
PUET
Dist PU
Robust-PU Holistic PU LaGAM PU
PUL-CPBF
PULDA
Figure 3: Performance of PU methods the ADNI dataset across five evaluation metrics.
and Dist-PU emerge as the most consistent performers. In contrast, VPU frequently shifts be- tween recall-dominant and precision-dominat behaviors depending on the dataset (recall-leaning on ADNI, precision-leamee leaning on CIFAR-10) The radar visualizations in Fig. and Appendix Fig. [O] underscore this instability. Despite extreme outliers like VPU's recall spikes or Self-PU's erratic shapes, nnPU remains a remarkably resilient baseline, maintaining a balanced configuration across diverse modalities.
Disambiguation-uuian -uided methods provide balanced and stable performance. While rarely achieving the absolute highest accuracy scores (Table [T. this family delivers the most reliable overall stability across modalities. Within this family, the two P3Mix variants lead and comple- ment each other: P3Mix-E pushes recall under heavy contamination (notably on ADNI, IMDb, and vision tasks), while P3Mix-C trades some recall for higher precision. Robust-PU is stable low- variance baseline, especially competitive on text (20News), and LaGAM-PU performs strongly on tabular data (Spambase); by contrast, Holistic-PU clearly trails across datasets. Consistent with Fig. 3 Fig. 9 and Fig. 11 this family produces balanced profiles over five metrics, reflecting well- rounded decision behavior. The main outliers are P3Mix-E's recall spikes with lower precision on ADNI/CIFAR- 10 and Holistic-PU's jagged high-recall/ow-preee data, but these remain infrequent and do not undermine the family's 's overall stability.
shapes on vision and tabular
Generative Distribution Matching methods lag behind overall. From Table I this family con- sistently underperforms across modalities and rarely enters the top two. While PAN and CGenPU achieve moderate success text data (e.g., 20News), they fail to generalize to vision (e.g., CIFAR-

---

Published as a conference paper at ICLR 2026
nnAU P3MIx-F
PUSB
VRU Robust-PU
MPP-FUU Holistic-PU
LBE-FU LAGAN-PU
RUET PUL-OPBF
Dist. VAE-PU
RULDA
Seir-Pu CGenPU
P3Mix-C
PAN
20News
CIFAR-10
Connect-4
FI Score
0.85
0.8
0.85
X
0.80
0.80
0.6
0.5
0.75
200
400
600
1000
2000
3000
200
400
500
Duration (seconds)
Duration (Seconds)
Duration (seconds)
0.9
FI Score
0.85
0.8
0.85
0.7
0.80
0.6
0.75
0.5
0,75
0.00
0.02
0.04
0.06
0.05
0.15
0.20
Max GPU Memory (GB)
Max GPU Memory (GB)
Max GPU Memory (GB)
Figure 4: Time and space analysis of PU methods on three representative datasets (20News, CIFAR- 10 and Connect-4)..
10) and medical imaging (ADNI). VAE-PU is the weakest and least stable estimator overall. This brittleness is exemplified by PAN's critical failure on Spambase, where discriminator-indeee liability collapse produces extreme recall-precision imbalance (Appendix D.2.3). As shown in the radar charts (Appendix Fig. TT), profiles are compact and jagged, indicating a severe imbalance be- tween FI, AUC, precision, and recall. Although rare outliers exist, such as the balanced profile of CGenPU on 20News, they are insufficient to offset the group's broader lack of robustness.
re-
Performance is Highly Contingent on Data Modality. A critical finding of our benchmark is that no single method emerges as a universal winner. The best method depends heavily on data modality. For instance, LBE-PU achieves sfate-of-the-art accuracy simple images (even surpass- ing supervised PN on MNIST) but degrades on complex ones like ADNI. contrast, VPU and P3Mix variants
more stable, delivering consistently strong though not always top-tier perfor- mance across a wider range of data types. Surprisingly. nnPU, despite its simplicity. remains highly competitive and often outperforms newer methods. This suggests that progress in the field is not always linear, and some earlier principles remain highly robust. These findings stress the need for comprehensive benchmarking and careful alignment of method choice with problem characteristics.
4.2
COMPUTATIONAL ANALYSIS
Fig. iillustraes the critical trade-off between predictive effectiveness (F1 score) and computational costs (training duration and GPU memory usage) across the selected datasets of diverse modalities. The results reveal substantial variation in the efficiency profiles of the benchmarked methods:
Training time. Foundational methods such nnPU and PUSB exceptionally efficient, com- pleting epochs in seconds on 20News and MNIST, and staying close to the lower bound on CIFAR- 10. This efficiency directly attributable their relatively simple objective formulations, which avoid complex generative or adversarial components. Dist-PU, VPU, and P3Mix introduce a moder- ate computational overhead. While slower than the simplest baselines, they balance advanced mod- eling components with efficient implementation. Their cost-effectiveness stems from building upon lightweight extensions, such as distributional losses, mixup strategies, or simple pseudo-labeling rather than fundamentally altering the training pipeline. In contrast, PUL-CPBF, Molistic-PU, and VAE-PU are the most computationally demanding. Their prolonged epoch durations are direct con- sequence of their complex, multi-stage architectures. These pipelines involve iterative processes like clustering, learning,, or adversarial training, which require multiple forward/backward passes the optimization of auxiliary models, incurring substantial computational costs.

---

Published as a conference paper at ICLR 2026
SeltPU
P3Mbx- P3Mix-C
Hollstic-PU
LaGAM-PU
PUL-CP8F
CGenPU
Risk-Minimization Estimators
Disambiguation-Guit Guideed ERM
Generative Distribution Matching
20News Accuracy
0.8
0.8
0.6
0.01
0.05
0.09
0.9
0.01
0.05
0.09
0.5
0.9
0.05
0.090.1
0.5
0.9
CIFAR-10 Accuracy
0.8
0.8
0.7
0.6
0.6
0.4
0.4
0.090.1
Connect-4 Accuracy
0.0
0.7
0.6
0.09
0.09
0.090.1
Label Ratio
Label Ratio
Label Ratio
Figure 5: Effectiveness of PU methods with respect to different label ratio.
Memory consumption. A clear divide in memory usage exists, directly linked to architectural complexity. A large group of methods, including nnPU, PUSB, VPU, Robust-PU, and Dist-PU, are highly memory-efficien
1 GB) they rely on a standard single-classifier pipeline. In contrast, methods with more complex designs have a significantly larger memory footprint. VAE-PU is the most demanding (up 7-8 GB) due to generative components -ecodder-eccoders, discrimina- tors), while others like P3Mix-E and PUL-CPBF increase memory by incorporating mechanisms like EMA teachers and multi-view data processing.
Trade-off analysis. group of methods, notably VPU, Self-PU and Dist-PU, achieves strong balance, c.g.. high Fl with short training time and low memory overhead. P3Mix- yield compa- rably high Fl scores at modest extra cost. In contrast, Robust-PU and MPE-PU yield only compet- itive performance while requiring more substantial resources. Established baselines like nnPU and PUSB are exceptionally efficient, requiring the least training times. However, this comes the ex- pense of modest reduction in F1 score compared to the top-tier methods. In contrast, Holistic-PU, VAE-PU and PUL-CPBF demonstrate less favorable trade-off. These methods are the most com- putationally intensive yet often yield lower or less stable F1 scores, severely limiting their practical applicability.
5 FURTHER ANALYSIS
In this section, we conduct a deeper investigation into the robustness the benchmarked methods. We systematically vary experimental conditions that reflect the quality of labeled data: the size of labeled samples controlled by label ratio and the underlying labeling mechanism controlled by propensity score.
5.1 IMPACT OF LABEL FREQUENCY ON MODEL PERFORMANCE
We vary the value of label ratio from 0.01 to 0.09 with step of 0.02 and from 0.1 to 0.9 with step of 0.1. Results on selected datasets are shown in Fig. 55 and Appendix [D.3] revealing distinct patterns of stability and label efficiency across the three families.
Overall performance trends. Most methods within the Mokk-Miniizzation Estimation and Disambiigation-Gruiie Guided ERM categories demonstrate remarkable label efficiency. Their perfor-
8

---

Published as a conference paper at ICLR 2026
120
Minake Grimaten
Searkguttoon Exoeniied
Genorat Dissibution Haseiig
100
Fl Score (c-0.05 (N)
60-
-88
c0-
40-
Score
20
PULDA
PUSS
HPE-PU
mau
vPu
Robordi P
P3MM
PIM
Sei
Pul-cpgr
La04
Holaatic PM
CGempi
PAN
Figure 6: Fl scores of PU methods under different labeling assumptions on CIFAR-10. The bars show the performance in low-label regime (c 0.05), while the lines indicate performance in a high-label regime (c 0.5). Methods with grey background are those especially designed for selection bias.
mance curves typically show rapid initial ascent at low label ratios (e.g. < 0.1) before gracefully saturating, indicating a strong capability to learn effectively from limited labeled examples. For instance, VPU and P3Mix-C approach their peak performance CIFAR-10 with ranging only from 0.03 to 0.05. In contrast, Generative Distribution Matching methods exhibit high sensitivity both label ratio and data modality. Their performance curves are often erratic. Some methods show competitive performance with extremely small labeled sets (e.g. VAE-PU and CGenPU on image and text data), demonstrating their potential deal with scarce data. However, this behavior does not generalize to all modalities. More importantly, these methods often exhibit poor scalability, showing little improvement as the label ratio c grows. Such limitations likely stem from the inher- ent instability of optimizing complex generative objectives under the sparse and ambiguous signals characteristic of PU data.
Analysis of notable performers and outliers. The analysis reveals several distinct performance profiles: 1) Efficiency: VPU and P3Mix variants are exceptionally effective in low-label regimes, demonstrating high label efficiency by achieving competitive accuracy with minimal supervision; 2) Scalability: LBE-PU exhibits notable late-stage gains, improving substantially as more labeled data becomes available, suggesting it scales well with supervision; 3) Instability: methods like PUL- CPBF and Holistic-PU show high performance variance, with performance fluctuating significantly low label ratios; 4) Saturation: Finally, methods like VAE-PU remain at a performance bottleneck, showing little improvement even as labeled data becomes plentiful, indicating that their primary limitations are architectural rather than related to data scarcity.
5.2 ROBUSTNESS TO SELECTION BIAS
To assess robustness to labeling bias, we evaluate all methods under three realistic SAR variants, in contrast to the standard SCAR setting. Experiments were conducted at both a low (c = 0.05) and high 0.5) label ratio, with results shown in Fig. and Appendix D.4
Low-c regime (c = 0.05). When labeled data is scarce, the choice of labeling mechanism has a profound impact. Under the standard SCAR assumption, top-tier methods, e.g. VPU, Dist-PU,, and P3Mix-C, maintain their dominance, while the selection- eelection-bias-wwarr methods like PUSB and LBE- PU remain competitive. However, when the mechanism shifts to SAR, universal performance degradation is observed. The Risk-Minimization Estimation group demonstrates strong robustness across different labeling assumptions, with no method suffering from catastrophic performance fail- ure. Crucially, methods explicitly designed for bias mitigation (i.c.. PUSB, LBE-PU) exhibit supe- rior resilience, incurring smaller performance penaltics than their SCAR-assuming counterparts. A notable anomaly is VAE-PU, which paradoxically achieves higher accuracy under SAR than SCAR, though its overall performance remains uncompetitive. Ultimately, these results confirm that the benefit of bias-aware modeling is prominent in the low-label regime.
High-c regime (c = 0.5), When labels are plentiful (c = 0.5), most methods converge to high Fl scores, and the performance gap between SCAR and SAR settings narrows significantly, though it

---

Published as a conference paper at ICLR 2026
does not vanish. Consistent with the low-c regime, the Risk-Minimization Estimation group tinues to demonstrate high stability. Crucially, we find that robust SCAR learners like VPU can outperform dedicated SAR-aware methods in this high-data regime. This finding highlights criti- cal interaction between label ratio and selection bias, implying that the optimal strategy for handling SAR conditions may depend the amount of labeled data available.
Practical Takeaways. These findings yield a clear practical recommendation. When label acquisi- tion is suspected be non-random, particularly when labeled data is sparse, employing a bias-aware method like PUSB or LBE-PU is highly preferable. Conversely, in scenarios where labels are plen- tiful and the mechanism approaches near- -ear- conditions, lightweight and robust methods with semi- semi-suupervied regularization (VPU, P3Mix) offer state-of-the-an performance.
CONCLUSION AND FUTURE DIRECTIONS
Conclusion. This work addresses the absence of a standardized benchmark PU learning by introducing the open-source framework, PU-Bench. Through this framework, we conducted the largest systematic evaluations to date in this field, analyzing 18 representative algorithms across an extensive range of datasets and conditions. The results demonstrate that the performance of PU learning methods highly context-dependent, varying significantly with data modality, label frequency, and labeling mechanism. As foundational empirical study, this work provides the necessary grounding to guide future theoretical and algorithmic advancements. We anticipate PU- Bench serve cornerstone for the community, accelerating research by providing rigorous, standardized toolkit for comparing the state-of-the-ar and inspiring next-generation solutions.
Future directions. Through our empirical analysis, we identify the following future directions: 1) Rigorous evaluation: Our benchmark reveals that more recent and complex methods can be outperformed, at least some datasets, by simpler baselines (i.e., nnPU, VPU). This underscores the necessity for future research to conduct more rigorous, standardized evaluations against these strong, efficient baselines to validate claims of novelty and improvement. 2) Real-world stress test: A key vulnerability exposed by our study is the poor performance of most methods under severe data constraints (e.g., high label sparsity and selection bias), as characterized by many real-world applications. This reveals a significant gap between current algorithmic capabilities and practical needs. Therefore, critical direction for future research is the design of methods that inherently robust to the challenges of extremely limited and biased supervision.
ACKNOWLEDGMENT
This research was supported by the Collaborative Research Project (RDS10120240248) at Xi'an Jiaotong- Liverpool University. Additionally, this research has received the financial support pro- vided by the Fundamental Research Funds for the Central Universities, Jilin University.
REPRODUCIBILITY
We provide full configuration files and scripts reproduce all experiments. All results are determin- istically reproducible from fixed random seeds: we set and log seeds for dataset splits, data loaders, model initialization, and CUDA/cuDNN backends. Each experiment is specified by single YAML. configuration and a seed; re-running with the same seed exactly reproduces the reported metrics. Detailed packaging and usage instructions are provided in Appendix
REFERENCES
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data under the selected at random assumption. In Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 94 of Proceedings of Machine Learn- ing Research, pp. 8-22. PMLR, 2018. URL http::: htttps://prroeeedingg. mlr. mt...eess//99/ bekker1 8a. html
10

---

Published as a conference paper at ICLR 2026
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: survey. Machine Learning. 109(4):719 19-9), 2020. doi: 000000///10994002-00080000
Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition, 10(7):1145-11555 1997.
Hui Chen, Fangqing Liu, Yin Wang, Liyue Zhao, and Hao Wu. A variational approach for learning from positive and unlabeled data. Advances in Neural Information Processing Systems, 33:14844 14854, 2020a.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. International Confer- ence on Machine Learning, pp. 1510-1519. PMLR, 2020b.
Dheeru Dua and Casey Graff. Uci machine learning repository. http:///rrchive.css uci.
edu./ ml 2019.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, pp. 113-220. ACM, 2008. doi: 40.1144/1100180.11444 109.144499990009999.
Saurabh Garg, Yifan Wu, Alexander J Smola, Sivaraman Balakrishnan, and Zachary Lipton. Mixture proportion estimation and learning: A modern approach. Advances in Neural Information Processing Systems, 34:8532-8544, 2021.
Chen Gong, Qizhou Wang, Tongliang Liu, Bo Han, Jane You, Jian Yang, and Dacheng Tao. Instance-dependent positive and unlabeled learning with labeling bias estimation. IEEE trans- actions on pattern analysis and machine intelligence, 4668):41634417,, 2021.
Chen Gong, Muhammad Imran Zulfiqar, Chuang Zhang, Shahid Mahmood, and Jian Yang. A recent survey on instance-dependent positive and unlabeled learning. Fundamental Research, 5(2):796- 803, 2025. ISSN 2667-3258. doi: http::/ddi.org/I/1 h0tp://00000000..m00022200000000 .199. URL https:
7/www. www..sciencediree... rect. com/ htt.://cw.......cooaaaaaaaooaaaa
Wenpeng Hu, Ran Le, Bing Liu, Feng Ji, Jinwen Ma, Dongyan Zhao, and Rui Yan. Predictive adversarial learning from positive and unlabeled data. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7806-7814, 2021.
Clifford R Jack Jr, Matt A Bemsstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Har- vey. Bret Borowski, Paul J Britson, Jennifer L Whitwell, Chad Ward, et al. The alzheimer's s discase neuroimaging initiative (adni): Mri methods. Journal of Magnetic Resonance Imaging, 27(4))685-69) 2008.
Yangbangyan Jiang, Qianqian Xu, Yunrui Zhao, Zhiyong Yang. Peisong Wen, Xiaochun Cao, and Qingming Huang. Positive-unlabeled learning with label distribution alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, (511)))15345-13666 2023. doi: 10. 2L.1099/TAMM.2223.33331
Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selection bias. In International conference on learning representations, 2019.
Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in neural information processing systems, 30, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Ken Lang. Newsweeder: Learning to filter netnews. In Proceedings of the 12th International Conference on Machine Learning, pp. 331-339, 1995.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11):2278-2324 1998.

---

Published as a conference paper at ICLR 2026
Changchun Li, Ximing Li, Lei Feng, and Jihong Ouyang. Who is your right mixup partner in positive and unlabeled learning. In International conference on learning representations, 2022.
Changchun Li, Yuanchao Dai, Lei Feng, Ximing Li, Bing Wang, and Jihong Ouyang. Positive and unlabeled learning with controlled probability boundary fence. In International Conference on Machine Learning. pp. 27641-22653 PMLR, 2024.
Lin Long, Haobo Wang, Zhijie Jiang, Lei Feng, Chang Yao, Gang Chen, and Junbo Zhao. Positive- unlabeled learning by latent group-awart meta disambiguation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23138-23147 23147, 2024.
Andrew L Maas, Raymond E Daly, Peter T Pham. Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, 2011.
Jan Mielniczuk and A Wawrzeńczyk.. Single-sample versus case-control sampling scheme for posi- tive unlabeled data: the story of two scenarios. Fundamenta Informaticae, 191(2):1-17, 2024
Sepideh Molaci and Saced Jalili. Disease candidate genes prediction using positive labeled and unlabeled instances. BMC Medical Genomics, 18(1):73, 2025.
Byeonghu Na Hyemi Kim, Kyungwoo Song, Weonyoung Joo, Yoon-Yeong Kim, and Il-Chul Moon. Deep generative positive-unlabcle learning under selection bias. In Proceedings of the 29th ACM international conference information & knowledge management, pp. 115--1164, 2020.
Masashi Nagaya and Norimichi Ukita. Embryo grading with unreliable labels due to chromosome abnormalities by regularized pu learning with ranking. IEEE transactions on medical imaging, 41(2):320-3 331,200-3311000 2021.
Aleš Papič, Igor Kononenko, and Zoran Bosnić. Conditional generative positive and unlabeled learning. Expert Systems with Applications, 224:120046, 2023. ISSN 0957-4174. doi: https://doi. org/10. http::/jessaa.22..2220000 URL http://:wwwwww..ccceeeeeeeeeeeee wwwp://twwt..ccccc.ee/e.c...ceee
David M W Powers. Evaluation: From precision, recall and f-measure to roc, informedness, marked- ness and correlation. Journal of Machine Learning Technologies, 2(1):37-63 2011.
Yuqi Wang, Qiuyi Chen, Haiyang Zhang, Wei Wang, Qiufeng Wang. Yushan Pan, Liangru Xie, Kaizhu Huang, and Anh Nguyen. Biomedical information retrieval with positive-umlabeled learn- ing and knowledge graphs. ACM Trans. Intell. Syst. Technol., November 2024. ISSN 2157-6904. doi: 10. 10.1145/3702647, URL ftttps://dot www.:/i...rr11..1 1ttp//11..11/33726...1
Jonathan Wilton, Abigail Koay, Ryan Ko, Miao Xu, and Nan Ye. random forests via recursive greedy risk minimization. Advances in Neural Information Process- ing Systems, 35:24060-24071, 2022.
learning using
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprini urXiv: / 708.07747 2017.
Wang Xinrui, Wenhai Wan, Chuanxing Geng, Shao- Yuao Li, and Songean Chen. Beyond myopia: Learning from positive and unlabeled data through holistic predictive trends. Advances Neural Information Processing Systems, 36:67589-67602. 2023.
Peng Yang, Xiao-Li Li, Jian-Ping Mei, Chee-Keong Kwoh, and See-Kiong Ng. Positive-unlabelae learning for disease gene identification. Bioinformatics, 28(20):2640-2644 2012.
Haiyang Zhang, Ivan Ganchev, Nikola S. Nikolov, and Mark Stevenson. Userreg: A simple but strong model for rating prediction. In ICASSP 2021 2021 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3595-3599, 2021. doi: 10.1 10.1109/CASSS339228.8 2021 299.9999992222229991666666,
12

---

Published as a conference paper at ICLR 2026
Haiyang Zhang, Qiuyi Chen, Yanjie Zou, Jia Wang, Yushan Pan, and Mark Stevenson. Document set expansion with Dositive-unlabeled learning using intractable density estimation. In Nicoletta Calzolari. Min-You Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREEC-COLING 2024), pp- 5167-5133, Torino, Italia, May 2024. ELRA and ICCL. /BL.itpsss//ooaaattooooooooorrg
1rec- mainn. 460/
Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-pu: Dositive-unlabeled learning from a label distribution perspective. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14461-14470, 2022.
Yi Zheng, Hui Peng, Xiaocai Zhang, Zhixun Zhao, Xiaoying Gao, and Jinyan Li. Ddi-pulearn: a uositiie-umlabeled learning method for large-scale prediction of drug-drug interactions. BMC bioinformatics, 20(Suppl 19):661, 2019.
Yao Zhou, Jianpeng Xu, Jun Wu, Zeinab Taghavi, Evren Korpeoglu, Kannan Achan, and Jingrui He. Pure: Positive-unlabele recommendation with generative adversarial network. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2409-2419, 2021.
Zhangchi Zhu, Lu Wang, Pu Zhao, Chao Du, Wei Zhang, Hang Dong, Bo Qiao, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Robust positive-unlabeled learning via noise negative sample self-correction. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3663-3673, 2023.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
TABLE OF CONTENTS
Section
Subsection
Title
Page
A
Notations
15
Dataset and Methods
B.1
Details of Datasets
13
B.2
Details of Implemented Methods
16
C
Data Generation Settings for PU Learning
20
C.1
Data Generation Settings for Existing PU Learning Methods
20
C.2
Lingletraining-set and case-control Conversions
21
C.3
Labeling Strategies
22
C.4
Evaluation Metries for PU Learning Methods
22
D
Experimental Settings and Results Experimental Settings
23
D.I
23
D.2
Performance under the Conventional Setting Effectiveness of PU Methods w.r.t. Label Ratio Robustness to Selection Bias
27
D.3
32
D.4
33
Computation Resources Package, Documentation, and Maintenance Use of Large Language Models (LLMs)
34
G
14

---

Published as a conference paper at ICLR 2026
A NOTATIONS
Table A.I: Summary of notation. Uppercase letters (e.g., X) denote random variables; lower- case (e.g., x) denote realizations. Densities (e.g.. f(x)) are evaluated pointwise; expectations (e.g.,
Ex~ (xX)) are taken with respect to the distribution of X.
Symbol
Description
X
Feature vector, E X' Latent class label [0. 1) (positive= 1)
Y
S
Labeling indicator {0, 1} (labeled= 1)
D
Observed training sample [[2g, 1
P. .V
Sets of positives and negatives the underlying PN dataset
CP
{ x (2,, 8;) E DD 8; = -1] (labeled positives) {z, (z,, 8,) D. 8, 0 (unlabeled; mixture of positives and negatives)
U
Up
Unlabeled positives, Up 2
N
Training set size used to construct PU splits Target size of the labeled-posiin possiiie set LP Size the unlabeled set in the PU training split Validation set size
N
N,
N.
N N
Positive/negative counts the PN training set Positivennegative counts in the test sel
N
N
Class prior, p(Y =1)
e(xr
Labeling propensity. p(S = 1 = 1,x = Ele(X)
Label frequency, p 1Y
f(x)
Marginal density of X Positive density, p(z Y = 1) and negative density, p(z|Y = 0) gabeled-positivee density, p(x| = and unlabeled density, p(r|S = 0)
f+(x). f2(x)
fe(r), (a())
ho(x)
Estimate of p(Y 1|X I))
p(x)
Estimated posterior p X=x)
a()
Logistic sigmoid, (2) I+e
Parameter vector in the logistie-regression example for p(x) o(w' x)
w
B DATASET AND METHODS
B.1 DETAILS OF DATASETS
2011 is sentiment analysis dataset containing 50,000 movie reviews from the Internet Movie Database, evenly split into positive and negative labels. Reviews are preprocessed to remove HTML tags and are tokenized for text classification tasks.
IMDb Maas et al.
20News (Lang 1995) a text classification dataset consisting of approximately 18,000 newsgroup documents evenly distributed across 20 topics. Each document is represented as bag-of-words vector, with labels corresponding to its topic category.
MNIST (LeCun et al. 1998) contains 70,000 grayscale images of handwritten digits (0-9), each of size 28x28 pixels. The dataset is split into 60,000 training and 10,000 testing images for digit classification.
Fashion-MNIST (Xiao et al. 2017) consists of 70,000 28 x8 grayscale images of fashion items across 10 classes, also F-MNIST to refer to Fashion-MNIS in this paper.
CIFAR-10 (Krizhevsky & Hinton 2009) comprises 60,000 32x32 color images evenly divided into 10 object classes. Each class contains 6,000 images, split into 50,000 training and 10,000 test samples.
ADNI (Jack Jr al. 2008 (Alzheimer's Disease Neuroimaging Initiative) is medical imaging dataset containing structural MRI scans for Alzheimer's disease research. Each scan is labeled according to the patient's clinical diagnosis.
15

---

Published as a conference paper at ICLR 2026
Spambase (Dua & Graff 2019) contains 4,601 email samples represented by 57 numerical features, such as word frequency and character frequency. Labels indicate whether each email is spam or not.
Connect-4 (Dua & Graff 2019 is dataset derived from the board game "Connect Four." It contains 67,557 game states represented by 42 categorical features corresponding to the positions on a 6 x board. Each position can take one of three values: (first player's piece), (second player's piece), or b (blank). Labels indicate the outcome of the game from the perspective of the first player: win, loss, draw.
Table B.I: Class labels and their indices for multi-class datasets.
Dataset
Classes (index)
20News
Alt (0), Comp (1), Misc (2), Rec (3), Sei (4), Soc (5), Talk (6)
IMDh
Negative (0), Positive (1)
MNIST
0, 1, 2, 3, 4, 5, 6, 8,
F-MNIST
T-Shirt/ (0), Trouser (1), Pullover (2), Dress (3), Coat (4), Sandal (5), Shirt
(6), Sneaker (7). Bag (8), Ankle Boot (9)
CIFAR-10
Airplane (0), Automobile (I), Bird (2), Cat (3), Deer (4), Dog (5), Frog (6),
Horse (7), Ship (8), Truck (9)
Alzheimer MRI
NonDemented (0), VeryMiildDemented (1), ModeraleDementeed (2), MildDe-
mented (3)
Connect-4
Loss (0), Win (1), Draw (2)
Spambase
Not Spam (0), Spam (1)
Table B.2: PU learning datasets and statisties with index- class mapping. Nrr, _N," denote the number of positive and negative samples in the training set, respectively, Nie Nie for the test set; No for validation size.
Dataset
Pos. VS (eg.(Indices)
Input Size
Train (N",, NN)
Test ,N
Validation
Total Size
20News
0,1.2.3 VS.
384
(6,326, 4,874)
3,,25)
0.01 (N. 114)
18,816
VS.
(12,375, 12,375)
(12,500, 12,500)
0.01 (N, 250)
50,000
MNIST
0,2,4.6,8 VS. 1,3.5,7,9
28x28
(29,197, 30,203)
(4,926, 5,074)
0.01 (N. 660)
70,000
F-MNIST
0,2,3,4,6 VS. 1,5,7,8,9
28x28
(29,700, 29,700)
(5.000, 5,000)
0.01 (N. 600)
70,000
CIFAR-10
0,1,8,9 VS. 2,3,4.5,6.7
32x32x3
(19,800, 290700)
(4,000, 6,000)
0.01 (N. 500)
60,000
ADNI
1,2,3
128x
(2,352, 2,516)
(622, 658)
(N, 52)
6,400
Connect-4
VS. 0,2 VS.
126
(35,222, 18,28 (8,282)
(8,595, 4,617)
0.01 (N, 541)
67,557
Spambuse
57
(1,435, 2,208)
(363,558)
0.01 (N. = 33)
4,601
B.2 DETAILS OF IMPLEMENTED METHODS
nnPU (Kiryo al. 2017) introduces a non-negative risk estimator for PU learning. Unlike unbiased PU learning where the empirical risk can become negative during optimization, nnPU constrains the risk to remain non-negative by truncating any negative component to zero. This modification prevents the model from overfitting to noise when training flexible models such as deep neural networks with limited positive data.
PUSB (Kato et al., 2019) operates under the invariance of order assumption, which posits that the class posterior and the labeling probability induce the same ordering on the input space. It estimates the density ratio between labeled positive and unlabeled data either by minimizing pseudo classi- fication risk or through direct density ratio estimation. The classifier is then constructed by ranking instances according to this density ratio and setting the decision threshold at the precision-recal breakeven point.
VPU Chen et al., 2020a) introduces a variational principle that avoids class-prior estimation by directly minimizing the KL divergence between the positive data distribution and the distribution induced by the classifier. The method optimizes variational loss computed as the difference be- tween the log expectation of the classifier output over unlabeled data and the expectation of the log
16

---

Published as a conference paper at ICLR 2026
Table B.3: Summary of the collected methods.
Type
Method
Repositorics
Reference
mnPU
15
ALLpe://githe httpa//oooo//oooo/o/////--e
NeurIPS-2017
PUSB
14
:Ltpe:/ /gtchu/ httpa///wwoaaaaa/t//eeeeeeaaii.
ICLR-2019
VPU
4
Ettpa:/ htppa:///ichht com/ Feynnan//pnn
NeurIPS-2020
Minimization
Risk-
MPE-PU 8
Ettpa:/ http://www.ccoccmmaaaa/a/.......
NeurIPS: S02IPP-22221
Estimators
LBE-PU
0
Unpublished. required
JEEE TPAMI-2022
https: http:///ithub... http:///stttaateeeeeoeeaaaaaaaaa
NeurIPS-2022
Dist-PU
36
CVPR-2022
PULDA
sttpa:/ nom wttp:///wwaaagggggggg/uuuu
IEE TPAMI 2023
Self-PU 5
ICML-20000
P3Mix 19
Avalable upon requesi
ICLR
Disambiguaaion
-Guided
Robust-PU
39
https: http:///thhhhchccc.e httpaa/cccoo/oeeeee//e/0/
KDD-2023
Supervised ERM
Holistic-PU
Sttps:/ happ:://iithht githob. com/ Max wwtm///w3///oostteeeooo
NeurIPS-2023
LaGAM-PU 21
sttpa:/ /ithhb lammmllonn-aaao LLGGAA
CVPR-2022 2024
PUL-CPRF 20
Unpublished, required by
ICMI,22024
Generative
VAE
25
https://githin com
CIKM-2020
Distribution
PAN 1
httpa:/ /ttl://gttha wtt.chhbbboomooooo.o morn motm//mmonnnnnooon nowwnass
AAAI-2021
Matching
CGeaPU
27
:ttps://githun ccmm com/ wpwpich/Ceeeeeeeepeee ch/CeenPU
ESWA-20Z3
output positive data. To handle limited positive data and prevent overfitting, VPU incorpo- rates a MixUp-based consistency regularization that enforces prediction smoothness on interpolated samples between labeled positives and unlabeled data.
MPE-PU Garg et al. 2021) addresses PU learning through two coordinated components: Best Bin Estimation for mixture proportion estimation and Conditional Value Ignoring Risk for classifica- tion. BBE estimates the proportion of positives in unlabeled data by leveraging a Positive-versus- Unlabeled classifier to identify a high-score bin containing concentrated subset of positives, then computing the ratio of sample fractions in that bin. CVIR trains the classifier by iteratively removing the estimated proportion of unlabeled samples with the highest classification loss and training on the remaining samples as provisional negatives. The TED" framework alternates between updating the proportion estimate via BBE and refining the classifier via CVIR until convergence.
LBE-PU (Gong et al. 2021 addresses deeence-depedeentt PU learning where the probability of labeling positive example depends on its features, known as labeling bias. It formulates the prob- lem via graphical model characterizing the relationships among input features, latent true labels, and observed labeling status. The method jointly estimates the labeling bias function and the clas- sifier parameters through maximum likelihood estimation, which is optimized via the Expectation- Maximization algorithm with gradient-bart based updates applicable to both linear (logistic) and deep (MLP) model instantiations.
PUET (Wilton et al. 2022) introduces a recursive greedy risk minimization framework for learning decision trees from PU data. It reinterprets tree learning directly minimizing PU-based risk estimators (uPU or nnPU) rather than traditional impurity measures like Gini or entropy. The method constructs PU Extra Trees by recursively partitioning nodes maximize the PU risk reduction, where each split is selected based on closed-form risk estimates computed from the positive and unlabeled subsets that node. Additionally, the framework provides a risk reduction importance metric that directly quantifies each feature's contribution to minimizing the empirical risk on PU data.
Dist-PU Zhao et al.) 2022) addresses the PU methods where classifiers tend over over-predict the negative class as training progresses by introducing a label distribution alignment perspective. It enforces consistency between the expected value of predicted labels and the ground-truth class prior over the unlabeled data, thereby globally constraining the proportion of negative predictions. To avoid trivial solutions where predictions uniformly concentrate around the class prior, the method incorporates entropy minimization to en- courage confident predictions and Mixup regularization mitigate confirmation bias and smooth decision boundaries.
preference inherent in cost-sensitive
PULDA (Jiang al., 2023) extends the label distribution alignment framework by introducing margin-base based formulation that enhances instance-wise discriminability. Beyond aligning the expec- tation of predicted labels with the ground-truth distribution, it incorporates functional margins via
17

---

Published as a conference paper at ICLR 2026
confidence penalization terms that push model outputs away from ambiguous regions near the de- cision boundary, preventing under-confident solutions. Furthermore, PULDA unifies this objective with a class prior estimation process (using BBE) to eliminate reliance oracle priors, and em- ploys an exponential moving average (EMA) based stochastic optimization algorithm that achieves provable convergence rate under standard assumptions.
Self-PU (Chen et al. 2020b) integrates self-paced learning, self-calibrated loss reweighting, and teacher-student self-distillation for positive-unlabeled classification. The self-paced component pro- gressively mines high- high-confidence unlabeled examples into a dynamically expanding trusted set, em- ploying an in-and-out mechanism iteratively update soft pseudo-labels for both positive and nega- tive classes with balanced sampling rates. For remaining unconfident unlabeled data, a meta-learning based self-calibrated instance-wwar loss adaptively reweights the combination of cross-entropy with soft labels and non-negative PU risk, constrained by a balancing factor to regulate supervision qual- ity. The framework further enforces consistency regularization through collaborative distillation between two student networks trained with asynchronous learning paces (different sampling ratios), and between each student and its corresponding teacher network maintained via exponential moving average of weights.
P3Mix (Li et al. 2022) addresses decision boundary deviation in positive-unlabeeed learning through heuristic mixup augmentation of a disambiguation-fiee objective. Treating unlabeled in- stances as pseudo-negative it identifies marginal pseudo-negative instances (those exhibiting am- biguous predictive scores within thresholded range) and selects their mixup partners exclusively from a candidate pool of positive instances with high prediction entropy (near-boundary positives), while other instances select partners randomly from the full training set. This generates augmented instances with partially positive soft labels that push the decision boundary toward the fully su- pervised position. The method proposes two robust variants: P3Mix-I incorporates early-learning regularization using mean-teacher estimated auxiliary targets to prevent memorization of imprecise supervision, while P3Mix-C performs pseudo-negative correction by reassigning high-confidence positive predictions to the positive class prior to mixup.
Robust-PU (Zhu et al., 2023) initializes the model via nnPU pre-training and executes an itera- tive three-stage training strategy. In the hardness measurement stage, it computes classification losses for all samples using temperature-scaled logistic or sigmoid loss, treating positive samples with positive ground truth and unlabeled samples negatives. The sample weighting stage maps these hardness values to sample weights via the minimizer function of SPL-IR- tion, where the mapping threshold is dynamically controlled by training scheduler implementing linear, convex, concave, or exponential pacing functions to gradually relax selection criteria over iterations. The weighted supervised training stage treats unlabeled data as negative and performs binary cross-entropy optimization with the computed sample weights for multiple epochs. This it- crative process progressively incorporates harder samples prevent noise accumulation from early misclassification.
regulariza-
Holistic-PU (Xinrui et al. 2023) employs balanced resampling of positive and unlabeled data into equally-sized batches, initially treating unlabeled samples as negatives during training. It records the prediction score trajectory for each unlabeled sample across training iterations. trend score is computed for each sample using a robust mean estimator applied to ordered differences between prediction scores at different time steps, capturing the tendency of negative samples to exhibit con- sistently decreasing scores while positive samples display chaotic initially increasing patterns. The unlabeled data is then partitioned into pseudo-positive and pseudo-negative sets using Fisher's Natural Break algorithm based on the trend score distribution. Finally, supervised classifier is trained on the resulting pseudo-labeled dataset.
PUL-CPBF (Li et al. 2024) establishes that the probability boundary of the asymmetric disambiguation-free risk is governed by the asymmetric penalty. The method first trains a set of weak classifiers with diverse probability boundaries by minimizing asymmetric disambiguation free empirical risks under specific penalty values, forming a probability boundary fence. For each unlabeled instance, it locates the class posterior probability within the fence-defined range and gen- erates a stochastic label via uniform sampling. then trains a strong classifier via self self-training with consistency regularization.
18

---

Published as a conference paper at ICLR 2026
LaGAM-PU (Long al. 2024) employs hierarchical contrastive learning extract latent group semantics through unsupervised clustering alignment, dichotomized cut-off based on binary predic- tions, and local neighbor smoothing. It then performs meta-learning-bese based label disambiguation via bi-level optimization: an inner loop virtually trains the model with current pseudo-labels, and outer loop refines pseudo-labels using projected gradients with exponential moving average minimize evaluation loss on a support set, alternating between label updates and classifier training,
VAE-PU (Na et al. 2020) proposes generative approach to positive-unlabeled learning that elim- inates the Selected Completely At Random assumption. The method employs a variational au- toencoder architecture featuring two distinct latent representations: one capturing label informa- tion and another encoding observation indicators. Virtual positive-unlabeled instances are synthe- sized by combining the label-related latent factors derived from labeled positive examples with the observation-related factors extracted from unlabeled data. The optimization objective integrates the evidence lower bound with an adversarial alignment loss that matches generated samples to unlabeled data distributions and label consistency loss that ensures generated instances exhibit positive-class characteristics. Training alternates between updating the generative network to pro- duce informative pseudo-example and optimizing the classifier using these synthesized instances.
PAN (Hu et al. 2021) introduces a predictive adversarial framework where a classifier assumes the role of generator identify likely positive instances from unlabeled data against a discriminator. objective comprising three components: a classification
The approach optimizes a divergence-bant term training the discriminator to recognize known positive examples, alignment term encourag- ing prediction consistency between the classifier and discriminator on unlabeled data, and a sym- metry correction term ensuring balanced gradient updates for positive and negative instances. This formulation enables training without requiring paassp knowledge as input. The system alternates between discriminator updates using standard classification losses and classifier updates employing a
mechanism based on discriminator feedback.
CGenPU (Papić et al.) 2023) proposes a conditional GAN-based framework for positive-unlabeled learning that employs generator, discriminator, and auxiliary classifier in single-stage three- player minimax game. The training objective combines adversarial loss with an auxiliary loss that minimizes distribution divergence between labeled and generated positíves while maximizing sepa- ration between generated positive and negative samples. An anchoring term stabilizes early training by enforcing high confidence on labeled positives. This approach simultaneously learns both class distributions without requiring prior knowledge class priors or labeled negative examples, ad- dressing the instability and architectural complexity of prior multi-generator methods.
19

---

Published as a conference paper at ICLR 2026
C DATA GENERATION SETTINGS
C.1 SETTINGS IN EXISTING PU WORK
Table provides comprehensive summary of the data generation protocols employed in recent PU learning literature. For each method, we categorize the experimental setup based on three critical factors. First, we identify the data sampling scenario, distinguishing between the case-control (cc) and single-trainingsset (ss) paradigms. Second, we note whether the authors demonstrated robust- ness by systematically varying the label frequency-the proportion of true positives that are labeled (indicated by vary c). Finally, specify the core labeling mechanism (e.g., SCAR, SAR) assumed in their experiments. For studies where these configurations were not explicitly stated, we inferred the settings by meticulously analyzing the described methodology and, where available, the associ- ated source code to ensure an accurate and consistent comparison.
Table C.I: Data generation settings the existing PU learning methods. ec denotes case-control and denotes single-training-set "Varying c" indicates that the corresponding work demonstrates robustness by using different values of c.
Method
Datasets
ss/ce
Varying
SCAR/SAR
nnPU 1S
CIFAR-10 10, MNIST, 20News, Epsilon
cc
No
SCAR
PUSB
CIFAR-10, MNIST, Mushrooms, Spambase, Shuttle, Page Blocks, USPS, Connect-4, Swis- sProt
Yes
SAR
VPU
CIFAR-1O, F-MNIST, STL-10, Page Blocks, Grid Stability, Avila
Yes
SCAR
MPE-PU
CIFAR-10, MNIST, IMDb
Not Given
No
Not Given
LBE-PU
USPS, UCI: Australian, Madelon, Phishing. Vote, Banknote, Breast, HockeyFight, SwissProt
Yes
SAR
PUET 30
CIFAR-10, MNIST, IMDb
No
SCAR
336
CIFAR-10, F-MNIST, ADNI
Yes
SCAR
PULDA
CIFAR-10, 10, F-MNIST, ADNI, Tiny-ImageNet
cc
SCAR
Pe
CIFAR-10, MNIST, ADNI
No
SCAR
P3Mix
CIFAR-10, F-MNIST, STL-10, Credit Fraud
cc
SCAR
Robust-PU 39
10FAR-10, MNIST, F-MNIST, STI.-10, ADNI,
cc
No
SCAR
UCI: Mushrooms, Shuttle, Spambuse
Holistic-PU 3
CIFAR-10. F-MNIST. STL-10, ADNI, Credit
Yes
SCAR
Card Fraud
LaGAM-PU 21
CIFAR-10. STL-10, CIFAR-100, ADNI
No
SCAR
PUL-CPBF 20
CIFAR-10, F-MNIST, STL-10, ADNI
cc
No
SCAR
VAE-PU 2X
CIFAR-10 MNIST, 20News
Yes
SAR
PA
CIFAR-10, 10, MNIST, IMDb, 20News
SCAR
CGenPU 27
CIFAR- 10, MNIST
cc
Yes
SCAR
20

---

Published as a conference paper at ICLR 2026
C.2 TEELEE--RAINNGG-SS AND AASE-CONTROL
Single-training-set. This scenario preserves the original training set's size and composition. The number of positives to be labeled is N₂ |c(Pl] []]] These are drawn without replacement from P to form LP. The remaining samples, (P) LP) U., constitute the unlabeled set 24
Case-control. This scenario first selects LP C P according to the chosen labeling strategy (Ap- pendix C.3). The unlabeled set is then defined as the entire training pool, U P U U,W i.e., LP is returned U. Consequently, LP U and [14] N, and the class mixture in 2 is identical to that of the full population. When enumerating indices, include all negatives and many positives as needed to reach [P), prioritizing positives not in CP and reusing LP if necessary.
N
P
N
I.P
UP
UN
I.P UP
UN
Single-training-set
Case-control
Figure 7: Comparison between the single-training-set set and case-control. In both cases, negatives (Y 0) are never labeled (S = 1), so all negatives in the training pool are assigned to the unlabeled set the single-trainingsset case, the labeled positives LP are sampled without replacement from P and removed from U, yielding (P LP) U.W, whereas in the case-control scenario, CP sampled from P but then returned so that P U N follows the class mixture of the overall training data distribution.
Class coniiioandi densilies
Unlabeled mixtures
(.0e)
1.0
Casecontrol
1.00
0.8
Single-tiriing
0.6
0.50
0.4
0.2
0.00
0.0
-1.0
-0.5
0.0
0.5
1.0
2.0
-1.0
+0.5
0.0
0.5
1.0
1.5
2,0
Figure 8: One-dimensional Gaussian toy example illustrating how single-training-se and case- control affect only the unlabeled density. The class-conditional densities f+ (x) and (x) are fixed Gaussians with distinct means (left), while the unlabeled density fu changes its effective class mixture under single-training-set versus case-control (right). Under case-control, fa matches the overall training population mixture, whereas under single-training-set removing fraction of posi- tives into CP reduces the positive mass in 24 and shifts fa(r) closer to fz)
21

---

Published as a conference paper at ICLR 2026
C.3 LABELING STRATEGIES
The details of the four labeling strategies evaluated under our framework are:
S1 (SCAR). Select Ny instances uniformly without replacement from P to form CP, This corre- sponds to a constant labeling propensity e(x: = c. for all x E P.
S2 (LBE Strategy 1). Using the auxiliary posterior p(xx) (cf. Table A.1 for and w), define instance-dependent propensities
(x) [i(x)]] k>
(x) Normalize (x) Ep ei(z") and select LP by weighted sampling without replacement of
size N₂ using probabilities mi (ar), which favors high-posterior positives; in experiments
= 10 (Gong et al. 2021).
S3 (LBE Strategy 2).
Using the same p(z). define
(2(x) = [] - --)((=)] x> 1.
Normalize n2(21) Erep o2(#7) and select LP by weighted sampling without replacement of size Nt using probabilities (I). This scheme favors boundary otherwise ambiguous positives; in our experiments we also k 10 (Gong et al., 2021).
(x)
S4 (PUSB Posterior Sharpening). Using the same p(z), construct sharpened scores
sa (x) [xx]]] a a>
Rank x E P in descending order of sa (2) and deterministically take the top-Ne as LP. This rule concentrates labeling on the highest-posterine positives; in our experiments we use = 20 (Kato
ct al. 2019).
C.4
EVALUATION METRICS FOR PU LEARNING METHODS
For fair and comprehensive evaluation, we compile the metrics reported in the original papers of each algorithm in Tab. C.2 We adopt AUC, F1, Accuracy, Precision, and Recall as our primary measures. Together, these metrics offer complementary perspectives on performance across diverse experimental conditions. Below, we briefly introduce each metric used to assess PU-learning meth- ods.
AUC. The Area Under the ROC Curve (AUC) (Bradley) 1997) assesses a classifier's ability to rank positive instances above negative ones, independently of any fixed threshold. It can be viewed as the probability that a randomly selected positive instance receives higher score than randomly selected negative instance. AUC is widely regarded robust, threshold-free metric for binary classification.
The F1-score (Powers 2011 is the harmonic mean of Precision and Recall, combining them into a single measure. defined as
F1-score.
2 Precision Recall
Precision + Recall
where Precision TF+FP and Recall TP+FN. The Fl-score is particularly useful imbal-
TP
TP
anced datasets, it penalizes extreme disparities between Precision and Recall, yielding a more balanced assessment of classification performance.
Accuracy. Accuracy is the proportion of correctly classified instances among all samples:
TP TN
Accuracy TP - FF
where TP, TN, FP, and FN denote true positives, true negatives, false positives, and false neg- atives, respectively. Although accuracy is simple and intuitive, it can be misleading under severe class imbalance, where it is dominated by the majority class.
22

---

Published as a conference paper at ICLR 2026
Precision. Precision (Powers 2011 measures the proportion of true positives among all predicted positives:
TP
Precision = TP + FP
where TP and FP denote true positives and false positives, respectively. High precision indicates few false positive errors.
Recall. Recall (Powers 2011 quantifies the proportion of true positives among actual positives:
TP
Recall
TP + FN
where FN represents false negatives. High recall indicates that the model captures most of the positive samples.
Table C.2: Evaluation metrics reported their original papers (V reported, - = not reported).
Method
AUC
F1
Accuracy
Precision
Recall
IMPU
PUSB
VPU
LBE-PU
MPE-PU
PUET
Dist-PU 36
PULDA 13
Self-PU[5]
P3Mix
Robust- PUstt-UU0 39
Holistic-PU
LaGAM-PU 21
PUL-CPBF 20
VAE-PU 25
PAN
CGenPU 27
D EXPERIMENTAL SETTINGS AND RESULTS
D.1 EXPERIMENTAL SETTINGS
Neural backbone architectures. IMDb and 20News handled by the MLP backbone sum- marised Table D.2 Each document is first converted off-line into 384-dimensional dense vector using the TranencceTTanssrmmerss all-MiniLLM-L6- model!] The vectors stored in compressed NumPy format and loaded at training time, removing any text-processing overhead while guarantee- ing identical representations for all PU learners. MNIST and F-MNIST share the LeNet architecture detailed in Table [D.3] The network receives single-channel 28x 28 images and produces one logit; when method explicitly requires two outputs (e.g. HolisticPU variants) only the final linear layer duplicated. CIFAR- 10 uses the Custom CNN whose layer-wise definition is given in Table D.4 De- signed for 32x32 three-channel inputs, the same "replace-lastlayee rule applies for methods that need 2-way logits. ADNI uses CNN tailored for structural MRI, detailed in Table D.5) it receives
single-channel 128 x 128 images and outputs one logit, and for methods requiring two outputs only the final linear layer is replaced. Spambase and Connect-4 are processed by the same MLP backbone shown in Table D.2 Spambase feeds the network with raw numeric features, whereas Connect-4 first one-hot-encodes the board state into 126 binary features.
-inn I. 1.6
23

---

Published as a conference paper at ICLR 2026
Table D.I: PN Train reports the standard supervised training split as (Np NT). PU Train reports the PU training composition as (Nr, ,,,, where Ni denotes the number of labeled positives, N the number of positive instances present the unlabeled pool, and N₂ the number of negatives
in the unlabeled pool. Test uses (NT,, NT Validation shows the held-out fraction with its size No-
Dataset
PN Train (NT)
PU Train (N, N,..N. NT))
Validation
Test ,
Total Size
20Ncws
(6,326, 4,874)
(632, 6,326, 4,874)
0.01 (N, 114)
(4,254,
18,846
IMDb
(12,375, 12,375)
(1,237, 12,375, 12,375)
0.01 (N,
(12,500, 12,500)
50,000
MNIST
(29,197, 30,203)
(2,919, 29,197, 30,203)
0.01 (N, 600)
(4,926, 5,074)
70,000
F-MNIST
290, 29,700)
(2,970, 29,700, 29,700)
0.01 (N, 600)
(5,000, 5,000)
70,000
CIFAR-10
(19,800, 29,700)
(1,980, 19,800, 29,700)
0.01 (N, 500)
(4,000, 6,000)
60,000
ADNI
(2,352, 2,516)
(255, 2,552, 2,516)
0.01 (N, 52)
(622, 658)
6,400
Connect-4
(35,222, 18,282)
(3,622, 35,222, 18,282)
0.01 (N, 541)
(8,895, 4,617)
67,557
Spambase
(1,435, 2,208)
(143, 1,435, 2,208)
0.01 (N, = 37)
(363, 558)
4,601
Table D.2: Layer-by-layer specification of the MLP backbone (IMDb / 20News / Spambase / Connect-4). The input feature dimension is E {384, 384, 57, 126} depending on the dataset. Note: in Dropout(p) denotes the dropout probability.
Layer type
Output shape
0 Input
d
Linear (d - 512)
512
2 ReLU
512
Dropout (p 0.3)
512
Linear (512 + 256)
256
ReLU
256
Dropout (p 0.3)
256
Linear (256 + 128)
128
ReLU
128
Dropout (p = 0.2)
128
10 Linear (128 + 64)
64
11 ReLU
64
12 Linear (64 + 1)
Table D.3: Layer-by-layer specification of the LeNet backbone (MNIST / F-MNIST), Note: Max- Pool2d uses x kernel with stride Conv2d kernel sizes, strides, and padding are specified the rightmost column.
#
Layer type
Output shape
Kernel stride / pad
Input
(1, 28, 28)
Conv2d (1 -> 10)
(10, 24, 24)
5 x 5, stride 1, pad 0
2
MaxPool2d
(10, 12, 12)
2 2, stride
3
ReLUU
(10, 12)
Conv2d (10 - 20)
8,0,8.8)
5 5, stride 1, pad 0
MaxPool2d
(20,4.4)
2 2, stride
ReLU
(20,4,4)
Flatten
(320.)
Linear (320 + 50)
(50,)
ReLU
(50.)
10
Linear (50 + 1)
(1,)
24

---

Published as a conference paper at ICLR 2026
Table D.4: Layer-by-layer specification of the CIFAR-10 Custom CNN backbone. Note: p in
Dropout(p) denotes the dropout probability: BatchNorm2d applies batch normalization; Conv2d parameters (kemee/ssrride/pad) are detailed in the rightmost column.
# Layer type
Output shape
Kernel / stride pad
0 Input
(3, 32, 32)
1
Conv2d 96)
(96, 32, 32)
3 3. stride pad
2
BatchNorm2d (96)
(96, 32, 32)
3
ReLU
(96, 32)
Conv2d (96 96)
(96, 32, 32)
stride pad
5
BatchNorm2d (96)
(96, 32, 32)
6
ReLU
(96, 32, 32)
Conv2d (96 - 96)
(96, 16, 16)
3, stride pad
8
BatchNorm2d (96)
(96, 16, 16)
9
ReLU
(96, 16, 16)
10 Dropout (p 0.2)
16,16,10 16, 16)
11
Conv2d (96 + 192)
(192, 16, 16)
3, stride pad
12
BatchNorm2d (192)
16,
13
ReLU
(192, 16)
Conv2d (192 > 192)
(192, 16)
3 x 3, stride 1, pad I
BatchNorm2d (192)
16,
16
ReLU
16,
17 Conv2d (192 -> 192)
(192,8,8)
3 3, stride 2, pad
18
BatchNorm2d (192)
(192,8.8)
ReLU
8192,8,8)
20
Dropout (p 0.5)
(192, 8,8)
21
Flatten
(12288,)
22
Linear (12288 1000)
(1000,)
23
ReLU
(1000,)
24
Dropout (p 0.5)
(1000,)
25
Linear (1000 + 1000)
(1000,)
26
ReLU
(1000,) )
27
Linear (1000 +
(1,)
25

---

Published as a conference paper at ICLR 2026
Table D.5: Layer-by-layer specification of the ADNI (Alzheimer MRI) CNN backbone. Note: in MaxPool2d(() denotes the kernel size x k); p in Dropout(p) denotes the dropout probability: AdaptiveAvgPool20(1. 1) performs global average pooling.
# Layer type
Output shape
Kernel homel/sttie /stride/ pad
0 Input
(1, 128)
Conv2d (1 + 32)
(32, 128, 128)
x 3, stride 1, pad
2
BatchNorm2d (32)
(32, 128, 128)
3
ReLU
(32, 128, 128)
Conv2d (32 - 32)
(32, 128, 128)
x 3. stride 1, pad
5
BatchNorm2d (32)
(32, 128, 128)
6
ReLU
(32, 128, 128)
7
MaxPool2d
2)
(32,64, 64)
2, stride 2, pad
8
Dropout (p 0.1)
64)
9
Conv2d (32 - 64)
(64,64, 64)
x 3, stride 1, pad
10
BatchNorm2d (64)
(64,64, 64)
11
ReI.U
644,64,64)
Conv2d (64 - 64)
(64, 64, 64)
3. stride 1, pad
13
BatchNorm2d (64)
(64,64,
14
ReLU
(64.64,64)
15
MaxPool2d 2)
(64,32,32)
2, stride 2, pad
16
Dropout (p 0.1)
(64.32,32)
17
Conv2d (64 + 128)
(128,32,32)
x 3, stride 1, pad
18
BatchNorm2d (128)
(2,8,32,32)
19
ReLU
(128, 32,33)
20
Conv2d (128 -> 128)
(128,32,32)
x 3, stride 1, pad
21
BatchNorm2d (128)
(128,32,38
22
ReI.U
(1,
23
MaxPool2d 2)
(128, 16, 16)
2, stride 2, pad
24
Dropout (p 0.2)
(128, 16, 16)
25
Conv2d (128 + 256)
(256, 16, 16)
x 3, stride 1, pad
26
BatchNorm2d (256)
(256, 16, 16)
27
ReLU
(256, 16, 16)
28
Conv2d (256 256)
(256, 16, 16)
x 3, stride 1, pad
29
BatchNorm2d (256)
(256, 16, 16)
30
ReLU
(256, 16, 16)
31
MaxPool2d - 2)
(256,8.8)
x 2, stride 2, pad
32
Dropout (p 0.3)
(256, 8,8)
33
AdaptiveAvgPool2d (1,
(256, 1,
34
Flatten
(256.) )
35
Linear (256- - 641
(61,)
36
ReLU
(64.)
37
Dropout 0.3)
(64.)
38
Linear (64 + 1)
(1,)
26

---

Published as a conference paper at ICLR 2026
D.2
PERFORMANCE UNDER THE CONVENTIONAL SETTING
D.2.1 FULL EXPERIMENTAL RESULTS
This subsection complements the main conventional results in Table 1 For each dataset, we use the to-sset-t--bbbboeee mapping Appendix AppendixDD.Jandd and the PU data specification Table [.I] The training set size N and class prior are fixed per dataset. PU training splits are generated with the same conventional configuration used for the main table. All learners are trained under the unified pipeline with identical optimizer and schedules defined in the YAML configurations, and only the classification head is adapted when a method requires two logits. We select the checkpoint with the best validation macro-Fl and report Accuracy, Precision, Recall, macro-F1, and AUC the held-out test set.
Risk-Minimization Estimators
Disambiguation-Guided ERM
Generative Distribution Matching
Recall
Recall
Recall
1.00
.00
6.94
20News
0.88
Accuracy
Accuracy
Accuracy
Precision
Precision
Precision
AUC
Recall
Recall
Recall
0.90
0.50
0.90
IMDb
F1
F1
Accuracy
Accuracy
Accuracy
Precision
Precision
Precision
VPU Dist-PU
P3MIXE
PSMIX-C PUL:CP8F
MPE-PU PULDA
LBE-PU
PUET
Robust-PU
Holistic-PU
LaGAM-PU
Figure 9: Performance of PU methods on text datasets across five evaluation metrics.
27

---

Published as a conference paper at ICLR 2026
Mismiiionn Estimators
Disammiiquttion-Giided ERM
Generative Distribution Matching
Recall
Recall
Recall
0.97
0.96
755
MNIST
0,50
0.90
F1
0.85
F1
4.00
Accuracy
Accuracy
Accuracy
Precision
Precision
Precision
AUC
AUC
AUC
Recall
Recall
Recall
1.00
1.00
.00
86
F-MNIST
F1
F1
0.45
Accuracy
Accuracy
Accuracy
Precision
Precision
Precision
AUC
AUC
Recall
Recall
CIFAR-10
85
0.5
0.66
0.70
0.35
0.35
Accuracy
Accuracy
Accuracy
Precision
Precision
Precision
Ph
Sait-FU
PaMixEE
PSMI-C PUL-PPF
CGenRi
MPE-PU
LOE
PUET
DUst-PU
Robust-FU
Holistic-FU
LaGAM-PU
PULDA
Figure 10: Performance of PU methods on vision datasets across five evaluation metrics.
28

---

Published as a conference paper at ICLR 2026
Risk-Minimization Estimators
Disambiguation-Guieed ERM
Generative Distribution Matching
AUC
AUC
AUC
Spambase
0.89
0.84
0.85
0.55
F1
0.35
F1
0.40
Accuracy
Accuracy
Accuracy
Precision
Precision
AUC
AUC
AUC
Recall
Recall
Recall
Connect-4
0.72 0.86
0.02
0.65
Accuracy
Accuracy
Accuracy
Precision
PN MP
nePU LBL-PU
PUSB
VPU
PN Robust-PU
Self PU
PIMIXKE ... P3MiX-C
PN
VAE
PAN ... CGenPU
PULT
Dust-PU
Pu
LaGAM-PU PU PUL-CPU
PULDA
Figure 11: Performance of PU methods on tabular datasets across five evaluation metrics.
29

---

Published as a conference paper at ICLR 2026
Table D.6: Holm-Bonferroni corrected p-values from two-sided paired (-tests the test accuracies in Table I under the conventional setting (cc/SCAR/c=0.J) Each entry compares PU method
against nnPU on the same dataset. Values below 5e-02 are shown in bold (significantly better than
nnPU) or underlined (significantly worse).
Method
IMDb
20News
MNIST
F-NNIST
CIFAR-10
ADNI
Connect-4
Spambase
nnPU 15
PUSB [4
8,2c-02
1.4c-01
9.5c-02
1.1c-01
4.3e-02
1.5-011
1.6c-01
1.1c-01
VPU 4
4.8c-02
5.8c-02
8,4c-05
8.7e-02
8.9c-02
8.5e-02
4,3c-02
9.2c-02
MPE-PU 8
1.2c-01
1.6c-01
1.6c-01
2.0c-01
2.7e-01
2.3e-01
2.9c-01
2.1e-01
LBE-PU
8.9c-02
1.5c-01
3.2e-03
2.1c-03
9.1c-02
6.9c-02
9.4c-02
8.3c-02
PUET 30
9.5c-02
6.2c-02
9.8c-02
9.3e-02
6,4c-02
3.7e-02
8.1c-02
7.3c-02
Dist-PU 36
3.9e-02
6.7e-02
8.7e-02
1.1e-01
8.9e-04
6.2e-03
7.6e-02
3.7e-02
PULDA 13
1.8c-01
1.0c+00
8.1c-01
1.0+000
1.3e-04
1.0c+00
5.9e-05
7.4c-02
Self-PU [
4.1c-02
1.8c-01
6.6e-02
2.0c-01
1,9c-01
19c-01
2.3e-01
3.8c-02
P3Mix-E
3.2c-02
8.9e-02
9.3c-02
8.5e-02
6.7e-02
7.6e-02
9.4c-02
8.2e-02
P3Mix-C
19
1.3e-02
5.5e-02
8.6e-02
9.7c-02
7.1e-02
2.0e-02
6.8e-02
7.3e-02
Robust-PU
33
1.3c-01
8.9c-02
1.1e-01
1.5e-01
1.3c-01
14c-01
1.8c-01
1.6c-01
Holistie-PU
32
9.0c-02
4.2e-02
7.5e-02
4.3e-02
3,4e-02
1.0e-01
5.9c-02
1.1e-01
LaGAM-PU
21
6.7e-02
5.8e-02
8.5e-02
8.9e-02
8.3e-02
5. 5.9e-02
7.4e-05
5.3e-05
PUL-CPBF 20
7.5c-02
8.1c-02
3.2e-02
3.3e-02
9.5c-02
7.1c-02
4.1e-05
3.8e-02
VAE-PU 25
3.6c-02
8.3e-01
4.8c-02
2.9c-01
2.7e-02
4.5c-02
3.9c-01
3.0c-01
PAN 11
5.6c-02
2.8e-02
6.8c-02
9.1c-02
8.7c-02
1.1c-01
1.2c-01
8.9c-02
CGenPU
27
9.1c-02
1.4c-01
9.5c-02
8.0c-02
1.0c-01
1.0c-01
7.8c-02
8.4c-02
D.2.2 STATISTICAL SIGNIFICANCE OF PERFORMANCE DIFFERENCES
For each dataset and PU method, we collect the test accuracy with 10 random seeds (2, 25, 42, 52, 99, 103, 250, 666, 777. 2026) under the conventional cc/SCAR/c=0.1 configuration. We then perform two-sided paired t-tests comparing each method against nnPU on the same dataset.
Paired (-test. For each pair of methods (e.g., method A vs. nnPU) given dataset, let {x; ) and {y: denote the test accuracies from n = 10 independent runs, where H; and y/8 are paired (same random seed). We compute the paired differences d, I, V/s their sample mean d n-T (d; d)2. The test statistic is
> di, and sample standard deviation
d
sa/ Vn
which follows I-distribution with n degrees of freedom under the null hypothesis of dif- ference. The raw two-sided p-value is given by Praw P(TTx-i 2 [t]), where Tw-1 denotes a t-distributed random variable with degrees of freedom.
Holm-Bonferroni correction. To control for multiple comparisons across hypotheses (one per method vs. nnPU) on the same dataset, we apply the Holm Bonffrrooi procedure. Let S P(2) S P(m) denote the sorted raw p-values. For each hypothesis in sorted order, the corrected p- value is
put = min max k=l....f
[mm - k + 1): p(s)]] }
We reject hypothesis j if Peor & a (we use a = 0.05 all experiments).
Table D.6 shows that, under the conventional setting, several
Estimation meth- ods and Disambiguationn Guidde Supervised ERM methods achieve statistically significant accuracy gains nnPU non-trivial subset of datasets, whereas self-trainin and holistic variants more often suffer significant degradation. Generative distribution matching approaches rarely yield sig- nificant benefits and even be significantly worse than nnPU, highlighting their instability in this regime. Overall, these significance tests confirm that the performance gaps observed in Table I are not driven by random seed noise.
30

---

Published as a conference paper at ICLR 2026
D.2.3 FAILURE CASES ANALYSIS
LBE-PU (Gong )021) exhibits competitive performance on vision datasets under conventional settings, and on simpler images it can match or surpass strong PN baselines, This effectiveness stems from its labeling-bia model e(x) capturing meaningful variation in positive labeling mechanisms. On ADNI under the conventional setting, however, only small fraction of positives are labeled (ICP) 255) while most remain unlabeled ([ = 2552 vs. [W] 2516). As shown in Fig. 12 (top panels), this extreme imbalance breaks LBE-PU in two ways. First, the classifier ho (ar) concen- trates labeled positives near 1.0 but assigns similarly high scores to many negatives, causing severe overlap between p and N in both training (panel a. and test distributions (panel a.2). Second, the estimated propensities e(x) for labeled versus unlabeled positives exhibit substantial IQR overlap (panel a.3), indicating that the learned e(xr) deviates from the SCAR assumption and fails to model the true labeling mechanism. Together, these failures prevent LBE-PU from recovering a reliable decision boundary ADNI.
While PAN (Hu et al. 2021) demonstrates effective performance on datasets such as MNIST and IMDb-where its adversarial training framework successfully promotes separation between positive instances and challenging unlabeled samples- exhibits distinct failure pattern on Spambase. As illustrated in Fig. 12 (bottom panels), this complementary failure mode stems from the Discrim- inator's role in defnning reliability weights. Specifically, Panel b.3 reveals that the Discriminator systematically assigns higher scores true negatives than both labeled and unlabeled positives (a(D( > (D(()) cp, (D()))).) Consequently, the reliability weights rl o (((x)) be- come concentrated on unlabeled examples that resemble negatives. This skew causes the Recognizer to learn a decision function (a()) that positions nearly all positive instances alongside a substantial portion of negatives above the default threshold of 0.5 (panels b. b.1.b.2), resulting in exceptionally evaluation.
high recall but merely moderate precision under
(a. 1) Train ha(x) LP, 2/. N
(a.2) Test ha(x) for Y= vs. Y=
(a.3) Estimated e(x) on CP Up
CP
Y-1
CP
Donsiy
10
0
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
holx)
ha(x)
e(x)
(b.1) Train ha0d) on EP, 24. W
(b.2) Test ha(x) for Y vs. Y= 0
(b.3) a(D()) on CP, V.. NN'
Density
5.0
1/p
Y
1/
X
2
2.5
0.0
0.50
1.00
0.50
1.00
0.50 o(D(x))
1.00
ha(x)
hy(x)
Figure 12: Failure analysis of LBE-PU on ADNI (top) and PAN on Spambase (bottom) under con- ventional settings. Top panels: 1a a.2) Training and test score distributions (zx for labeled positives LP, unlabeled positives Up, and negatives N; shaded bands indicate interquartile range (IQR) overlap between Up and M. (a.3) Estimated labeling propensity e(x) for LP and Up with IQR overlap highlighted. Bottom panels: (b. b.2) Training and test score distributions (2 with threshold at 0.5. (b.3) Discriminator scores o (I())) on CP, Up, and N.
31

---

Published as a conference paper at ICLR 2026
D.3 EFFECTIVENESS OF PU METHODS W.R.T. LABEL RATIO
We study label efficiency by varying the label ratio while keeping the training size N and class prior unchanged. Following the main text (Fig. 5). sweep from 0.01 0.09 with step of 0.02, and from 0.1 to 0.9 with step of 0.1.
mnn
PUSB
Dist-PU
P3Mix-E
P3Mix-C
Robust-PU
Hollstic-PU
LuGAM-PU
PUL-CP8F
CGen
Risk-Miiniiization Estimators
Dlaambiquatlo--Gudeedd ERM
Generative Distribution Matching
0.8
Accuracy
IMDb
0.5
0.01
0.05
0.090.1
0.5
0.9
0.01
0.05
0.09
0.1
0.5
0.01
0.05
0.09
0.5
MNIST
Accurayy
0.8
0.6
0.01
0.05
0.090.1
0.5
0.9
0.01
0.06
0.00 0.1
0.5
0.9
0.01
0.05
0.00 0.1
0.5
0.9
1.0
1.0
F-MNIST
Accuracy
0.9
0.8
0.8
0.6
0.8
0.09
Accuracy
0.7
ADNI
0.4
0.4
0.01
0.05
0.090.1
0.5
0.9
0.01
0.05
0.09
0.5
0.01
0.05
0.09
0.5
Spambase
Accuracy
0.8
0.8
0.8
0.01
0.05
0.080.1
0.5
0.01
0.66
0.08 0.1
0.5
0.01
0.05
0.00
0.5
0.9
Label
Label
Figure 13: Effectiveness PU methods with respect to different label ratios c.
32

---

Published as a conference paper at ICLR 2026
D.4 ROBUSTNESS TO SELECTION BIAS
We evaluate robustness under different labeling assumptions by varying the propensity of observing positive labels. The experiments follow the labeling strategies defined Appendix C.3 (S1, S2, S3, S4). For each strategy, we keep the training size N and class prior fixed. We report Fl on the test set at two label ratios, c = 0.05 and 0.5. Bars correspond the low-label regime (c 0.05) and lines correspond to the high-label regime (c 0.5). All other training settings remain the same as in the conventional configuration.
(c-00 (c-0.0511%)
(0-5.5))
Spumbose
Dist.PU
man
PUSS
VPU
RET
PILDA
LBF PN
RUl-CPPr
FMMi
PaMinc
RobissetPP Pg
LaGit PM
St
Helletic
CGenai
PA4
MRE PN
100
Racheen nake Gramato
Dratdipatce- Soperieed Fay
Gerenove Obatadion
160
F1 Scure (c=0.05) [M)
80
Score (C-0.5))))
60
40
10
20
PRSS
PIET
MU
Diat
RORU
LBE-PU
VRU
PULDA
PaMtice
Self
Holsmc
PaMix
Robust Rd
Laciss Pd
PAN
Counii
IREI
Figure 14: scores PU methods under different labeling assumptions on tabular datasets. Bars denote low-label regime (c = 0.05), lines denote high-label regime (c 0.5).
100
Supariied past
Deieetiv OwNEEaie Macting 100
80
15
20heeas
Score
RULDA
VPU
MIS-PU
Dist PJ
PUSSB
PUET
LBE-UU
Rebesn
PUt-CPBF
Hnliste-PU
PaMile
LAGAA-PU
PAN
CGMMPU
oitn
Dranoipation
Score (c-0.55)0))
FL (c-O.5)
INDh
PULDA
Otan-u
RUET
LiF pu
Robust Pi
P
Pasele g
LeGAM-N ru
Hointee-PU
Sal-Pu
PRW
URe au
Figure 15: F1 scores of PU methods under different labeling assumptions on text datasets. Bars denote low-label regime (c = 0.05), lines denote high-label regime (c 0.5).
33

---

Published as a conference paper at ICLR 2026
100
100
HMST (6)
F2 (N)
Fi Score (c-0.05)
40
20
LIE-VE
HPE
PUET
PULDA
you
Lasadd pM
Pul-cPar
Pailee
Rudont P1
Selt
PIM
PAHe
100
No MER SIE Ierrar
Braretp www.eettteeeee...ooooooo and Spperied Fag
(c-0 05) 153
180
FAWUST
Fl (c-
60
40
PULOA
MFEE-UU
RUFT
Obse yPu
PieSa
menpy
Lif
vpu
Lacast Pi
Robist m1!
Sitt
Halentoo
PAW
CGemP
URE Pu
Fl Score (c-0.55- (N)
(C-0.5)
AONI
60
40
RULDA
PusB
LBE
mop
FiMba.c
SeirP
Patele
Haistte PM
LUGAM
Robastt
Co
WREAU
Figure 16: Fl scores of PU methods under different labeling assumptions on vision datasets. Bars denote low-label regime (c 0.05), lines denote high-label regime 0.5).
E COMPUTATION RESOURCES
All experiments were conducted using PyTorch 2.8.0 and Python 3.12 running on Ubuntu 22.04, with CUDA 12.8 for GPU acceleration. The hardware consisted of single NVIDIA RTX 5090 GPU (32 GB memory), 25 vCPU Intel Xeon Platinum 8470Q processor, and 90 GB of system RAM.
F PACKAGE, DOCUMENTATION, AND MAINTENANCE
This section outlines packaging, documentation, licensing, and long-term maintenance policies to support reliable use and extension of the benchmark.
Data Acquisition. Datasets obtained directly from authoritative sources through established APIs: vision (MNIST, F-MNIST, CIFAR-10) via torchvision; text (20News, IMDb) via scikit- learn and Hugging Face; tabular (Connect-4, Spambase) via OpenML: and medical imaging (AlzheimerMRI) via the ADNI database. Source-level acquisition lets users verify provenance and replicate exact conditions. Automatic downloading with caching reduces network overhead. Each loader performs integrity checks to verify data completeness and consistency before use. PU sam- pling is deterministic given the configuration and a random seed, which guarantees exact replication of splits and reported metrics when the same configuration is rerun.
Documentation and usage. The package includes comprehensive documentation designed for PU learning workflows. It provides task-level examples and step-by-step instructions for construct-
34

---

Published as a conference paper at ICLR 2026
ing PU datasets-supportigg both SCAR and SAR mechanisms, well as case-control and single- rrainin-set scenarios. Users can specify class priors, label frequencies, model backbones, and PU learning objectives via YAML configuration files. Key components of the data generation pipeline, training loop, and evaluation suite are clearly structured and documented to facilitate adoption and extension.
Code maintenance and versioning. We are committed to maintaining the codebase, responding user feedback, and encouraging community contributions through structured review process. We adopt strict version control practices, maintain detailed changelogs, and provide tagged releases. Configuration files and experiment logs are archived alongside each release ensure consistent behavior across versions and enable faithful replication when combined with the fixed-seed protocol described in the Reproducibility section.
License. Our code will be released under the MIT License, which permits free use, modifica- tion, distribution, and sublicensing, provided the original copyright notice and permission terms are retained.
G USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models (LLMs) were employed for non-substantive, low-risk tasks in preparing this manuscript. Their use was strictly limited to grammar and typographical proofreading of human- drafted text and verifying the internal consistency of names and cross-references. To ensure scien- tific integrity and prevent any influence from hallucinated content, LLMs had zero involvement in generating, interpreting, or refining core scientific artifacts.
35


Reviews:
[{"review_id": "lRBlO0OtTR", "reviewer": "Reviewer_AKQF", "summary": "This paper introduces PU-Bench, the first unified open-source benchmark for Positive-Unlabeled (PU) learning. It provides a standardized benchmark with a unified data generation pipeline, a framework of 16 state-of-the-art PU methods, and protocols for reproducible assessment. Through studies on 8 diverse datasets (2,560 evaluations), PU-Bench reveals trade-offs between effectiveness and efficiency, robustness and label frequency, and selection bias. This benchmark is meaningful for PU learning.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The benchmark is well-designed with a unified data generation pipeline, a collection of state-of-the-art methods, and standardized evaluation protocols. \n* The benchmark provides a more comprehensive setting and evaluation for PU learning, which is expected to promote the development of the field.\n* The empirical study is extensive, covering multiple datasets and various conditions.", "weaknesses": "* While the benchmark itself is meaningful, the methods included are existing ones and lack certain theoretical analysis.\n* Certain methods perform very well on some datasets but poorly on others. Additional visualizations and analyses of failure cases could provide valuable insights.\n* The benchmark should be expandable, and experimental details can be specified via YAML. Some representative methods should also be compared [1-3].\n[1] GradPU: Positive-Unlabeled Learning via Gradient Penalty and Positive Upweighting. AAAI2023\n[2] Positive Distribution Pollution: Rethinking Positive Unlabeled Learning from a Unified Perspective. AAAI2023\n[3] Positive-unlabeled learning with label distribution alignment. TPAMI2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2FSVgqRd7", "reviewer": "Reviewer_QcbA", "summary": "The authors developed PU-Bench, an open-source framework that addresses the critical lack of standardized evaluation in PU learning research. This modular system includes a configurable data generator, unified training pipeline, and comprehensive evaluation suite. Extensive experiments are presented to compare 16 SOTA methods across datasets.", "questions": "What is the variance in performance across different random initializations, and how does this affect method rankings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Through the results presented in this paper, the authors provides actionable insight for algorithm selection based on data modality, label availability and computational constraints. The key findings reveals valuable insight. No single method performs better universally, performance was highly dependent on the specific data type being used. The study also revealed clear trade-offs between predictive effectiveness and computational costs. All methods showed performance degradation when moving from random to biased labeling scenarios, though Risk-Minimization methods demonstrated greater robustness.\nThis work addresses fundamental challenges in PU learning research by establishing standardized evaluation protocols that enable fair method comparison and reproducible results.", "weaknesses": "1. The paper lacks statistical significance testing to determine whether performance differences between methods are meaningful or due to random variation. Results from single random seeds without multiple runs raise questions about reliability.\n2. Also, fix data splits could affect the generalizability of findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6TfsbnWhSx", "reviewer": "Reviewer_eHTJ", "summary": "This paper focuses on Positive-Unlabeled (PU) learning. It points out that although numerous PU learning methods have been proposed, the field still suffers from inconsistent data generation, disparate experimental settings, and divergent evaluation metrics, which have led to irreproducible results and unsubstantiated performance claims. Furthermore, the lack of a standardized and comprehensive benchmark has hindered rigorous evaluation. To address these issues, the paper introduces PU-BENCH, an open-source framework that provides a unified data generation pipeline and an integrated evaluation framework, aiming to catalyze reproducible, rigorous, and impactful research within the PU learning community.", "questions": "Please reply to my comments listed in \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper compiles and organizes a wide range of state-of-the-art PU learning methods, covering various algorithmic types and multiple public datasets. Moreover, for several advanced methods that were not publicly released, the authors conducted reimplementation efforts, which is commendable and worth encouraging.\n\n2. Through extensive experiments, the paper clearly demonstrates the generality, robustness, and evaluation effectiveness of the proposed framework, highlighting the value of PU-BENCH in fostering reproducible and comparable research within the PU learning community.", "weaknesses": "1. In line 77, within the Data Sampling Scheme section, the description of the two sampling schemes is somewhat complex. It might be clearer and more intuitive if the authors could provide a simple illustrative example—such as showing how data are obtained under each sampling method—or include a figure or diagram to visualize the difference between the two schemes.\n\n2. Although the paper presents a comprehensive benchmark for PU learning, including a unified data generation pipeline and an integrated framework, it lacks an analysis of the rationale behind these design choices. It would be helpful if the authors could explain why this particular configuration is reasonable and what criteria or considerations guided the design decisions.\n\n3. The paper does not explain why these specific datasets were selected, which weakens the justification of the experimental setup. Providing reasoning for the dataset and method selection would make the evaluation more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)