Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms
Authors: Wei Wang
Dong-Dong Wu
Ming Li
Jingxiong Zhang
Gang Niu
Masashi Sugiyama
Venue: ICLR 2026 Poster (2026)
Keywords: Positive-unlabeled learning
weakly supervised learning.
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose the first positive-unlabeled learning benchmark to promote accessible, realistic, and fair evaluation of positive-unlabeled learning algorithms.

Abstract: Positive-unlabeled (PU) learning is a weakly supervised binary classification problem, in which the goal is to learn a binary classifier from only positive and unlabeled data, without access to negative data. In recent years, many PU learning algorithms have been developed to improve model performance. However, experimental settings are highly inconsistent, making it difficult to identify which algorithm performs better. In this paper, we propose the first PU learning benchmark to systematically compare PU learning algorithms. During our implementation, we identify subtle yet critical factors that affect the realistic and fair evaluation of PU learning algorithms. On the one hand, many PU learning algorithms rely on a validation set that includes negative data for model selection. This is unrealistic in traditional PU learning settings, where no negative data are available. To handle this problem, we systematically investigate model selection criteria for PU learning. On the other hand, PU learning involves different problem settings and corresponding solution families, i.e., the one-sample and two-sample settings. However, existing evaluation protocols are heavily biased towards the one-sample setting and neglect the significant difference between them. We identify the internal label shift problem of unlabeled training data for the one-sample setting and propose a simple yet effective calibration approach to ensure fair comparisons within and across families. We hope our framework will provide an accessible, realistic, and fair environment for evaluating PU learning algorithms in the future.

Published as a conference paper at ICLR 2026
ACCESSIBLE, REALISTIC, AND FAIR EVALUATION OF LEARNING ALGORIHTMS
Wei Wang 1.2#1 Dong-Dome Wu 2
Ming Li 3.2
Jingxiong Zhang 2 Gang Niu
Masashi Sugiyama 1,2
RIKEN, Tokyo 103-0027, Japan
2 The University of Tokyo, Chiba 277-8561, Japan
8 University of Maryland, College Park, MD 20742, USA
wei wangeriken. ip
[dongdbogwu1230, 1m1303mmmmmm gang. niu. u.uumm (al (egmall............... com 76564729078edu k.u.-okko...... jp sugi @k lok.u-ton w-tokyo..ocooooo ac. ip
ABSTRACT
Positive-unlabeled (PU) learning is weakly supervised binary classification problem, in which the goal is to learn binary classifier from only positive and unlabeled data, without access to negative data. In recent years, many PU learning algorithms have been developed to improve model performance. However, exper- imental settings are highly inconsistent, making it difficult to identify which algo- rithm performs better. In this paper, we propose the first PU learning benchmark to systematically compare PU learning algorithms. During our implementation. we identify subtle yet critical factors that affect the realistic and fair evaluation PU learning algorithms. On the one hand, many PU learning algorithms rely a validation set that includes negative data for model selection. This is unre- alistic in traditional PU learning settings, where no negative data are available. systematically investigate model selection criteria for
To handle this problem, PU learning. On the other hand, PU learning involves different problem settings and corresponding solution families, i.c., the one-sample and two-sample settings. However, existing evaluation protocols are heavily biased towards the one-sample setting and neglect the significant difference between them. We identify the inter- nal label shift problem of unlabeled training data for the one-sample setting and propose a simple yet effective calibration approach to ensure fair comparisons within and across families. We hope our framework will provide accessible, realistic, and fair environment for evaluating PU learning algorithms in the future.
INTRODUCTION
In binary classification, both positive and negative data are usually necessary to train an effective classifier. However, in many real-world applications, collecting negative data can be more challeng- ing than collecting positive data (Hsieh et al., 2015; Zhou et al., 2021). In positive-unlabeldd (PU) learning, only positive and unlabeled data are needed. The objective to train a binary classifier that assigns positive or negative labels to unseen instances. Therefore, PU learning is promising weakly supervised binary classification approach for many real-world problems where negative data are difficult to obtain, including recommender systems (Yi et al., 2017; Chen et al., 2023), anomaly detection (Ju et al., 2020; Tian et al., 2024; Takahashi et al., 2025), knowledge graphs (Yin et al., 2024), and link prediction (Wu et al., 2024; Mao et al., 2025).
In recent years, there has been significant progress PU learning algorithms. PU learning can be divided into three groups: cost-sensitive PU learning algorithms (du Plessis et al., 2014; Zhao al., 2022), sample-selection PU learning algorithms (Chen et al., 2020b; Wang et al., 2023a), and biased PU learning algorithms (Teisseyre et al., 2025). Cost- Cost-sensitive algorithms assign different weights to positive and unlabeled data to approximate the classification risk. Sample-selection algorithms select high-confidence negative data from unlabeled data, which are then given supervised learning
$ Equal contributions. "Corresponding author: Wei Wang <wei wwngwwaww@wwe.......... wwwg@waa@@r............ jpp.

---

Published as a conference paper at ICLR 2026
algorithms. Biased PU learning algorithms model the biased generation process of positive data and exploit various correction approaches.
Although many PU learning algorithms have been developed to improve generalization perfor- mance, there is lack of unified experimental setup in the literature for fairly comparing different PU learning algorithms. The experimental settings of different papers are not consistent with each other, making it difficult to tell which algorithm is better. It has been observed that subtle differ- ences in experimental settings can gready affect the model performance of PU learning algorithms. Additionally, subtle algorithm details, including data augmentation, algorithm tricks, and warm-up strategies, can also greatly affect model performance (Zhu et al., 2023b; Wang al., 2023a). There- fore, a unified experimental protocol is necessary to further promote the development of PU learning algorithms. In this paper, we propose the first PU learning benchmark to systematically and fairly compare state-of-the -ftt----tt--ttttt-e PU learning algorithms with unified experimental settings. We propose careful and unified implementations of the data generation, algorithm training, and evaluation pro- cesses for PU learning algorithms. This makes it easier for users to validate the effectiveness of their newly developed algorithms.
In our implementations, we observe that many PU learning algorithms rely on a validation set con- taining both positive and negative data for meta-learning, model selection, or early stopping (Chen et al., 2020b; Zhu et al., 2023b; Long et al., 2024). However, accessing negative data is unrealistic and contradicts the original motivation of PU learning (Elkan & Noto, 2008), which goes against obtain some
the advantages of PU learning in not depending on negative data. Actually, if negative data, we can directly apply supervised learning techniques, which can greatly boost model performance (Sakai al., 2017). Therefore, standardizing the composition and use of the validation set is vital to fairly and practically evaluating PU learning algorithms. In this paper, we system- atically revisit the model selection criteria for PU learning by using only positive and unlabeled validation data, and validate their effectiveness with both theoretical and empirical analyses.
In addition, there are different fam- ilies and corresponding solutions of PU learning algorithms, but exist- ing evaluations fail to consider the differences between these families. From the perspective of data gener- ation processes, there are two types of PU learning problems: the one- sample (OS) and two-sample (TS) settings. In the OS setting, the pos- itive and unlabeled training sets are generated sequentially. An unlabeled dataset is first sampled from the marginal density. Then, if an instance in the unlabeled dataset is positive, its positive label is observed with constant probability. If an instance in the unlabeled dataset is negative, its label is never observed, and the instance remains unlabeled. Finally, the ob- served positive data constitute the positive training set, while the remaining unlabeled data constitute the unlabeled training set. In the TS setting, the positive and unlabeled training sets are generated in- dependently, meaning that the density of unlabeled training data is the same as the marginal density. This indicates that the density of unlabeled training data is different in these two settings. Figure shows an example of the distribution of unlabeled data under the OS and TS settings. We can find that the class priors of the two settings are different. This causes an internal label shift (ILS) problem for the unlabeled training data when adopting the OS setting as the evaluation setting. Unfortunately, this problem has typically been overlooked. Existing evaluation protocols are heavily biased towards the OS setting and compare OS and TS algorithms together without specific manipulations. This can deteriorate the performance of TS PU learning algorithms and lead unfair experimental compar- isons. Therefore, we identify the ILS problem for the first time in the PU learning literature and propose a simple yet effective calibration approach to overcome with theoretical guarantees.
(a) One-Sample
(b) Two-Sample
Figure I: An example of the comparison of the distribution
of unlabeled training data in different PU learning settings.
We draw the following key takeaways from our benchmark results:
No single algorithm outperforms all others every dataset or evaluation metric; some early, simple methods already achieve strong classification performance. Therefore, we should choose which PU learning algorithm to on by-see basis.
2

---

Published as a conference paper at ICLR 2026
The model-selection problem in PU learning must be addressed when designing new algorithms or conducting empirical comparisons, and different selection criteria should be used for different test metrics.
The performance of TS PU learning algorithms degrades significantly when evaluated under the OS setting without proper adaptation, OS protocols in the existing PU learning literature do not reflect the true performance of TS methods. Hence, differences between OS and TS settings must be considered ensure fair cross-family comparisons.
PRELIMINARIES
In this section, we present the background of PU learning and existing state-of-the-ar algorithms.
2.1 POSITIVE-UNLARELAN LEARNING
Problem Setting, Let A' S Id denote the d-dimensional feature space and y = [+1, -1; denote the binary label space. Let p(æ, y) denote the joint probability density over the random variables (z, y) E X' x y. In PU learning, we are given a positive training set Dp = {( + +1) and an unlabeled training set Du (=:);=np+1 -
ply + +1) denote the class prior probability of the positive class. Let p(zly +1) and p(zly = 11 denote the positive and negative class- conditional densities, respectively. Let p(æ) denote the marginal density. The goal of PU learning is learn binary classifier X' R from Dp U DU that maximizes the expected accuracy
Let
ACC(f) = Ep(,,)) (yf(x) > 0),
where E denotes the expectation and II denotes the indicator function. However, since the 0-1 loss function is difficult to optimize, we usually use a surrogate loss function L, such as the logistic loss. Then, the classification risk to be minimized can be expressed
R(f) Ep(a,v) [ (ʃ (æ) .y]]
Data Generation Assumption. There are mainly two data generation assumptions for PU learn- ing, i.e., the TS setting (du Plessis al., 2014; Niu et al., 2016; Chen et al., 2020a) and the OS set- ting (Elkan& & Noto, 2008; Coudray et al., 2023). In the TS setting, we assume that Dp and Du are generated independently, where Dp is sampled from the positive conditional density p(æ|y = +1) and Du is sampled from the marginal density p(x). In the OS setting. Du and Dp are generated sequentially. First, Du is sampled from the marginal density p(z). Second, for each example in Du. if it is positive, its positive label is observed with constant probability negative, its negative label is never observed and the example remains unlabeled with probability 1. Finally, the observed positive data constitute Dp and all the unlabeled data left constitute Dú:
O. If an example is
2.2 POSITIVE-UNLABLLE LEARNING ALGORITIIMS
From methodology taxonomy perspective, PU learning algorithms can be divided into three groups: cost-sensitive algorithms, sample-selection algorithms, and biased PU learning algorithms. Cost-sensitive algorithms assign different weights to positive and unlabeled data approximate the classification risk (du Plessis et al., 2015; Kiryo et al., 2017: Hsieh et al., 2019). Some algorithms are equipped with other regularization techniques to further improve performance, such entropy min- imization (Zhao et al., 2022; Jiang et al., 2023) and mixup technique (Chen et al., 2020a; Li et al., 2022). Sample-selection algorithms select reliable negative examples from the unlabeled dataset for supervised learning (Chen et al., 2020b; Garg et al., 2021; Wang et al., 2023a; Li et al., 2024). Biased PU learning algorithms consider the density of positive data to be biased and adopt different strategies to model the bias (Bekker et al., 2019; Gong et al., 2022; Coudray et al., 2023; Wang et al., 2023b; Teisseyre al., 2025).
MODEL SELECTION FOR POSITIVE- DOLLBEED LEARNING
In this section, first explain motivation for studying the model selection problem in PU learning. Next, we review the criteria used for model selection in PU learning, including the proxy accuracy, proxy area under the curve score, and oracle accuracy.
3

---

Published as a conference paper at ICLR 2026
3.1 MOTIVATION
Although model selection is well established for supervised learning, it non-trivial for PU learning because negative data are inaccessible. This problem is partieularly important for deep learning algorithms because they have many hyperparameters, including universal hyperparameters (e.g., learning rates and weight decay) and algorith--spccffc hyperparameters. Previous work has usually conducted model selection by assuming validation set with labels (i.c., both positive and negative labels) is available. However, this assumption is inconsistent with the definition of PU learning, in which negative data are unavailable. Therefore, it is important to study the model selection problem systematically for PU learning. According to the original definition of PU learning (Bekker & Davis, 2020), assume that the validation set consists of positive validation set D/p {(,,, +1) tn
and an unlabeled validation set D'y { i=np+1
3.2 PROXY ACCURACY
Although the validation accuracy cannot be directly calculated because of the absence of negative data, it has been shown that the expected accuracy can be expressed using only positive and unla- beled data (du Plessis et al., 2014). This motivates us to apply it for model selection.
Definition (Proxy accuracy (PA)). The proxy accuracy of binary classifier f on the PU validation dataset is defined as
" - (f(xt) 0) an (f(x) < 0) if the setting is TS: ni +1=
PA(f)
2st
E T (f(z))
> 0) mpt i==1
A T (f(#() < 0) if the setting is OS.
Up
PA can be calculated using only PU validation data when the class prior æ is known or estimated (Ra- maswamy al., 2016; Yao al., 2022: Zhu al., 2023a). The following proposition then holds. Proposition 1. For two classifiers fi and f2 that satisfy [[PA(f)] < E [PA(f2)), we have ACC(f) < ACC(f2).
The proof can be found in Appendix B.I. According to Proposition 1, classifier with higher expected value of the proxy accuracy can achieve higher expected accuracy even when the true labels are inaccessible. This means that when the number of validation data is large, the best model chosen using the PA metric will achieve the highest accuracy in expectation. One limitation of PA is that knowledge of the class prior is necessary. However, knowledge of π is an intrinsic and common issue in PU learning. Addressing this issue is beyond the scope of our paper. In practice, we estimate it using off-the-shelf estimation methods (Ramaswamy et al., 2016; Garg et al., 2021; Yao al., 2022), and we et al., 2022).
obtain this knowledge in some real-world applications (Sugiyama
3.3 PROXY AUC SCORE
has been shown that the area under the curve (AUC) score can be robust to corrupted labels for binary classification (Charoenphakdee et al., 2019; Wei et al., 2022). Therefore, it is promising to employ it for PU model selection. First, we introduce the expected AUC score as follows:
AUC(J) p1p Eppe)=--1) [ (f(a)> f(x)) + 1 (f(x) s(a'))
(4)
We then consider the unlabeled validation data be corrupted negative data and calculate the AUC score as follows, which is suitable for both OS and TS settings.
Definition 2 (Proxy AUC score (PAUC)). The proxy AUC of a binary classifier f on the PU vali- dation dataset is defined
1
(I(f(e)) f(x,)) + ( (((aa)=f(a())) (5)
PAUC(f)
The following proposition then holds.
4

---

Published as a conference paper at ICLR 2026
Proposition 2. Under both the OS and TS settings, for classifiers fi and f2 that satisfy E [PAUu(fF)] < B[PAUC(f2)] we have AUC(fi) < AUC(f)).
The proof can be found in Appendix B.2. Proposition 2 shows that classifier with higher expected value of the proxy AUC score will achieve a higher expected AUC score, regardless of whether the setting is OS or TS. Therefore, when the number of validation data is large, the model selected with the highest PAUC can also achieve the highest expected value of the AUC score. An advantage is that the class prior ı is not necessary when calculating the PAUC.
3.4 ORACLE ACCURACY
Finally, we introduce the oracle accuracy metric if the true labels of unlabeled data available. Definition 3 (Oracle accuracy (OA)). The oracle accuracy of binary classifier f on the PU valida- tion dataset is defined
tabt Li-n +1
(v/f((=) > 0) if the setting is TS;
OA(S)
=
(6)
- u tno -i=1 (1 > 0) if the setting is OS.
Here, 1/f is the true label of æt
Notably, the implementations for the OS and TS settings differ slightly, as it is important to en- sure that the validation data have the same distribution as the test data. OA is natural metric for supervised learning. However, due to the absence of negative data, it cannot be calculated in the traditional PU learning setting. Unfortunately, this metrie has actually been widely used in the PU learning literature because of a lack of standardized benchmarking. Therefore, this paper only in- cludes the results of OA for comparison. We recommend using PA and PAUC in future PU learning experiments, especially in real-world applications where negative data cannot be obtained.
4 INTERNAL LABEL SHIFT IN POSITIVE- UNLABBLED LEARNING
In this section, we first introduce the ILS problem in PU learning. Then, we provide a calibration approach solve it with both theoretical and empirical analysis.
4.1 PROBLEM STATEMENT
The difference between the OS and TS settings lies in the density of the unlabeled training data. Specifically, the density the unlabeled training data equals the marginal density in the TS setting but differs from it in the OS setting. We formalize the ILS problem as follows.
Definition 4 (Internal label shift in OS PU learning). In the OS setting, the density of Du is p(x) πp(хTy +1) + (1- π p(x y -1). where the class prior under the OS setting. Here, the positive and negative condiconnai densities are the same those of the test data; however, the class prior is ग
c)rr(1 (1 cx). which differs from æ, the class prior of the test data. This mismatch causes internal label shift between the unlabeled training data and the test data.
Many cost-sensitive PU learning algorithms have been developed for the TS setting. In these algo- rithms, positive and unlabeled data assigned different weights approximate the classification risk (du Plessis et al., 2014; Chen et al., 2020a; Zhao et al., 2022). Because the weights are the- oretically derived, small discrepancies in data assumptions can degrade performance. Conversely, sample-selection PU learning algorithms select reliable negative data from Du and need not rely strictly on the specific data generation process (Zhu et al., 2023b; Wang al., 2023a; Li et al., 2024). However, many papers adopt only the OS setting and ignore the distribution mismatch, caus- ing experimental datasets to violate the assumptions of TS approaches.
To demonstrate how ILS affects model performance, we use uPU (du Plessis et al., 2015) as an example in Section it is representative TS algorithm and underpins many subsequent cost- sensitive methods. Under the TS assumption Du
lid. p(æ), du Plessis al. (2015) proposed the
Our analysis and calibration approach be extended other TS algorithms well.
5

---

Published as a conference paper at ICLR 2026
40- 8
80-
Accurccy
Accurccy
Accincy 8
Accurscy
90
10k.
IOLL
Pastrive
Poattive data
Posttive data
Possiive dara
(a) uPU
(b) unPU
(c) inPU-GA
(d) PUSB
Aucuracy
20
Accurcy
Accuray
Accurssy
20
45
10
2k
46
10k
2k
46
Mk
1kk
4k
102
# Postivee data
# Positive data
e PostEvo data
N Positive data
(e) Dist-PU
(f) VPU
(g) uPU
(h) nnPU
80
80-
Accuracy
Accurscy
Accurac
40
Accuracy
40
Positive
data
Positive data
data
(i) mmPU-GA
( PUSB
(k) Dist-PU
(I) VPU
Figure 2: Classification accuracies of TS PU learning algorithms in OS and TS settings of PU
version of CIFAR-10 with varying amounts of positive data. Figures (a) to (f) are for Case 1, and Figures (g) to (l) are for Case 2.
unbiased risk estimator (URE)
BP R(n) Σ (1)( E(((w:)) 1)) up
ap+nu
Σ (((x)),-1). AU s=nr+1
which enjoys risk consistency because B[A[(]] R()). Let f arg fen [((J) and s arg min R(f) denote the classifiers that minimize the empirical risk in Eq. (7) and the risk in Eq. (2), respectively, where F is the model class. It is known that f - f as mp - 00 and my 8 under the TS setting (Niu et al., 2016). Under the OS setting, however, [[ * R(f), so f- no longer holds (see Appendix B.3). Consequently, minimizing losses designed for the TS setting may not yield high-performing classifiers when datasets are generated under the OS setting, leading to unfair comparisons when all methods are evaluated in the OS setting. The bias stems from the ILS problem: under the OS setting, the class prior of Du differs from x, breaking the consistency of many TS algorithms and degrading their performance.
4.2 THE PROPOSED CALIBRATION APPROACH
To address the bias, we incorporate the true densities of Du for TS algorithms. The following theorem shows that the risk rewrite for the uPU approach differs under the OS setting.
Theorem 1. Under the OS setting, the classification risk in Eq. (2) can be equivalently expressed as
R()) = -lEp(=/y==1) 1) [L((()), +1) + (c (x) 11] -1)] + (1 cτ) Ep(w) [[(x,),-]]]]
The proof is given in Appendix B.4. Theorem shows that the classification risk can be equivalently expressed as expectations w.r.t. the densities of positive and unlabeled data under the OS setting. We then obtain a calibrated risk estimator using the positive and unlabeled datasets:
np R(f) = up Σ
((((()))) + (c- ()( + 1 7i di Fl
Σ 1=
(f(xx))
(8)
When the class prior н is known or estimated, we obtain an unbiased estimate of as c np/ nu). Let f arg min R(J) denote the optimal classifier that minimizes the calibrated risk

---

Published as a conference paper at ICLR 2026
dop
Podtin dot
* Prottim doss
Postiivn dxp
PPettive dea
(a) uPU
(b) nnPU
(c) nnPU-GA
(d) PUSB
(c) VPU
10.
Rusiiivedal
Positicc
Positive
Poullire
Poskiive
(I) uPU
(g) anPU
(h) nnPU-GA
ii) PUSB
VPU
Figure 3: Classification accuracies of TS PU learning algorithms in OS and TS settings of PU
version of ImageNette with varying amounts of positive data. Figures (a) to (e) are for Case 1, and
Figures (f) to (i) are for Case 2.
estimator in Eq. (8). Let Rap (F) and Ray (F) denote the Rademacher complexities defined in Appendix B.5. Then, the following theorem holds.
Theorem 2. Assume that there exists a constant Cj such that sup ll & Cy and a constant Ct such that Vy, supllle,, l(z.y) v) & Cg. We also assume that Vy, the binary loss function E(2, y) Lipschitz continuous
with Lipschitz constant Le- For any 8 0, the following inequality
holds with probability at least 1 - 5:
R(f) ((f)) <(8 L,) (F) + (4 4CT) L, (F)
(4 - 2c) + (2 - 20
In 2/8 2
Vmp
Vnu
The proof is given in Appendix B.5, Theorem 2 shows that f - f as np - 00 and nu - 00. because (FF 0 and Rer.p. (F) 0 for all parametric models with bounded norm, such as deep neural networks trained with weight decay (Golowich et al., 2018). Notably, Eq. (8) can be equivalently transformed into Eq. (7) if we incorporate Dp into Du when computing the last loss term w.r.t. unlabeled data in Eq. (7) (see Appendix B.6). Thus, when Dp is used in both loss terms, the ILS bias is eliminated, because the union of positive and unlabeled data is unbiased w.r.t. the marginal density. This motivates simple yet effective calibration approach that adapts TS algo- rithms to the OS setting. summarized in Algorithm 1. We augment Du with Dp when computing the loss on unlabeled data, so the replenished set is marginally unbiased and suitable for TS PU learners.
4.3 EMPIRICAL ANALYSIS
Algorithm Calibrated Two-Sample PU Learning
Require: Two-sample PU learning algorithm AA positive training set Dp. unlabeled training set Dv. maximum epochs Timx, maximum iterations Tmx--
We validated the existence of the ILS problem and the effectiiveness of the proposed calibration approach. We used uPU (du Plessis et al., 2015), nnPU (Kiryo et al., 2017). unPU- GA (Kiryo et al., 2017), PUSB (Kato al., 2019), VPU (Chen et al., 2020a), and Dist-PU (Zhao et al., 2022), six representative TS PU learning algorithms.
Ensure: Classifier produced by A.
1: for 1.2....,, Tixx do
2: Shuffle Dp and Dv:
3:
for l: = 1,..., Jmss do
4:
Fetch mini-batch DE from Dp and D. from Du; Call A.TRAINNONE_RRRAAEEA ONE_BATCH(DD, D UD!)
5:
6: end for
We used 7: end for
10 (Krizhevsky & Hinton, 2009) and ImageNette (Deng et al., 2009) as the datasets. We synthesized PU training datasets with different definitions of positive and negative labels, where the details are presented in Appendix C. We did not include the results of Dist-PU on ImageNette since Dust-PU did not work well on this dataset. We considered both the OS and TS cases using the same experimental settings, and the only difference lay in how positive data were generated. Figures and 3 show the experimental results CIFAR-10 10 and ImageNette with varying amounts of positive data, respectively. We can observe that using TS approaches directly in the OS setting yields infe- rior performance. Their performance consistently drops when the number of positive data increases,
7

---

Published as a conference paper at ICLR 2026
Table 1: Test results (td) of accuracy, AUC, and F1 score for each algorithm on CIFAR- 10 (Case 1) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, "-c" indicates using the proposed calibration technique in Algo-
rithm 1.
Test metric
Accurac
AUC
F1
Val metre
PAUC
PAUC
PAUC
QA
PUBN
26.40:0.46
56.24=0024
67.33
92.96-0.49
03.44=0074
94.33:11.23
20.52
50.07:411.8
56. 56.95:4:244
PAN
76.64-0.78
77.56+0441
78.91+0.59 59
87.11 12.11-0886
67.28+0091
85. 7.0006
79 19.08+00.75
79.41+0061
78. 74.04+00.5
CVIR
85.45=1005
63.32=0444
96.4740.48
86.1943.88
54.71:0.33
=0.51+0400
PIMIX-E
50.00:0000
92.62:066
10.07:0000
67, 12.45:112.03
PIMIX-C
86.3620..5
55.73-90.6
10.55
92.70
93.09=0.05 0.05
92.161043
86,44
85.93-10.0
86.72:19.8
LBE
82.7110.75
73.60.1.229
63.03.10.3
92.09 0.15
95.21 0.04
92.25 92.26610.31
83.79.0044
78.72.10.76
14.31
Count Loss
00.39-00.32
79.86=0138
82. 0..39
90.43-069
90. 90.40 45
10:20+1.22
82.60=1228
$1.85 $1.83-20133 3.39
13.11=00.99
PU
85.57:00.16
00.61:0.55
65.91
91 56 0.49
92.89 0 70
11.00:1.00
85.85 1:009
54.90::0.56
85.47:0.32
Pol
50.30++0.10 10
50.00-0.00
51. 11:44
56.56-111 11.51
69.45 00.45-500 04
00.60+0..1
66 66:440003
66. +6.67+00.00
82. 90.77
PUe
77.85
78.51
10.45
86.8 061
86.60.0.45
11.58
19.551055
78.01
78.99 10.28
GLWSS
84.4620045
79.83:2230
0.440.
98.54
93.48
25.652036
$2.692 1.66
56. 66.26-20.32
uPU
1.224:2.225
26407=2.43
82 0.44
88.72 08.72=0.40
89.05=0117
N7. 07.36
81 21,05
77.01:1441
SI. 01.56
uPU
85.9
64.202:0.49
56.48 08.22
92.65 603300..
93,00
22.231
85 45.9610.45
$3.04
86, 12:0100
mPI
82.07=0111
75. 13.56=033 2020
82 40+00.3
92. 12.22-00 15
22.32-0
91 9.44
+0.0
79. 02.64-2 6..5
83 49+00.5
mPU
85.5310.20
86,03 0.66
$6.35 10.25
92.19 0.35
93.00 0.55
92.95 10.38
85.9010.26
88.71 15.70
36, 29 10.30
mPU-GA
84.26+00.0
84.18-0040
64.03-+0.00 30
79
92.26-00300
05.25+0.666
64.87-0022
$4.63-0-.22
54.58-0.53
GN=
25,901022
36.26:0.31
16.13:1.225
92.81
92.96-0.47
02.0010:22
35,9010.31
85 15.661227
85,3719.19
PUSB
81.53
$2.49 1.00
82.91 10.70
$1.53 0.77
82,89 1002
10.90
83.29 1047
$3 1.077
$4.12
PUSB-e
86.15
84.7620.17
56.49 M0.4920.17
06.15-0.37
84.7620.17 17
96.4920.37
86.09=3144
83 13.8911119
56. 56.23-20.18
VPU
84.91-0.52
65.71-7332
65 8000000.:4
91.89 0.08
92.89 54
90. 8.-00050
84.15-+0.59
42.75++17.9
+0.91
VPULec
86.41=0.75
82.85=166
87 67.65++0:25
92. 0.13
93.51=0.51
91. 7941
86. 86.73+00.55
84. 14.5
87.41
Dist PU
81.6410.45
79.31
83 K3.56.10.66
90.91 0.54
91,90 0.48
10.59
83.34
$1. $1.94.10.23
$3.261008
Dust-UUse
87.06:2.45
87.38-0033
85.47:0..25
94.93-031
94.55:0.21
94.90:10.32
37.63-0033
07.28:0029
38.18:00255
even though we have more knowledge of the true labels of positive data in the unlabeled dataset. By using our proposed calibration approach, the performance can be improved greatly and can even sometimes surpass the performance in the TS setting. This shows the effectiveness of our calibration approach in improving TS approaches under the OS setting.
BENCHMARKING
LEARNING
In this section, we first introduce the benchmark settings, then we present the benchmark experi- mental results. The code package is available at https: / ht////gtttbbb/t// com/ htt///wwdddd//e/ Ad/UUBench.
5.1 BENCHMARK SETTINGS
We included seventeen representative PU learning algorithms: uPU (du Plessis et al., 2015), nnPU (Kiryo et al., 2017),, anPU-GA (Kiryo et al., 2017), PUSB (Kato et al., 2019), PUbN (Hsieh et al., 2019), VPU (Chen et al., 2020a), PAN (Hu et al., 2021), CVIR (Garg et al., 2021), Dist- PU (Zhao et al., 2022), M (Li et al., 2022), M-XIXCC (Li et al., 2022), LBE (Gong et al., 2022), Count Loss (Shukla et al., 2023), Robust-PU (Zhu et al., 2023b), Holistic-PU (Wang et al., 2023a), PUe (Wang et al., 2023b), and GLWS (Chen al., 2024). We evaluated our methods two image datasets (CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNette (Deng et al., 2009)) and two UCI datasets (USPS and Letter) from the UCI Machine Learning Repository (Kelly et al., 2023). ImageNette is a curated subset of the larger ImageNet corpus, containing ten easily distinguishable categories: tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, and parachute. We synthesized PU versions of these datasets; detailed infor- mation can be found in Appendix C. We used ResNet-34 (He al., 2016) for image datasets and a multilayer perceptron (MLP) with a hidden layer width of 500 equipped with the ReLU (Nair & Hinton, 2010) activation function for tabular datasets.
Following the widely used validation protocol (Raschka, 2018: Guirajani & Lopez-Paz, 2021; Wang et al., 2025), we divided some training data from the positive and unlaheled datasets into the positive validation set D/P and the unlabeled validation set D', respectively. We used various test metrics, including accuracy, AUC score, Fl score, precision, and recall. We first trained a model with training sets Dp and Du. Then, we evaluated its validation performance based on the metrics in Section 3 as well as its test performance on a test set with true labels. We randomly selected a set of hyper- parameter configurations from given pool. For each validation metric, selected the checkpoint with the best validation performance on D2p D',, and recorded the corresponding test metrics. We recorded the mean test metrics and standard deviations obtained with different data splits.
8

---

Published as a conference paper at ICLR 2026
Table 2: Test results std) of accuracy, AUC, and F1 score for each algorithm on CIFAR- 10 (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric shown in bold.
metrne
AUC
Test F1
Val menice
PA
PAUC
CIA
PA
PAUC
GA
PA
PAUC
CA
PUTN
78:26-1.00 1.01
79.50
79.94-0.36
$7.81+00.65
58.00
58. 38.08-1
80:47:0:65
09.17:00.88
79:91:+0221
BAN
61772-2.74
60.6124334
63.48_2.71
68.71
44.56
09.63:3.43
70.73-1.400
70887 1.72
09:2511..04
CVIR
1.49
79.50
1.46
80.44 0.68
88.10
1
87.9811 1.33
$8.6810.81
80.86 0.97
80.69 1.33
0..44
PAMIX-E
59.04:4.55
50.00:0.000
59.13::.62 4.63
26 1-4 4.26
84.5210.84
14.11:4.116
70.45 20455-2.00
44.44 18.14
70.451:2.00
PAMIX
78.05-0.55
77.422 1.40
78.70-0.50
26.1320.79
09.522036
09.0610922
79.9020.49
LBE
72.47_1.55 1.50
63.5412:55 2.56
75.96 0.58
84.02 10:90
14.75
43.97.0.97
77.
72.96
76.06 10.8
Count Loss
74,44-0.44
74:75=045
76.57=0775
88 82
62.941.05
04.5
77.1-0054
76.70=00.55
78.27=0999
Robust-PU PU
78.94-0779
78.45-0011
79 09.00-0.11
85.
25 00
57.1549.76
0..33
77.16:10.06
79.79
Hotisacc PU
55.00-0..16
56.04 1.99
71.18:1 1.20
78.03
2.55
67.9616.67
76.9313.15
09 020020004
64.49 18.12
23.64
PUe
68:0==041
67. 11:40=1100 150
71 01.05=0552
78 01
78 06
7344
73.0500.11
71 06 1.35
GLWS
77.71=0.71
76.22
20.35
$7.4520.51
00.4020.37
09
20.4720.47
UPU
66.21=1.40
69:00-11000 1.04
00.06
16.46:1.655
78 78
27:9720:00
71 02
72.7810.47
70 1.55
02222-0026
79.29
79.02 0.99
85. 19 10.46
51.7610.35
57.1
79 19.48.0.22
18.1910:455
78.60 1.22
mPP
74.27 1.26
62.67 1 1.09
77.62 0.68
86 0:07
16.5310.16
56. 06..8
78 0.500
72.57 0.51
79 20.22
naPU-c
75.74-0::
78 18.49
20,44
666520.31
26.1620.22
01
790720039
maPU GA
76.39=1.15
06.73
76.38_00.1
56.41
16.09
06.58-10.84
0995
78.761: 1.11
28.2220.55
c
78.00 0.52
79.12 0.9
83.25 1.30
15.821104
11.67
0.266188
77.78 17.7810.48
79.011092
PUSS
25.74=061
78 11555
78.35 0.
75.24 20.00
78 25.5
7H. 78.35-0.221
79 03.38:00.43
79 79.3320.59
79.7920.61
PUSBIC
79406-0.455
77:98:11554
79. 09.19-0332
0.445
77.9810.54
79.1920.33
0.106
77.43 01.4320.40
79.29
VPI:
26.99-10 0
63.22-55.00
77.31-036
85 65.47
67.06+0.45
56.07+0.67
75 15-1 31
59 15:22115.70 74
75.45 1.33
VPU--
17.00-011
78:20=0:00
86. 09.99
67,5020.46
56.3220.28
80.12:022
50.522055
00.5620.11
Dis-PU
73.40-0.99
74.33-0.5
74.09=0.00
SU. 0.45
52,0940.00
01.48:2075
76
70.05=0.155
Dial-Plee
22.57-3447
7441+2667
24.30-2.73
30.
82.4942.68
11:94422.00
34
25-27-2677
73,8--3.25
Table 3: Test results (mean std) of accuracy, AUC, and F1 score for each algorithm on Ima- geNette (Case 1) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold.
Text metre
Test ACC
AUC
Test F1
Val menie
PA
PAUC
CA
PAUC
PA
PAUL
QA
PUNN
75.60-0.02
77.07-0.44
78.99-0.57
$4.82+00.25 0.22
86,25+00885
67.45+0037
77.3++0.1 16
76.30++1.26
70.25 70.25++0.66
PAN
10.74_1.17
51.5220.46
16.90_1883
53.71
15.4811003
18.75111.79
65.2410.19
32.31 14.81
45.43::2.65
CVIR
76.78 0.36
78:26 1.62
81.01 0.67
87.96 0..35
$8.2910.65
89.2810.08
80.1210336
79.5210.35 0.77
81.51 0.45
PIMIX-E
74.81-2.30 2:36
49.71 0.44
25.09:2.30
85.84-10.66
22.91
76
7-
PIMIXCC
78.51-141 1.61
78.91
85.
1.50
17.33-1.227
10.26-1.24
90.35
00.48.0.37
LBE
78.52_ 0.41
78.7310065
79.20 0.36
86.84 10.37
36.3110.61
36.1610.78
0.39
77.09 1.48
78.14
Count Loss
24.96-0.85
75:05 1.6
78.07=0.773
85. 0.225
20.54
85. 25.7520.74
77 71
77.0520.77
78. 72.9210091
Robust-PU PU
77.670027
25.53-2.04
78.73-0.4
83. 93 04.64
05.2220.11
04.56
77 0.2477447
71.7824.44
0..06
Holistic PU
51.36-0.47
3,66
53.62 24
58.85
16.45
15.25
68.181031
64.231..10 1.18
51.5811.1 1.27
6747=1.8
71 16.461122
75. 13.3551.5
77.2941.99
17.
20
70.97
11.46:22 1.56
GLWS
26.34_0.36
74.96 1.62
70.0
87. 00.00
86.8920.71
36.9620.74
75.9320.45
75.522
79.5620.67
UPU
01507-095
69.1426.15
73 03749-0774
22.24
74.88
71 71.5822445
74.95
1..00-0.99
42.5414.40
77.76_0666
10.33
05.82
10.65
07.1610227
6133319.88
77.19
nnPU
1.34
66.81
77,50 0.74
10.38
121077
16.22
06
1..97.0..7
anPU-e
26.51-0611
76.95-0755
T766-002
83 87 00.7
55.06
1.9351.224
09
1..59
71.320099
unPU GA
15.70_0.36
78.7220064
0.47
53.74.10.6
56.86.10.88
14.55
78.33110110
75 73.98111 1.22
19.1310.24
c
77.65 1.558
72.91 1 2:35
78.56 0.06
$1.45 1 1.32
04.7510.55
$2.89 1121
77 1.51
65.42.1..22
78,3410.12
PUSB
72.73-01.54
71.09-00.74
76.73-0335
73 0.55555
77.992068
76.91 20.33
77 26:20.33
TB 08.6520009
78.6620.16
P
26.37-0.16
17.3620.36
77.81
16.4820.155
11.3120.33
0..6620.60
37.3220177
76.4220.444
78152000
VPU:
50.910004
61.72-001
21.21
60.35+0.277
73.84-4-.69
53.86-6633
0.14=0.11
VPU-
77,480032
78:0=00.50
76.06-0991
03.35
54.65400.99
54.2620966
75.09-0000
77.6220.69
Dist PU
2.50
71 2:34
24.98-0779
131
13.1811.51
83.9220.75
75.58
75.76111011
77.00-0.775
22.01-0.99
65.88-1.55 1.55
75.84-11 16
79.51
77.85-60.61
50.91+11.18
74 14.78-0.55 58
$2.05 10.16
74.10-008
5.2 BENCHMARK RESULTS
Tables 1 to 4, and 7 to 18 in Appendix D report detailed experimental results in terms of differ- ent metrics on CIFAR-10, ImageNette, Letter, and USPS, and the hyperparameters are determined with PA, PAUC, and OA, respectively. In addition, Figures 4 to 7 show the overall performance of different algorithms. For ease of presentation in figures, we did not include the algorithms where the performance obviously inferior. We can draw the following conclusions based on the exper- imental results: 1) The TS algorithms without calibration perform worse due to the ILS problem, indicating the existence of an evaluation pitfall in the literature. The proposed calibration technique consistently improves the classification performance for TS approaches, demonstrating the effec- tiveness of the proposed calibration technique. 2) There is no algorithm that can win in every case of the dataset and evaluation metric. Besides, some early algorithms can already achieve satisfac- tory classification performance. 3) Our proposed validation metrics are effective in hyperparameter selection. However, the effectiveness may also depend on the test metric. For example, we can ob- serve from Table that the model selected using PAUC can achieve better performance than using OA when the test metric is the AUC score.
9

---

Published as a conference paper at ICLR 2026
Table 4: Test results itd) of accuracy, AUC, and F1 score for each algorithm on Ima- geNette (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric shown in bold.
Text metri
Test ACC
AUC
Test
PAUC
PA
UA
PUBN
75.30110.55
75.57 21161
77.39100
K3.9720.64
04.5500.30
K3 05.00
76.39-0311
75:9822122
70.42g0000
PAN
03.31
64.72 1.699
64.37 12.8
65.281 1.32
10.38.11.92
69.191261
66.19-0.27
58.68155000
63.0512.55
CVIR
76.6001.744
77.87=0.0
7.429-0447
05.440026
86. 0..35
87.22+0.49
78.430.45
7.667
79939
PIMIX-E
60142:44.27
01.61
71.512247
67.11=154 1.54
46.19:110
FIMIX
74.17:00.00
15.40
83.0810.33
76.85
LBF
+0.94
83.0
83.39
74.99
Count Loss
79.27:00.28
21.22
74.43-0.66
0.52
K2 0..6
K2.05
76 0.228
76.11 0::11=0.30
76. 06.46:0445
Robut PU
1..58:1.19
72.78 72.78.:004 4:443
90.194081
50:5710:71
51.6920.38
73:75
69.94 69.94
74.06 105
HolliticPUU
56. 12:993
54, 74470+22.100 2.16
59. 19-553
61 46
9983+1.0
62.22+0.09
64.75=1 1.38
60.81 =1 16
08.3
PUl
64.65:20.99
67.63-0.53
72.74+1488
69:33-1007
06.66:20771
CLWS
76.99+0.21
8581+0055
96.55--0.37
85.77:0.29
78:40=0.13
78.65+0.119
uPU
00.42:2.82
66.4221:008
16.2911000
67.494 3.9
02.2440.86
72M2240.2
67.95 67.95=0.6
67 17:50::57
66.46:1
uPthe
12.57=11.56
73.20+1000 05
75.07
79. 29
81. 76+09222
82.22+0887
72:00=133 1.44
69 52:22.22
74 58
69.33:0.52
0..5
05
naPL-c
74,4220775
13.05
74.17=1007
92.192039
N1.77=
15:24:1255
mPCC-GA
19+1.31
75.23+1000
01.37+0.66
0R
63.37=00.99
7544-049
74
74 850000
mPPL-GAv
12.27=1.25
11.85
74.62-0.11
79.16 1.52
SL 0166)+0.47
79.72+1.52
75.8600.11
71 02:00 68
74
PUSH
1..2
72.57 20.76
75.3029..6
71. 11.4
72.6520.76
75.35
75. 75.76=0435
74.73-0774
06
PUSB-
72.98+1.11
73,60+0.38
74.95+0.52
73.00+1.10
73 64+00.3
74 74.8++0.50
7340-0.94
71 71-5500448
74 70+00.37
VPU
7.542=1.87
73.26-00.73
75.20=0.72
24.24=19
VPU-E
76.30=0.79
71.75
71.36=1000
44
03.00200.79
7% 38:32=0044
077
Dit-PU
63.97+1.03
67.54+00.0
70.64+0.26
74.62-00.66
69 0.11
01-02-0-33
70.22-075
Dis-Pile
60.43-4.37
68.02:11.27
67,29:1774
37.51
74 14.10:1.22
71.97+2.69
67.65-0299
69 00-0611111 1.11
65. 65.39 3.35
(a) Accuracy w/ PA
(b) PA
Accuracy w/ PAUC
(d) F1 PAUC
(e) Accuracy w/ OA
(1 FI O/
Figure 4: Overall performance w.r.t. accuracy and the F1 score across all datasets. Hyperparameters were tuned using PA, PAUC and OA, respectively; bar colors indicate means.
6 CONCLUSION
In this paper, we conducted a comprehensive empirical study of PU learning algorithms. To the best of our knowledge, we proposed the first PU learning benchmark to systematically compare different PU learning algorithms in a unified framework. We investigated model selection criteria to facilitate realistic evaluation of PU learning algorithms. We also identified the ILS problem for the one- sample setting of PU learning and proposed calibration approach ensure fair comparisons of different families of PU learning algorithms. We hope that our framework can facilitate accessible, realistic, and fair evaluation of PU learning algorithms in the future. A limitation of our work is that we use relatively small benchmark datasets following previous work. In the future, it is also promising to investigate the performance of different algorithms on collected large-scale PU benchmark datasets.
ACKNOWLEDGMENTS
WW and DDW were supported by the Junior Research Associate (JRA) program of RIKEN. MS supported by JST ASPIRE Grant Number JPMJAP2405.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109:719-760, 2020.
Jessa Bekker, Pieter Robberechts, and Jesse Davis. Beyond the selected completely at random as- sumption for learning from positive and unlabeled data. In Machine Learning and Knowledge Discovery in Databases- European Conference, ECML PKDD 2019, Proceedings, Part 11, vol- ume 11907 of Lecture Notes in Computer Science, pp. 71 71-85, 2019.
Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learn- ing from corrupted labels. In Proceedings of the 36th International Conference Machine Learning, pp. 961-970, 2019.
Hao Chen, Jindong Wang, Lei Feng, Xiang Li, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, and Bhiksha Raj. A general framework for learning from weak supervision. Proceedings of the s1 International Conference Machine Learning, pp. 7462-7485, 2024.
Hui Chen, Fangqing Liu, Yin Wang. Liyue Zhao, and Hao Wu. A variational approach for learning from positive and unlabeled data. In Advances in Neural Information Processing Systems 33, pp. 14844-14854, 2020a.
Jiawei Chen, Hande Dong. Xiang Wang. Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions Information Systems, 41(3):1-39, 2023.
Qiuyi Chen, Haiyang Zhang, Leqi Zhang, Changchun Li, Jia Wang, and Wei Wang. PU-Bench: A unified benchmark for rigorous and reproducible PU learning. In Proceedings of the 14th International Conference Learning Representations, 2026.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang ang Wang. Self-PU: Self boosted and calibrated unlabeeed training. In Proceedings of the 37th International Conference on Machine Learning, pp. 1510-1519, 2020b.
Olivier Coudray, Christine Keribin, Pascal Massart, and Patrick Pamphile. Risk bounds for positive- unlabeled learning under the selected random assumption. Journal of Machine Learning Re- search, 24(107):1-31, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2009.
Marthinus du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In Proceedings of the 32nd International Conference Machine Learning, pp. 1386-1394, 2015.
Marthinus C. du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. Advances in Neural Information Processing Systems 27, pp. 703-711, 2014.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 213-220, 2008
Saurabh Garg, Yifan Wu, Alexander J. Smola, Sivaraman Balakrishnan, and Zachary C. Lipton. Mixture proportion estimation and PU learning: A modern approach. In Advances in Neural Information Processing Systems 34, pp. 8532-8544, 2021.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Proceedings of the 31st Conference On Learning Theory, pp. 297-299, 2018.
Chen Gong, Qizhou Wang, Tongliang Liu, Bo Han, Jane You. Jian Yang, and Dacheng Tao. Instance- -eeeedeed positive and unlabeled learning with labeling bias estimation. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 44(8):416344177, 2022.

---

Published as a conference paper at ICLR 2026
Ishaan Guirajani and David Lopez-Pazz In search of lost domain generalization. In Proceedings of the 9th International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Cho-Jui Hsich, Nagarajan Natarajan, and Inderjit Dhillon. PU learning for matrix completion. In Proceedings of the 32nd International Conference on Machine Learning, pp. 2445-2453, 2015.
Yu-Guan Hsich, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and biased negative data. In Proceedings of the 36th International Conference on Machine Learning, pp. 0920,2829, 2019.
Wenpeng Hu, Ran Le. Bing Liu, Feng Ji, Jinwen Ma, Dongyan Zhao, and Rui Yan. Predictive ad- versarial learning from positive and unlabeled data. In Proceedings of the 35th AAAI Conference Artificial Intelligence, pp. 7806-7814, 2021.
Yangbangyan Jiang, Qianqian Xu, Yunrui Zhao, Zhiyong Yang, Peisong Wen, Xiaochun Cao, and Qingming Huang. Positive-un unlabeled learning with label distribution alignment. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 455(2): 15363, 2023.
Hyunjun Ju, Dongha Lee, Junyoung Hwang. Junghyun Namkung, and Hwanjo Yu. Pumad: Pu metric learning for anomaly detection. Information Sciences, 523:167-183, 2020.
Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with selection bias. In Proceedings of the 7th International Conference on Learning Representations, 2019.
Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI machine learning repository, 2023.
Ryuichi Kiryo, Gang Niu, Marthinus C. du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In Advances Neural Information Processing Systems 30, pp. 1674-1684 1684, 2017.
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Changchun Li, Ximing Li, Lei Feng, and Jihong Ouyang. Who is your right mixup partner in positive and unlabeled learning. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Changchun Li. Yuanchao Dai, Lei Feng, Ximing Li, Bing Wang, and Jihong Ouyang. Positive and unlabeled learning with controlled probability boundary fence. In Proceedings of the 41st International Conference on Machine Learning, pp. 27641-27652, 2024.
Lin Long, Haobo Wang, Zhijie Jiang, Lei Feng, Chang Yao, Gang Chen, and Junbo Zhao. Positive- unlabeled learning by latent group-aware meta disambiguation. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13138-23144 2024.
Yuren Mao, Yu Hao, Xin Cao, Yunjun Gao, Chang Yao, and Xuemin Lin. Boosting GNN-based link prediction via PU-AUC optimization. JEEE Transactions on Knowledge and Data Engineering, 37(4):635-1649, 2025.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, pp. 807-814, 2010.
Gang Niu, Marthinus C. du Plessis, Tomoya Sakai, Yao Ma. and Masashi Sugiyama. Theoretical comparisons of positive-unlabeler learning against positive-negative learning. In Advances in Neural Information Processing Systems 29, pp. 1199-1207, 2016.
12

---

Published as a conference paper at ICLR 2026
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, 2019.
Harish G. Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In Proceedings of the 33rd International Conference Machine Learning. pp. 2052- 2060, 2016.
Sebastian Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv: 1811.2881 2018.
Tomoya Sakai, Marthinus Christoffel du Plessis, Gang Niu, and Masashi Sugiyama. Semi- supervised classification based on classification from positive and unlabeled data. In Proceedings of the 34th International Conference on Machine Learning, pp. 2998-3006, 2017.
Shai Shalev-Shae Shaale-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algo- rithms. Cambridge University Press, Cambridge, UK, 2014.
Vinay Shukla, Zhe Zeng, Kareem Ahmed, and Guy Van den Broeck. A unified approach to count- based weakly supervised learning. In NeurlPS, 2023.
Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, Tomoya Sakai, and Gang Niu. Machine learning from weak supervision: An empirical risk minimization approach. MIT Press, 2022.
Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Yuuki Yamanaka, and Tomoya Yamashita. Positive-unlabeled diffusion models for preventing sensitive data generation. In Proceedings of the 13th International Conference on Learning Representations, 2025.
Pawet Teisseyre, Timo Martens, Jessa Bekker, and Jesse Davis. Learning from biased positive- unlabeled data via threshold calibration. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, pp. 2314-2322, 2025.
Yuchuan Tian, Hanting Chen, Xutao Wang. Zheyuan Bai, Qinghua ZHANG, Ruifeng Li, Chao Xu, and Yunhe Wang. Multiscale positivc-unlabeled detection of Al-generated texts. In Proceedings of the 12th International Conference on Learning Representations, 2024.
Vladimir Vapnik. Statistical learning theory. John Wiley & Sons, 2, 1998.
Wei Wang, Dong-Dong Wu, Jindong Wang, Gang Niu, Min-Ling Zhang, and Masashi Sugiyama. Realistic evaluation of deep partial-label learning algorithms. In Proceedings of the 13th Interna- tional Conference on Learning Representations, 2025.
Xinrui Wang, Wenhai Wan, Chuanxing Geng, Shao-Yure Yuan Li, and Songcan Chen. Beyond myopia: Learning from positive and unlabeled data through holistic predictive trends. In Advances in Neural Information Processing Systems 36, 2023a.
Xutao Wang, Hanting Chen, Tianyu Guo, and Yunhe Wang. PUc: Biased positive- unlabeled learning enhancement by causal inference. In Advances in Neural Information Processing Systems 36, pp. 19783-19798, 2023b.
Tong Wei, Hai Wang, Weiwei Tu, and Yufeng Li. Robust model selection for positive and unlabeled learning with constraints. Science China Information Sciences, 65(11):212101, 2022.
Yuhao Wu, Jiangchao Yao, Bo Han, Lina Yao, and Tongliang Liu. Unraveling the impact of het- crophilic structures on graph positive-unlabeled learning. In Proceedings of the 41st International Conference on Machine Learning, pp. 53928-53943, 2024.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Rethinking class-prior estimation for positive-unlabeled learning In Proceedings of the 10th International Conference Learning Representations, 2022.
Jinfeng Yi, Cho-Jui Hsich, Kush R. Varshney, Lijun Zhang, and Yao Li. Scalable demand-and recommendation. In Advances in Neural Information Processing Systems 30, pp. 2412-2421, 2017.
13

---

Published as a conference paper at ICLR 2026
Hang Yin, Liyao Xiang, Dong Ding, Yuheng He, Yihan Wu, Pengzhi Chu, Xinbing Wang, and Chenghu Zhou. Lambda: Learning matchable prior for entity alignment with unlabeled dangling cases. In Advances in Neural Information Processing Systems, volume 37, pp. 78964-78995, 2024.
Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-PU; resitive-unlabele learning from label distribution perspective. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14441-14450, 2022.
Yao Zhou, Jianpeng Xu, Jun Wu, Zeinab Taghavi, Evren Korpeoglu, Kannan Achan, and Jingrui He. Pure: Positive-unlabeled recommendation with generative adversarial network. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 2009-4499220 2409, 2021.
Yilun Zhu, Aaron Fjeldsted, Darren Holland, George Landon, Azaree Lintereur, and Clayton Scott. Mixture proportion estimation beyond irreducibility. In Proceedings of the 40th International Conference on Machine Learning, pp. 42962 42982, 2023a.
Zhangchi Zhu, Lu Wang. Pu Zhao, Chao Du Wei Zhang, Hang Dong, Bo Qiao, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Robust positive-unlabeled learning via noise negative sample self-correction. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3663-3673, 2023b.
14

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLMs were used solely for correcting grammatical and spelling errors.
A COMPARISON WITH CONCURRENT WORK
A concurrent work (Chen et al., 2026) also proposed a PU learning benchmark. The authors primar- ily focused on establishing fair comparison and evaluation pipeline for PU learning algorithms. They noted the differences between one-sample and two-sample settings, yet they did not analyze the ILS problem or propose solutions. Furthermore, they assumed that the validation set had true labels, which is unrealistic, discussed in the paper.
B PROOFS
B.1 PROOF OF PROPOSITION
ACC()) p(payy=--) [[((() > 0)]+ (1 (00)p(1maappapppp000000 < 0)]
= (1 [I(((z) > 0)] B,(,) [I(f(z) 0)] TBp(aly=+1) [f((z) < 0)]
-Tpp(sls=+) (f((x) > B [I(f(z) < 0)] TBp((=/==+) [1 ((fxx) > 0)]
=27Bp(elp=+1) (f(x) > 0)] + Ep(e) [1] (f) < 0)] - π
-E[PA(f)] - n.
Here, the last equation is obtained since Du p(z)) for the TS setting and Dp Uuu itd p(æ)) for the OS setting. Therefore, for two classifiers f1 and f2 that satisfy E [PA(F))] < TE [PA( we have ACC(fi) < ACC(f2). The proof is complete.
B.2 PROOF OF PROPOSITION 2
For the TS setting,
T(f(x) > f(a') 1 2 (f(x)) s(a'))
AUC(J) Eptzlyy- E. Ep(er/s'--1
[(f(x) f(x') ) (f(x) s(x7))
pp Eptrr
π
Ep( B Bp(e [f(x)) [(a')) _ (f(a)) - [(x'))
π
[[((60) > ((x()) 1 ! [f(x) s(a\))
E E Ep(ey)
π
2 2T
E [PAUC(()] 2- 2x
For the OS setting,
AUC(F) = =lEp(aly- Bp(a'1
(f(xz) f(x')) z11 (f(x) f(x'))
1
1
1 (f(x) = s(a'))
E Ep - B )((r)
[(f(z) f(x')) 2
π
E +1) Ep(e [ff(x)
f(x')) _ (f(x) = f(30))
π
[[((E)) f(x')) I((xx) s(a\1) -
Bp Ep(e))
- 2-22 27
π
E [[PAUC(O]]
2- 2
15

---

Published as a conference paper at ICLR 2026
Therefore, under both OS and TS settings, for two classifiers fi and f2 that satisfy IE [PAUC(fi) [PAUC(f2)], we have AUC(f1) < A
B.3 BIAS OF THE RISK ESTIMATOR
Under the OS setting, we have
E [
R(f) =m)( -1]] [[(f(z), -1]]
=( x) (Bp(zly-+1) [[]_f=))))(((()))))))))))))))))) - pp(zly--1) [((f(x)),-1))
which is not equal O. Therefore, means that the bias of the risk estimator always exists. Then, the minimizers of E [ and R(() are not the same.
B.4 PROOF OF THEOREM 1
First, we have
((x) =Tp( = +1) + (1 7)p(zly = -1)
c)- p( y +1) + 1-
p(aelyy = -1).
1 CTT
cx
Therefore, we have
1
(1 - c)
p(zly= =
p(x)
p(zly +1).
1 - π
1 π
Then,
R(f) = +1) [10p:/))_))){---mmmmm/e111/1)111 (1) (()
+ ([((t)
-TBp(elu-+1)
=+Bp(=ly=+1)
+ (- ((((()) -1]],
which concludes the proof.
B.5 PROOF OF THEOREM 2
Definition (Rademacher complexity). Let XP {x), .......000 :np) denote nip i.i.d. random variables drawn from density p(æly = +1). Let PU { 11, ... Tap-net denote nu i.i.d. random variables drawn from density 7(x)). Let (s X R denote a class of measurable functions, OP (a), (2 In and ou (o 11 +2, ... Onp-nuu) denote Rademacher variables taking values from {+1,-1) uniformly. Then, the (expected) Rademacher complexities J are defined as
1 BP of f()))
Rnp (F) Exp Eop
sup
fEF np
np t
R (F)
Exx Eov
sup feF
Σ i-
azf(
nU
Lemma 1. For any we have the following inequality with probability at least 8:
R(() R(F) x2(2 c) R. (F) 2(1 Cx)) C.)..R (F)
sup
fer
7(2 C)
CT)
In 2/0 nu
+
T&P
2
Proof. First, we give the upper bound for the one-side uniform deviation supfer
feF (R(f) - R(f)).
up When an instance in XP is replaced by another instance, the value of sup (R(s) R()))
16

---

Published as a conference paper at ICLR 2026
changes at most (2 c)Ce//p;; when instance in Mny is replaced by another instance, the value of SuP (R(f) R(f)) changes most (1 Ct/yy Therefore, according to McDiarmid's inequality, have the following inequality with probability least 8/2:
(2 - C3
(1- (1-c))²C
In 2/6
sup (R(S) R()) <E feF
sup feF
(R()) R())
+
+
np
nu
2
( c)
(1- (T-
In 2/6
E
sup fef
[R(f) R(f))
+
7p
nu
2
Then, by symmetrization (Vapnik, 1998), it is routine work to have
(R()) R())
E
sup fer
< 2(2 e)mRn. (l F) 2(1 cT) CT))R (6 F).
According to Talagrand's contraction lemma (Shalev-Shwarr) & -Ben-David, 2014), we have
Rmp (& F) & LeR Ryn (lo F) < LeH (F).
By combining the above inequalities, we have the following inequality with probability at least 8/2:
sup (R(f) - R(f) <2 Ln (F) + 2(1 CT) (F)
feF
T22 - c)C (1 C
In 2/2/8
npp
Vnu
2
In similar way, we have the following inequality with probability at least 5/2:
sup (R(f) - R(()) <2(2-c)TL,R (F) + 2(1 cT) L (F)
fer
(T(2-
(1 (1-
In 2/8
TTP
V
Therefore, have the following inequality with probability least 1 8:
sup
[R() R(O <2(2 L. (F) 2(1 oτ)
L (F)
fEF
+ T( C) + (1 (1=
In 2/0 2
np
Vnv
The proof is complete.
Then, give the proof of Theorem 2.
Proof of Theorem 2.
R(F) - RR(*) -R())) R((F) + R((F) R( + R(**) R(($*)
<R()) R(()) + R(()) R( + R(S*) -(()*)
<2 R() B(N).
feF
By Lemma 1, the proof is complete.
17

---

Published as a conference paper at ICLR 2026
B.6 DERIVATION OF EQUIVALENCE OF RISK ESTIMATORS
R(f)
Rp tay
(1) (c- (c- -1))
Σ ((f(x)),-1)
np
nu
impp+1
np ap+ny
Σ
[(((x)))++)++
T
(mptt
{ff(zi), +
Σ {f((z)), -1)
ni
+ + nu
np
+ Yu i-npt1
ap+au
Σ
- -1))
{(((z;), -1),
(10)
Au
where the second equation uses the estimation np hp/ nu).
C MORE EXPERIMENTAL DETAILS
C.I MORE DETAILS OF BENCHMARK DATASETS
Table 5 summarizes their key characteristics, including the number examples, feature dimension- ality, positive class configurations, and task domains. For all datasets, vary the positive rate in (10%, 20%, 30%, 40%, 50% For the benchmark experiments in Section 5, we used the positive rate 30%.
Table Summary of datasets used in this PU learning benchmark.
Dataset
Examples
Features
Positive Classes (Case 1)
Positive Classes (Case 2)
Task Domain
CIFAR-I0
20,000
3,072
[0,12288.9]
(2.3.5.7.9)
Image classification
ImtgeNette
6,000
12,288
(0.12.899)
(2.3.5,7.9)
Image classification
[4.7.9.58]
(1.64498)
Digit recognition
Letter
13,000
16
(BW.IR..ONSJKKKIII
(DTAYQQ..LLLW.....
Character recognition
C.2 DESCRIPTIONS OF ALGORITTMM
uPU (du Plessis et al., 2015): An unbiased risk estimator that is convex when the loss function satisfies certain linear-odd conditions.
mnPU (Kiryo et al., 9017): A non-negative risk estimator that alleviates the overfitting issue in PU learning.
nnPU-GA (Kiryo et al., 2017): A gradient ascent formulation of nnPU.
PUSB (Kato et al., 2019): A method that accounts for selection bias in the labeling process. PUbN (Hsieh et al., 2019): A framework that incorporates biased negative data into empirical risk minimization.
VPU (Chen et al., 2020a): A variational approach that directly evaluates the modeling error of
Bayesian classifier from data.
PAN (Hu al., 2021): A predictive adversarial network built upon the generative adversarial network framework.
CVIR (Garg al., 2021): A mixture-proportion estimation method combining best bin estimation and conditional Value Ignoring Risk.
Dist-PU (Zhao et al., 2022): method that enforces consistency between predicted and ground- truth label distributions.
MIX-E (Li et al., 2022): A mixup-hased method that pairs marginal pseudo-negative instances
with boundary-near positive instances, with early-learing regularization.
M-MIXXC (Li et al., 2022): A mixup-based method that pairs marginal pseudo-negative instances positive instances, with pseudo-negative correction. LBE (Gong et al., 2022): An instance- deeendend PU algorithm that jointly estimates labeling bias and learns the classifier.
with boundary-
Count Loss (Shukla et al., 2023): unified approach introducing a count- count-based loss penalizing deviations from arithmetic label-count constraints.
18

---

Published as a conference paper at ICLR 2026
Robust-PU (Zhu et al., 2023b): A reweighted learning framework that dynamically adjusts sample
weights based on training progress and sample hardness.
Holistic-PU (Wang al., 2023a): A holistic method interpreting prediction scores temporal
point process.
PUe (Wang et al., 2023b): A causality- method that reconstructs the loss via normalized propensity scores and inverse probability weighting.
GLWS (Chen al., 2024): general weak-supervision framework formulated Expectation- Maximization, accommodating PU data as one supervision source.
C.3 IMPLEMENTATION DETAILS
All algorithms ducted on a single NVIDIA Tesla V100 GPU. We used the SGD optimizer and trained for 20,000 iterations across all datasets. Model performance the validation and test sets was recorded every 100 iterations. For each dataset, we generated three random data splits. For each split, 10 random hyperparameter configurations were sampled from predefined pool. Table provides the details of the hyperparameter configurations used for all algorithms.
implemented in PyTorch (Paszke al., 2019), and all experiments
Table 6: Hyperparameters, their default values, and distributions for random search.
Condition
Parameter
Default Value
Random Distribution
learning batch size momentum
0.001
1niiformmm4..5 (-465,
ResNet
64
(5,)
0.9
0.9
learning rate batch size momentum
0.001
10
10 -.5,
MLP
128
20ikee((477)
0.9
0.9
nnPU
tolerance threshold
0.0
PUBN
importance of unlabeled data
0.5
RundonChhoie((033070..0))
PAN
balance factor of the KL-divergences
0.0001
0.0001
predictive threshold
0.85
0.85
the candidate mixup pool
96
96
weight of positive loss
P'MIX-E
weight of the unlaheled loss
1
weight of the entropy loss
0.5
0.5
weight of the early-lcarning regularization
5
5
predictive score threshold
0.8
0.8
size of the candidate mixup pool
96
96
mixup coefficient
1.0
1.0
P'MIX-C
weight of the positive loss
1
weight of unlabeled loss
weight of entropy loss
0.1
0.1
LBE
warm up iteration
2000
2000
warm up iteration
2000
2000
training scheduler
linear
linear
temperature in the logistic loss
1
RandomChoice((11I]
Robust-PU
initial threshold final threshold growing step
0.1
RandomChiicer(011.000
2
RandomChoiiee(11.2)
10
RandomChoice(((110)
Holistic-PU
warm up iteration
2000
2000
19

---

Published as a conference paper at ICLR 2026
D DETAILS OF EXPERIMENTAL RESULTS
Tables to 18 report detailed experimental results in terms of different metrics on CIFAR-10, Im- ageNette, Letter, and USPS, and the hyperparameters are determined with PA, PAUC, and OA, respectively.
AUC
PUW
CVIR
PIMEX-C
LBC
Count Loes
Rohest-P/
GLWS
UPU
uPEi
emPl
miPL-p
aaPU-GA
ma mPPU-GA --
PUSO
PUSE-
vithe
Diss-ti
Dis
(a) PA
PUBN
CVIR
FIMIX-C
LIE
Caunt Loss
Robust-Pi
GLWS
uPV/
uPtice
aPPi
millle
anPU-GA
PUSE
PUSB-c
vptioe
Dint
Dist PUle
(h) PAUC
AUC
PLBb
CVIR
PIMIXCC
LBE
Cound Lues
Robust-Pii
GLWS
upU
uPLie
mpti
RaP
saPi-GA
mPI-GAA
PUSh
PUSHi
VPLte
Disi-Pi
Diat Pllc
(c) OA
Figure 5: Overall performance w.r.t. the AUC score different algorithms across all datasets. Hy- perparameters tuned using PA, PAUC and OA, respectively; bar colors indicate means.
20

---

Published as a conference paper at ICLR 2026
Precision
12
PUBN
CYR
PIMIX-C
LBE
Count Loss
Robasl Pt
GLWS
urt:
urtio
murt
mittle
maPl-GA
RaPL-GAAe
PUsB
PUSBie
Vptbe
DistPU
Dis-Ptlo
(a) PA
Precision
PULN
CVIR
PIMEXCC
LIEE
Count Lost
Rotser-PUi
GLMS
upU
uPtie
naPU
hePlie
MMPE-GA
RPPU-GA
PUSB
PUSB
1Pt-e
DistPU
Dist-Pi-c
(b) PAUC
Precision
PUBN
CVR
PSMIX-C
LBF
Count Loss
RohadtPPi
GLWS
uPtlc
map
miPtle
miPL-GA
naPt-GG Aue
PUsh
PUSB
VPEle
Dis-PU
Dise
(c) OA
Figure 6: Overall performance w.r.t. precision of different algorithms across all datasets. Hyperpa- rameters were tuned using PA, PAUC and OA, respectively; bar colors indicate means.
Table 7: Test results (mean of precision and recall for each algorithm CIFAR-10 (Case 1) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, "- indicates using the proposed calibration technique in Algorithm
metric
Precision
Val metric
PAUC
OA
PA
PAUC
PUBN 05.5
86.97-10.18
89.46:00.5
87.71 1.01
85.25
84.64:10.24
73.33
1.75
CVIR
82.12-+1.38
78.27++ 1.03
86.30-0.93
90.72-10.42
92.42+
0-.74
P3MIX-E
68.95-+6.22
50.00+0.00
82.77=5.96
90.99-11.77
100.00=0.00
67.19-t17.74
P3MIX-C
86.03-1-091
84.91 1.000
86.27-10.66
86.86:10.24
06.47
87.19
LBE
79.00++1.37
66.06+ 1.22
88.64=0.66
89.31
07.45+0.33
80.41 00.41
Count Loss
75.81:00.00
7..78+1.63
79.88=0.66
90.73-0..11
1..5
PU
84.19111.05
1.87
88.23-10.69
87.73:11221
80.72:12.99
82.8910.226
Holistic
50.00
78.03-10.77
PUe
74.23++1.27
12
85.70=22.17
85 85.52-#0.48
76,39-+-272
73.49++1..7
GLWS
79.61400.5
73.12422.1
82.86=00885
92.70=00.34
25.53:11.05
89.99
uPU
78.144+1..0
78.06++7.35
88 88.69-00.56
84.29-00.41
80. 13:4-443
73.44+00.66
0..5
77.69
nnPt
77.1740.27
68.25
78.73+ 1.19
91 0.30
95.62+40.62
88.90 93.99-.. 1.44
unPU-e
83.72+0.488
87.75+0.73
86.66=0.46
88 28-00-0
83,76+0.68
05.95
anPU-GA
81.92-t1.66
82.2910.38
1.68121162
88.18
87.122
82.54::0.65
85.33.10.90
89.78 1.03
0.725
1.15
10.75
82,19 0..41
PUSSB
76.20-1-1.4
78.134 1.39
0..64-1-.98
0.-1
90.41-10.19
90.44 0.20
PUSB-c
8644-+0.03
89.07 1.32
87.87=0.11
05.76+0.88
79.38+ 1.23
84, 04.66--0.25
90.61
13.15
79,93
VPU-c
84.974-165
77.43+2.41
89.08=0.15
88 38.67-0.88
93.37-00.83
85.82+0061
Dist-PU
76.3440.77
72.78+0.89
84.75=0..15
91.79=0.56
93.81+0.86
81.86++11
Disi-PU-e
84.07-11.13
88.2211.006
90.49:00.44
91.58.0999
86.67:12.22
86.02
21

---

Published as a conference paper at ICLR 2026
Recall
50
PUM
CVIR
P.
LoE
Coust Leas
Rohut-PP
GL
uPifo
maP!
miPiie
IPPL-GA
mitt-GA-
PÚSB
PUSSBo
VPi-ee
Dist-PU
Dis
(a)
100
Recall
PUNN
CVIR
PSMDXC
LBE
Count Lose
Robws-PU
GLWS
WPU
uPt
mPP
noPLle
naPC
HsPti-AA
PUSB
PUSB
VPL/o
Det-PU
Dist-PU
(b) PAUC
Recall
PUBN
CVRR
PIMEX-C
LBE
Count Los
RobessPUi
GLWS
WPU
uPtio
mPP
noPL
nePE GAPE-GA
IPPti-GA-
PUSB
PUSB
VPL/o
Det-PU
Dist-P
(c) OA
Figure 7: Overall performance w.r.t recall different algorithms across all datasets. Hyperparam- eters were tuned using PA, PAUC and OA, respectively; bar colors indicate means.
Table 8: Test results (sea of precision and recall for each algorithm CIFAR-10 (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, c- indicates using the proposed calibration technique in Algorithm 1.
Test metric
Precision
Recall
metric
PAUC
OA
PA
PAUC
OA
PUBN
73.2111441
1.72
80.1411.00 1.05
89.4210.70
78.25:3.04
79.75
57.23.11877
59.50
94.70
CVIR
73.134+1.91
76.32+ 1.46
4-090
90.57++0.80
85.61+ 1.27
85.81-4006
P3MIX-E
55.911+3.28
13.63
56.0443.39
96.22+199
66.67-27.22
96.07-2.117
P3MIX-C
74.10::1.68
74.09=2.13
75.63
16.67
85.03:1.76
84.7120.96
LBE
66.21.11.66
58.31 12.11
75.81
92.61.1.25
97.81
76.30.1..2
Count Loss
69.39:10.70
18.20-0.18
73.73--0.09
0..54
83.13-11.10
23.48-12.12
Robust-PU PU
75.32:1..111
X2.16:1:1 1.82
79.06
16.25:1:.51
72.97:1199
80,59
Holistic.
67.62.10.4
98.93-10.21
PUe
63.65-+0.22
62.69-2.220
71.16 1.47
86.71+00.2
88.03-+22.08
71,53+3.844
GLWS
71.86+1.10
69.66=
77.16+0.93
91.35+1.11
93.40=0.68
84.11=00.33
uPU
62.03-+-1.56
65. 14 1.52
69.80++0.66
84.79-42.37
82.83-42.57
72.40+3.72
uPU-c
72.31++0.26
82.55=0.255
80.15
88.234.026
74.2770.66
71.23=2.25
nnPU
68.391111.3
$7.41
74.03-10.92
91.01:1.511
98.65:10.40
85.19:0..73
73.1011.44
nnPU-GA
71.42::1.13
72.38=0.59
78.98
88.75
86.52-22.59
77.65
74.94.10.69
80.00 1.79
79.37
84.28
75.86
78.70.10.96
PUSSB
69.38-1-0.61
76.30 1.64
74.79 0.66
92.21-4-011
84.05-4-2.22
1.96
PUSB-c
76.45+0.77
79.54= 1.37
78.93+00.69
84.07+00.83
75.54++1.16
79.71
92.66=1..77
82.10-00.2
69.74+11.6
91.43
VPU-c
72.30:10.58
72.93 1.34
00.65-1043
89.8710.48
90.00:10.44
83.71 1.100
71.23
71.27-1..24
83.64
PUst-PU-e
69.46:144449
72.97-28.11
75.2412254
83.36110.70
78.05:13.74
7..22
22

---

Published as a conference paper at ICLR 2026
Table 9: Test results (tt) of precision and recall for each algorithm on ImageNette (Case 1) under different model selection criteria. The best performance w.r.t each validation metric is shown in bold. Here, "-c" indicates using the proposed calibration technique in Algorithm 1.
Test metric
Precision
Recall
Val metric
PA
PAUC
OA
PA
PAUC
QA
PUbN
70.84=0.32
7..57
L7
85.89+0.85
76.09
81.90+2.84
PAN
50.00110.66
53.43:11440
60.22:13.11
94.12:2.52
38.73:23.64
36.64:12.60
75.11
1..2
86.91.11.9
85.13.1.114
P3MIX-E
71.37+2.700
13.35+13.37
71.98++2.64
81.92+144
66.67+27.22
8144+1..8
P3MIX-C
74.23=#1.92
74.56+2.46
78.59+2.188
87.47+0.80
87.3440.62
82.78+1.94
LRE
7..29::0.82
81.74+1.60
80.73-H-0.7
81.76+00.93
73.48443.85
75.86+22.11
Count Loss
69.02=1.155
71.41+2249
14.81+
89.37
86,22++1.82
83.77+2.66
Robust-PU
75.89:10.88
81.601:2.37
79.16100.86
77.07:11.33
Holistic PU
50.17.10.26
54.2013.82
52.95.10.29
93.12.11.81
84.66 10.32
30.48.125
70.74+40.59
78.49-4-2.87
71.33-43.07
GLWS
1.181
68.10
75.17++0.67
90.92
92.88+00554
84,51=0.74
uPU
1.03
60.87:44.80
70.38-1-0.55
87.51
89.47-444.6
80.15-+1.07
uPU-c
70.53+22449
85.36+2.000
77.92+
86.05
54,50= 12.20
76.69 1.98
60,22:1...8
76.1114:2.64
90.46.1
nnPU--
12.51=1.01
81,53+2..56
77,04++-0.57
84.20++0.2
69,52-44,15
77.77+22.26
maPU-GA
6.674:1060
76.68
79.13+2.26
09.37
81.95+3.81
79.63+2.888
maPU-GA-c
1.16
86.08 2.79
77.78:10.87
80,1011.46
54,9318.53
7..05
PUSB
65.46.10.55
72.72 2.112
10.63
94.26 0.58
86.17
87.3111.22
PUSB-c
73.08+0.577
1.288
75.79++1.41
02
16
80.96-+2.74
VPU
61.32-5.55
33.33+:21.22
7..12+77.64
19.47+17.51
0.07=0.06
34.73+6.800
75.32-40.83
1..88+1.56
81.674...5
Dist-PU
63.85=2.20
66.01+2.40
69.52+00.11
92,79+0044
89,14+1.38
86.31+1005
Dist-UU-
67.29:1138
4.54914.56
72.4211955
84.35
43.02211.88
76.21
Table 10: Test results (td) of precision and recall for each algorithm on ImageNette (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, c- indicates using the proposed calibration technique in Algorithm
Test metric
Precision
Recall
Val metric
PAUC
OA
PA
PAUC
OA
PUBN 1.
1.
1.74
82.94+
77.08+3.44
74.08
PAN
31.63+00224
70.75=3.83
24.11
93.47=11.6
53.01
61.43+3.03
72.31.1100
18
82.2711144
80.42.1.40
E
59.73-15.02
33.05 13.49
60.00-15.03
82.05-19..12
66.67-127.22
82.13-19.07
P3MIX-C
69.1940.86
71.42+0.87
11.54+1.30
86.45-40.93
84.04+0.35
83.78-+0.78
LBE
70.02.1..26
75.72.1..6 1.63
78.52 10555
1.934
72.71.14.32
71.91
Count Loss
68.13-40.40
69.07
70.39+0..8
86.66-+0.67
84.79++0.94
83.73-40.75
Robust-PU
70.52+2.00
71.77=2.80
78.01
77.58+11.8
64.58244.76
70.6942.37
Holistic-PU
54.06:1:1.09
13.98
58.9610.87
62.10.16.17
71.7011.118
58.891 1.87
80.82.1..80
GLWS
69.84-40.87
69.39-0.41
72,88 4-068
89.59-40.58
90.12-+0.64
85.49 1.26
uPU
57.42.12.33
65.47
66.40 1.64
84.31 1.394
71.1015.64
68. 6..14
uPU-e
72.56:1-2.89
7.046=1.02
75.39::0.26
73.42::3.51
62.211:3.91
13.83=1.50
98.56
noPlic
72.24-40.48
76.75-22.6
0-069
78.73-4.300
68.48-16.77
73.47 3.66
nnPU-GA
67.45++1.90
76.34-0.93
77.55+
85.99-+-2.22
72.88+4.666
15.56-+2.15
BnPU-GA-c
69.67422.08
08.6020.62
73.81 1.46
78.90 1.89
64.65=20.71
76.22=3.06
PUSB
65.52.12.07
68.76:00.0
72.14 1.68
90.15:11.550
SL.84:10.88
82.39
P
71.30.1..11
76.51 10.32
74.72 1.46
1.0922
67.73.10..1
74.87 1.76
VPU
66.96:4-2.1
55.05++22.55
78.42+
81.55-+-2.30
22.49 18:22
63 3.62
Dis
59.86-4-1.20
63. 11.30-0..69
65.65 1.35
84.36-42..94
83.38+ 1.52
77.27 1.64
Dist-PU-c
58.89+44.09
66.51=11.59
68.25+0.62
83.16+7.558
72.05+2.14
63.51=5.76
23

---

Published as a conference paper at ICLR 2026
Table 11: Test results (td) accuracy, AUC, and Fl score for each algorithm on Letter (Case 1) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, "-c" indicates using the proposed calibration technique in Algorithm
AUC
Test
Val metrie
IAA
PAUC
OA
PA
PALE
CA
PA
PAUC
OA
53. 1.9
19.70:11.38
24:24:11448
14.484
11.0
89,59
89443:1.70
19.57
04.20+0.55
55.92+0.51
63.40=0.366
63.040049
CVIR
83.35
82.6020.75
04.67=0558
Ko 0065
57.63200.99
87.78=0.00
85.16.11150
04
85.86-20.25
PIMIX-E
$1.80:1
49.62:00.57
61 61 41112:4.12
60. 70:00055 5.00
01.42:10.26
67.00:7.22
67.12:00.44
43.85-1725
42.89 1 17.40
PIMIX-C
8003++ 1.13
77.58+2.53
80.02+1 14
85.08 1.33
84 44445++11 14.45+11.62
64.50+0.66
82,46+0.83
80.56- 1.25
82.83+00.56
LBC
85.63.1.13
81.37 12.19
91.81
96..6.299
94.38 0.23
87.17
83.321
87.44.0..3
Loss
73.15:1199
Robust-PU
90.0220.67
1::33:221333
95.
0.332
-0.31
90.20:0.61
Holisai PU
85.90100.9
15.22.19.45
98.12
95. 95.72.1.1
90.7411 1.64
87.14
5.
98.17 1.02
PUL
70.50:00.24
81 111885111000
82 00:0000000 78
89.37 1.07
91. 01.42
90.88:00.00
81 81.54
K2.32 1.64
81.05
GLWS
85 05.87
80.33
8o 0.535
026
93. 03.022045
92.65:003
80.00-00775
K3.53 1.17
07.25200.54
74.9911119
19.75-40.63
79
85387 0.59
09.29
86.19:0071
79
02:62:0.40
77.6
u
91.2310.26
85.97 1:001
92.73 0.15
96.84 0.15
91.36.10.05
96.4010.188
0.114
8308
92.60
anPU
13400.46
79.53+1
85 60:20 31
94. 0.51
95. 44.00.44
94,49 0..4
86. 19
82 4622
85. 2:040
naPU-c
91 01.37
89.25 1.14
91 X2 14
96. 0.30
96, 3920009
9.33
91.85 20
HS 28:2282217 1.71
91 20:21
mPPU-GA
85 12+60.13
02.85+00.68
84 27-27
93. 93-77-0.4
7--07-000
91.5 56-0.61
91 08-18-044
85, 25.7440225
K2.86+110
84
moPLi-GA-e
94.72=0225
04.72-0223
N7.75=0.25
PUSB
01.42.0.85
03
41-077877
00.66+0.577
46-00.9
91
10.30-0.00
VPU
89.85-1.007
67.88:18.64
90.13
05.67
96.03:10.77
95.4411..57
$9,1919888
44. 24.13-20.00
99.86-19.67
VPU
9183110.54
90.28 10.98
92.15 10.52
96.33 102.1.0..3 0.38
97 96.06 10.26
96.96 0.30
91.95 1042
89.38 1.17
91,93 1.048
Dac. PU
77.0740.77
71.75
77 77.5520.77
81.95=110 1.77
82. 1.22
82.07 1.66
20.15:00445
7h 0..6
80.07 2:1400
67.05112.41
70.03:2.22
72.9622.78
25.75
2..7
72.6121.30
68.7812.64
72.81
Table 12: Test results std) of precision and recall for each algorithm on Letter (Case 1) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, c- indicates using the proposed calibration technique in Algorithm 1.
Test metric
Precision
Recall
Val metric
PA
PAUC
OA
PA
PAUC
QA
86.86114.01
16.3311664
96.00.10444
90.85:11.27
PAN 49.11
48.50
34.01.113.0
97.89 1.15
100.00 1..00
$6.40:1.3.56
75.82+0.92
74.75+00.3
77.28+00.52
97.1740.70
P3MIX-E
50.79=0.93
15.70+14.00
45.23=18.89
99.04=00744
06.80=27.10
18.44
P3MIX-C
13.49=1.22
71.3742..7
75.15110.90
94.00=11.10
92.77:0092
92.26
LBE
78.5811.20
14.44
85.17.1.79
97.89 1.229
93.97.1..87
90.14
Count Loss
69.56 1.08
64.89+1..9
72.33-t0.83
94.95-20.69
95.42:+
89.44
Po
1.06
90.32.10.77
92.60110.67
91.8412.47
Holistic PL
79.36 1.07
71.38.18.69
82.39.4-2.08
96.62.10046
96.74-1 1.50
94.99 1.21
PUe
73.33+0.21
78.59++0.74
80.47-4-0.23
91.82+0.32
86.90+4.226
62.62.5252
GLWS
7..56:1.32
12.32+1187
79.44:t0.61
97.60:0..27
98.9820.30
96.84:0048
nPC:
67.24+ 1.28
77.65-+2.009
74.98++0074
93.05+00.1
81 11.66-1.1
80.56+t 1.73
uPU-c
89.10=0.60
93.95-t2.97
92.00++:0.92
95.49::0.48
71.42= 10.67
93.26=:0.91
79.27:1028
11.97-11557
94.4410.61
08.70-10.99
nnPU-e
90.57=0.67
73.06+1..8
93.26+
93,18+0049
84.51+4.36
90.08+1.44 1.46
maPU-GA
31.01
31.22-13.50
80.8111.119
91.07:0.63
6..51
S8. 00.553
89.60
92.37.13.22
89.60 0.51
92.17.0.332
84.05
92.17.1.333
PUSB
79.01 1.09
84.95-00.84
84.96-+- 1.33
95.94++0.66
90.70+t2.01
SR 72.78-4-2 78-7864
PUSB-c
90.00=1.000
90.13+0..00
89,87--0.76
92.79+1..55
90.49-+0.97
92.77+
VPU
89.11
26.74
91,24:t:.24
90.42+1.99
35.30=17..99
18.61=1..31
58.6010.48
95.2110.75
90.69
95.57
Dist-PU
70.13=0.47
72.05=:1.16
71.50++:000
93,51
89.28=1.499
91.14+ 1.75
Dis-PU-c
63.03
68.8513..51
66.07:13.22
87.01
68.80 1.90
81.54-12.35
24

---

Published as a conference paper at ICLR 2026
Table 13: Test results (td) accuracy, AUC, and Fl score for each algorithm on Letter (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, "-c" indicates using the proposed calibration technique in Algorithm
Val metrie
IAA
PAUC
OA
PA
PALE
CA
PA
PAUC
OA
57.4740.558
88.98-1:45
09.9
95.841145
90.59=1100
39.40=0161
19.15
19.74:1102
49 80.88+0085
57.64
66.64+0.40
21.43+17.50
CVIR
84.3440.75
84.22
04.72=0766
88.63-11
88.1820.65
88.67=1.22
86.57=0161
85 08.38
86.38:2063
PIMIX-E
25.70:12.92
55.57
65.08- 3.00
71 41-43:4430
81. 2..4
71.19 3.66
68.50:0..77
$2.06
13.86
64.22 14299
PIMIX-C
2..00-2.04
80.20+2..16
63.32++2.22
89 56
90.09+2.55
88.23- 3.66
$3.80+ 16
83.35-
63.66+2.56
LBC
87.32100.00
90.82.1390
94.51 00111 0.18
94.43.0.11
95.3110.57
88.44.0.33
62.6111..51
Loss
0..35
82.20
83.30-10.17
Robust-PU
00.52
9.55
91,0010444
19.68-2111
Holisai PU
87.88:11.37
86.12 86.12.11.8
88.65
95.09-0062
95..6.10..9
95.40
88.79 0..99
87.58 1.22
89.49-10.5
PUL
70. 00.00
78.05
82 02
88.18 1.99
91. 01.92200119
90.65:00.40
00.73
81 04.20.200
GLWS
06.77
79.88 1.42
88. 0.67
930000011
94 0.333
98.75:0.14
87. 03
02.9720.76
89 20.51
13.2241U77
72.0542.17
84.7=
35.7220.35
77.801945
u
91.3210.57
99.72.100.99
92.13
96. 0.25
96.86.10.22
96.301055
91.71
89.05 10.91
92.14
onPU
34.6820.334
75.22:2.111
87 38
94 0.77
95. 30500.11
95, 06.3420..44
85 00
79 12.94:2
87.75-20.30
naPU-c
91 01.2740.45
90.5024111
91 08:00.11
96:21
96 06.210.12
97.09:0.322
91 91442044
90.4250.199
91 2126520222
mPPU-GA
63+6000
83434-140
86 15+0
93.84-0.3 0434
93. 79+0000
03.68-0000
86. 37
85-00- 1.00
86.42-00.41
moPLi-GA-e
96.65+0.445
01.55-00.37
00:20444
PUSB-e
01:3370.777
01-53-071
91 01734-0776
46.44
91.47-0.76
91,20+00
91.23-0.51
YPU
90.5:10.28
74.93.16.54
91.18:10.08
96. 96.26.1032
91.91110110
96.23-10.20
90.50119226
64.26-10..57 10.97
10.10
VPU
91.9510.38
89 10.05
92,85 0.229
96 0.26
96 40.0
96.89110.12
91.94 10.3
89 89.13110.35
92.74 0.27
Dac. PU
78.4240.89
77.52441.51
79 79.4220771
04
84 29200.73
15.17:20.34
81.73-20.54
79.3620.51
81. 81.10200.2
75.3511.22
71.58:20.65
76.87
$4.35.0.23
83.73:0043
78.1421003
77.5121.25
78,0011.36
Table 14: Test results std) of precision and recall for each algorithm on Letter (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, c- indicates using the proposed calibration technique in Algorithm 1.
Test metric
Precision
Recall
Val metric
PA
PAUC
OA
PA
PAUC
OA
17.45:1.99
91.83:1:.24
PAN
50.04
13.55-113.52
18.13-114.80 14,80
19.7410..2
66.67 12..22
26.22.1.21 21.41
78.35
79.00++0.54
78.58++1.09
96.73+0.14
95.95-40.56
P3MIX-E
53.00=1.722
12.22+12.66
66.29+44.29
9..6
67.78=23.66
63.34-+2.72
P3MIX-C
1.88
13.75
82.40:11.72
94.13:0.75
95.91:1.19
85.25:1..36
LBE
81.36
79.16.16.57
84.70.11008
96.89 10.35
89.6111.60
93.03.11.5 1.51
Count Loss
75.71
80.17++3.40
82.45-t2.334
92.84:11.32
87.33:13.35
84.47+1.68
Pu
93.22.10.86
92.1710.988
Holistic PL
84.34-12.63
81.09
85.42.1-2.12
94.05 1.39
95.65-1:.660
14.18-1 1.43
PUe
1.03
70.71+1881
82.69++0.51
89.04+
94.32+
25.
GLWS
78.8920.68
11.48:1.33
83.68:t0.94
98.2340.11
98.94220.40
95.21220388
nPC:
7..01+1.43
27.00--4.04
77.28-+0022
87.70+ 1.18
87.40-44.78
78.61+
uPU-c
87.67=0.95
94.08
92.36-t:0:46
96.19:00.92
84.81
91.72=0.20
79.5910.97
93.41:1
nnPU-e
91.19=1.12
91 21.85+0063
92.71+-0.88 80
91.80+ L.68
89.05=0.60
10.66+11.05
maPU-GA
12.09:1.25
18.07
86.18:11.35
91.33:1:221
93.44:10.65
86. 36.84:1 16.84
91.1010.34
93.731199
91.70.10.36
92.09 0.99
84.69
91 69 10.83
PUSB
33699-11.06
95.27+40.95
86.214-0.86
93.19 1.01
91 (6.66-+ 6.3 1.39
89 93.95-+11 1.78
PUSB-c
89.84=0.67
92.04++0.36
92.96++-025
92.80+
90.44:20.72
58-58-11 1.67
VPU
92.35=:0.79
98,33-10.85
92.36:1:0.66
89.00 1.39
51.86-113.0
89.52-+0.97
92.02:10.61
92.76110.3
85.28
Dist-PU
72.28::1.26
1..46+11003
75.23++1444
94.04
87.89+ 1.96
88. 03.08-111 1.38
Dis-PU-c
70.1912211
77.82.10.81
73.93.10.49
08.32
77.27:11993
82.61:12.50
25

---

Published as a conference paper at ICLR 2026
Table 1S: Test results (tta aceuracy, AUC, and F1 score for each algorithm on USPS (Case 1) under different model selection criteria. The best performance w.r.t. each validation metric is shown bold. Here, "-c" indicates using the proposed calibration technique in Algorithm
Val merto
PA
PAUC
OA
PA
PAUC
OA
PA
PAUC
OA
22.22411.24
0.00
02.60=0.25
02.4223666
02.77:0177
PAN
84.52+0.50
84.52-0055
80.98+00200
90,99+048
90:06:0.46
81
CVIR
12.79=1448
82.01
82.01 +0.96
82.96
94.8220.66
00.0000.16
93. 93.4220.69
12.77-1.331
81 01.95=00799
82.761
PIMIX-E
18.9921.40
89 19.4921.29
19.17
96, 0444
96.334445
90, 90.2540.47
87.54=1 330
88.02
87.922 1.30
PSMIX-C
02.69-00.6
93.47+0.JP
93.22-00311
97. 98+0
9816++0.14
98.09 0900++0.00
91.41 31.41-0.5
02.38 0.55
02.05+0.366
LBF
91.45=00.2
90
11.16+0.18
Loss
0.66100669
Robsst-PU
01.73++0.27
88 1978
02.70++0.12
07.51
07.222222
07.23+0.15
89
83.74+55
10100+0012
Holiini-PUU
91:34=00.82
92.566+t1 1
93.4 03.46--00.34
07.22+00.3
47
97.76+0016
90.58-0055
91 02:1220002
92 02-27=0040
PUe
$1.82_11.00
84.22 10.50
$6.93 10.27
95. 15..1
95 25.5110 06.25110.13
94.0011.03
84.23 0.76
83 (1.601100
K5.24 0.59
GLWS
91,13-+0.37
86, 0.660
90.53 00.52-+0.47
96.21 0.001
97. 07.78+0.17
96, 18+00.9
90 90.40-0.36
86 86.38-+0.55
uPU
83.14+00.3
83.87++0.11
83.86+0.83
02.88+0.15
91 10+0.88
9301+0005
81-51-081
08++0000
07-04+0.70
uPtse
03.44
51. 11.30++110 1.16
03.30=00110
97 97.95=0.122
91 11.79
97.550009
92.05-00344
88 18894=1777
91 04
noPL
91160=10.25
87 $1.
41.222-0442
90 09
97 0.63
97. 10
89 89.22=0.27
87 02:20.33
89 4420
mcPC-
92,64+00.88
90 9299
03.24-10188
97 0.00
97. 3410.17
97.99
91 01:00 01.00-0.12
88 88.41:111
91 76+-0.23
miPC-GA
92.51+0.31
30+00.35
moPC-GAc
90.00+144
02.79-00222
97.58=005
97 46
97.66200111
51
33+032
PUSB
10.90:10.73
91.73
91 1-.8
00.98:0065
01.11
02.17:0.00
89 0.717
90.91
90
PUSR-
02:91+00.30
92. 02.84-00.44
02.83+00118
92. 30+00.2
92. 26
92.25+0.299
91. 01:34-0.5
91 78--0-11
91 76-+0.78
VPU
$6.1412.21
10.04
$9.8911.3
92.98 92.961395
91,3110.113
91.26110.19
84.36 3.09
10017
86.58 2.62
VPU-s
92.92:10.07
80. 50.17.1..56
93.29 10.30
17.5510.15
82.0.02
97.7910.118
91 0.08
63.80:11 17.92
91 0..38
87.73++0.55
85.15+2
10.100+0.14
92.52+0.85
91.03+0.777
PU
92.01.10.19
90,47.10.77
91,30.10.44
97.9210.16
90,9510221
97.2410.21
90.16_0222
8..84
89.4410.44
Table 16: Test results std) of precision and recall for each algorithm on USPS (Case 1) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, "-c" indicates using the proposed calibration technique in Algorithm 1.
Test metric
Precision
Recall
Val metric
PAUC
PAUC
OA
PUèN
92.93-+-0288
93.46-0.72
93.91+0.110
92.27+0.31
89,53+
91.65+00.35
PAN
84.90+t5.07
87.64=5.36
90,59+5.35
49.77
76.59=6.00
74.27=5.22
CVIR
7..1941..0
71.37=1.25
1..62+1.66
96.90+0.42
96.27:0.23
96.27:10.66
P3MIX-E E
85.2111..71
86.04 86.0411.34
89.31 13.14
9..47
90.43.1.08
2.
P3MIX-C
90.96-1-.73
9..40-0.59
91.48-10.43
91.88-10.91
93.37-40.55
92.63-0.311
LBE
85.53++1.55 1.52
78.00=1.86
89.02+2.12
96.24+0.81
97.18+1.13
1..69
Count Loss
87.21-11.11
85. 2.553
88.05
94.14410.74
1.97
91.65 1.43
Robust-PU
93,49-t
0.44
94.46+0.25
86.63+0.95
75.96+8.06
88. 16+0.116
Holistic-PU
1..61+1166
92.29=1.55
92.36.0.77
94,47400.28
90.12+1.455
02.20-0.226
PLe
75.50:11.85
74.71 0..58
81.821180
95.45:1.008
94.90-10.45
89.41
GLWS
83.50.10.64
76.68.10.77
82.47.10.75
98.55
98.90 0.19
98.59 0.06
uPU
76.2911:1.56
77.77:1
87.57:10.17
86.63-10.61
86.9011..47
uPU.c
94.51.10.21
95. 10.55
1.0322
8..63
83.45.1..19
89.88 0.60
83.01+0.499
77.78+
02.63
97.8440.22
98.78+0.14
97.76++0.36
enPUie
94.11.10.36
94.52
94.85 0..20
88.1610.500
83.18.12.52
88.86 1.44
maP11GA
89.02-+1.35
91.22-00.66
89.89 0.46
90.78-+-2.06
91.02-+0.92
92.75
anPU-GA-e
93,49+0.40
18-1890.39
93.02+0.36
89,144: 1.38
82.98+3.91
89.73-0.94
PUSB
81.80:11112
85.07=0.28
84.72 1.55
9% 08.04-:0.28
97.6110.28
07.33-10.42
PUSB-c
0.558
0..41
VPU
94.3844...9
66.67-4-7.22 27.22
97.19 0.39
76.55
0.16--0.08
78.43
VPU-c
94.22+00.79
9.88
93.30+0.50
88. 88.78+00.6
18.53-218.31
90.67-0.22
Dist PU
80.1910.788
73.27
79.18 1: 1.24
94.35 0.222
92.71.1..44
91.41 1.93
Dist-PU-e
94.2440.588
95.22-0033
94.27 04.27400.38
86.43-+0.43
81.61+2.06
85. 10=
26

---

Published as a conference paper at ICLR 2026
Table 17: Test results (td) accuracy, AUC, and Fl score for each algorithm on USPS (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, "-c" indicates using the proposed calibration technique in Algorithm
Test ACC
AUC
Val metrie
PAA
PAUC
OA
PA
PAUC
OA
PA
PALC
OA
94.4540.226
0.16
98.83-11.05
20.10
96.23-0.21
95.31.10.15
PAN
83.54-122
16 3.61
02.49+0.33
78,80-30
41.47:22.11
824551100
CVIR
9093400.2
0055500.0
06.74
96. 26.54
96.60=0.21
01.37=0:23
89. 00.34+0.22
01.05=0.16
PIMIX-E
94.06:0.43
03:90:0433
91.90:10 0.30
96. 0.20
98.3 98.24:0.366
98.14:0.119
93.92 0.30
93. 93.79
93.81 2.336
PIMIX-C
94.27+0..22
94.54-0..1
94.72 70+022+000 0.35
98. 38++0.34
98. 06.56-+--17
98 18.66-00.13
94.20-0.4
04.4 94,47++0.44
94 94.22+0.36
LBC
94.67.10.20
90.82
94.88 0.00
96. 18.51.10.15
98.1710.088
96.60110.77
96.65_0.1
91.1610.95
94.73 10.10
Loss
92.7510.22
03.76:1045
93.17.0.11
97.72:10.20
Robust-PU
03.72400.41
23.41
03110:1
98.34:11.35
93.33:20.31
Holisaic PU
95.15:10.28
94.83
95.02.0.53
08.76.10.11
98.73.10.17
98.49 0.20
94.99
94.65.10.3
98.3410.57
PUL
15.27
85.00=0044
86 00.55
93 03.0500.45
95 15.263
93.48 0.902
86 06.55
56. 38
87
GLWS
02.48
88.192044
92
98 0.055
98.0920.29
9000
92.76-0.44
89.15 05.35
42.49:0223
N3.3610.48
04:1220006
0340000
93.9911.255
20.79
94.1-0042
$4.32
US 022
u
94.67.10.10
93.36.10.66
94.7.10.28
98.7810.10
98,50110.00
0.11
9.136-00111
92. 92.85.10.88
94.2610.32
onPU
93.6441.11
88.01 1:30
94 00500.37
96 50
98. 58.57
98.7 20008
93.79
50.01 1.07
94 20:0
naPU-c
94.3240.20
94.002024
94 12
96 0.12
95067-00044
ya 05
s3 22.6770..27
94 0U
mPPU-GA
94.95+0.11
94. 08.0070007
98 +0
98 18.66+0..0
98.44-00
94.28 94.28-0.2
94 76
94 50-0008
moPLi-GA-e
98.77-0004
US 40
PUSB-e
03.64+0311
03.76-0.19
01:24+0.34
YPU
89.8212661
16.63:110.29
19.5611888
10.35
98.58-10.10
0..57
3.43
53.65
28.112241
VPU
94.9310.133
94.82 10.12
101.
96.78110 0.00
98.86 10.09
98.91 0.11
96.72_0.117
94.5510.16
94.81 10.14
Dac. PU
94.2220.12
94,0220.3
94, 04.7220.38
96 0911
92. 94----11222
9%. 09
94.63-0.13
93. 93.7240.25
$4.5520.35
94,10100.44
92.0910.00
94.1420.37
98. 95.4910.11
98.53:10.20
9.030
93.73_0.49
91.3010.59
93.77.00422
Table 18: Test results std) of precision and recall for each algorithm on USPS (Case 2) under different model selection criteria. The best performance w.r.t. each validation metric is shown in bold. Here, c- indicates using the proposed calibration technique in Algorithm 1.
Test metric
Precision
Recall
Val metric
PA
PAUC
OA
PA
PAUC
95.38:10.94
05.68=00.48
95.97-10.59
1..18
00.2
0..4
FAN
83.76.13.59
19.42
1..52
75.13.15.70
75.67
57.3
85.15+00.2
81.84-0.41
84.48++0.37
98.57400.55
98.
98.74++0.18
P3MIX-E
93,52+1.27
93.30=1.32
92.82+0087
94.37+0.51
94.34=0.58
04.85+0029
P3MIX-C
9..07:11.19
03.28
93.88-10.75
95.39110.26
95.70-0.08
95,39:10.10
LBE
92.57.1..82
86.54
95.05
96.5610.56
96.4511.116
94,47.1.00
Count Loss
88.88:20.75
90.40=0.96
90.75++0.87
97,27+0.57
97.58=0.41
95.80 1.05
Robust-PU
95.00.10.46
96.73:10.26
91.44.0.17
0..73.1.399
Holistie PU
95.52.10..4
95.53-10.65
95.60-1-0.47
94.47-10..2
93.79
94, 10.1-.99..2
77.96+
77.46=0.78
79.31+40.72
97.30++0.88
97.65-#0.21
GLWS
87.39-10888
80.72=0.66
86.83-t0.38
98.84:10.12
99.56=0.20
98.94:20.06
uPE
77.29+0.61
75.45
78.20-+-077
93.24-+-077
95 05.67-0..55
93.55++1 1.60
uPU-c
97.19:00.88
97.26=0.65
96.97++0.16
91.71++0.50
88.88 1.65
91.71+0..0
1.886.1186
80.69
98.16:10.35
99.32-00007
98.26 0.113
unPU-e
96,32+0.13
96.26=0.188
96.46-+0051
91 85-+0.53
91.23-0..3
92.73+0.29
amPU-GA
94.11:00.67
0..22-0.228
95.61:10.20
94.47:10.26
93.72-20.44
93.59-10.28
GA
96.77.10.20
97.1910.25 10.29
97.11.10.18
90.1610.11
90.86 1.11
90.52 0.59
PUSB
87.44+11.11
2.447
89.52-4-075
98.64-4-0.10
97. 0..32
98.29 0.22777
PUSB-e
96.44+0.41
96.58=0.49
96.84-+0..41
91.23-+0.44
90.14==0.63
91.85+0.27
VPU
96.67:20.70
98.82=0.56
97.26-1:0.20
82.02-43.96
$2.95
80.76 3.822
VPU
96.80.10.21
92.9010.07
Dist-PU
95.45:t0.21
05.77=0.70
95.01-t-:40
93. 13.82100.18
9181=1.11
94, 0..5000.550
Dist-U-e
97.04:0..15
98.27:10.26
96.98.10.06
90.65
85.26
0..89
27


Reviews:
[{"review_id": "kAFcAoxx8F", "reviewer": "Reviewer_DKd8", "summary": "This paper performs an extensive benchmarking of many PU learning algorithms in a unified setting that is designed to be realistic and fair, and the results provide novel insights on the performance of those algorithms. In addition, through theoretical analysis, it identifies the problem of directly apply a two-sample PU learning algorithm to the one-sample setting, and provides a simple solution to this problem.", "questions": "* How are the minibatch sizes for the P and U data set in Algorithm 1?\n* Why is SGD used while Adam is often better?\n* Training is done for 20,000 iterations. Is this generally large enough to ensure that the models are sufficiently well-trained, despite dataset size differences?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As far as I know, this is the first rigorous and extensive benchmarking for PU learning algorithms. I believe the benchmarking provides interesting insights that can be useful for further research in the field, and the methodology can be adopted in future work for more rigorous comparison.\n\nI think the theoretical analysis is interesting and it seems to be sound.\n\nThe paper is well written and easy to follow.", "weaknesses": "Currently the benchmarking is performed on four benchmarks while some other datasets are also commonly used, and it would be helpful to have benchmarking results on some of those datasets for comparison with results in the literature. Nevertheless, current benchmarking already requires substantial effort though.\n\nThe model details seem to be missing. Table 4 mention ResNet and MLP but there are no details.\n\nMinor comment: \"accessible\" doesn't seem needed as this is subsumed by \"realistic\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dB0t6nfnZb", "reviewer": "Reviewer_c2Ap", "summary": "This paper focuses on empirical studies on PU learning methods. It systematically investigate model selection criteria to reduce the dependence on validation set, and designs a simple approach to ensure fair comparisons within and across one-sample and two-sample settings.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides a realistic and fair evaluation framework of PU Learning, including model selection criteria and evaluation protocols. The framework is important to identify the effectiveness of components within existing methods and will promote the development of PU Learning.\n2. The paper is well-structured and written in a fluent manner, making it easy to follow and understand.\n3. The analysis is thorough and the experiments are comprehensive.", "weaknesses": "1. Some evaluations over real-world datasets (e.g., from medical imaging, fraud detection, or recommender systems) are expected to sufficiently demonstrate the benchmark's effectiveness on larger-scale, more complex, or truly real-world PU datasets.\n2. The proposed calibration approach for the Internal Label Shift (ILS) problem is presented as \"simple yet effective.\" However, the paper lacks a discussion of its potential limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NbU9ypArC8", "reviewer": "Reviewer_BgWp", "summary": "This paper establish the first comprehensive benchmark for evaluating PU learning algorithms. Through extensive experiment with many state-of-the-art algorithms, the authors identify the internal label shift problem between OS and TS PU learning settings. The proposed model selection criteria doesn't rely on negative validation data, which contradicts PU learning's core premise. In addition, the authors proposed a calibration methods with theoretical guarantees to address the label shift issue.", "questions": "See weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors conducted a thorough evaluation using four datasets with systematic data generation and consistent evaluation protocols. Three criteria are compared for model selection purposes. The experimental framework is particularly well-designed, addressing the unrealistic assumption that many existing evaluations make about having access to negative validation data.", "weaknesses": "1. The authors use relatively small benchmark dataset, which may not reflect large-scale performance. \n2. Deeper exploration on how estimation errors in class prior affect performance is not presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fwmQmPPInJ", "reviewer": "Reviewer_WGLx", "summary": "This work investigates the experimental settings in PU learning.\nThe authors argue that using a validation set containing both positive and negative labeled samples for model selection is unrealistic, and therefore propose a proxy metric as a surrogate for this purpose.\nThey further discuss two sampling processes in PU learning, namely One-Sample (OS) and Two-Sample (TS), and introduce a calibration technique to mitigate the class-shift problem when transitioning from the TS to the OS setting.", "questions": "See weaknesses.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The proposed proxy metric is useful for model selection in cases where a validation set with both positive and negative samples cannot be obtained.", "weaknesses": "1. The authors discuss two data generation processes, namely One-Sample and Two-Sample. However, it remains unclear why they do not consider benchmark settings with varying ratios of positive to unlabeled data, or different proportions of positive samples within the unlabeled set. By controlling the ratios, we can validate PU algorithms across varying settings to demonstrate their robustness. \n2. Regarding the proposed PA, although the authors argue that access to a fully labeled validation dataset is unrealistic, their assumption of knowing the class prior probability $\\\\pi$ is equally impractical. \n3. Although the authors theoretically prove that the proposed proxy metric preserves the partial order of the original metric, they provide no experimental evidence that the model selected by this proxy coincides with the one chosen using a validation set containing both positive and negative samples. \n4. The introduction of the calibration technique for the OS setting seems redundant, since the OS setting can be straightforwardly transformed into a TS setting by redefining the unlabeled training data to combine the original positive and unlabeled sets. Essentially, the proposed calibration achieves the same effect implicitly, as demonstrated below.\n\nWe start from Eq. 8:\n$$\nR = \\\\frac{\\\\pi}{n_p} \\\\sum_{i=1}^{n_p} ( \\\\ell( f(x_i), +1 ) + (c-1) \\\\ell (f(x_i), -1)) + \\\\frac{1 - c\\\\pi}{n_u} \\\\sum_{j=1}^{n_u}\\\\ell(f(x_i), -1).\n$$\nThis can be rewritten as\n$$\nR = \\\\frac{\\\\pi}{n_p} \\\\sum_{i=1}^{n_p} \\\\ell( f(x_i), +1)- \\\\frac{\\\\pi}{n_p} \\\\sum_{i=1}^{n_p} \\\\ell (f(x_i), -1) + \\\\frac{\\\\pi}{n_p} c \\\\sum_{i=1}^{n_p} \\\\ell (f(x_i), -1) + \\\\frac{1 - c\\\\pi}{n_u} \\\\sum_{j=1}^{n_u} \\\\ell(f(x_i), -1).\n$$\nSubstituting $c = n_p / (\\\\pi (n_p + n_u))$ gives\n$$\nR = \\\\frac{\\\\pi}{n_p} \\\\sum_{i=1}^{n_p} \\\\ell( f(x_i), +1) - \\\\frac{\\\\pi}{n_p} \\\\sum_{i=1}^{n_p} \\\\ell (f(x_i), -1) + \\\\frac{1}{n_p + n_u} \\\\sum_{i=1}^{n_p} \\\\ell (f(x_i), -1) + \\\\frac{1}{n_p + n_u} \\\\sum_{j=1}^{n_u} \\\\ell(f(x_i), -1).\n$$\nThus,\n$$\nR = \\\\frac{\\\\pi}{n_p} \\\\sum_{i=1}^{n_p} \\\\ell( f(x_i), +1) - \\\\frac{\\\\pi}{n_p} \\\\sum_{i=1}^{n_p} \\\\ell (f(x_i), -1) + \\\\frac{1}{n_p + n_u} \\\\sum_{i=1}^{n_u + n_p} \\\\ell(f(x_i), -1)\n$$\nThis derivation shows that the proposed calibration is equivalent to simply merging the positive and unlabeled sets into a single new unlabeled set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)