Positive-Unlabeled Learning with Uncertainty-aware Pseudo-label Selection
Authors: Emilio Dorigatti
Jann Goschenhofer
Benjamin Schubert
Mina Rezaei
Bernd Bischl
Venue: ICLR 2022 Submitted (2022)
Keywords: positive-unlabeled learning
semi-supervised learning
pseudo-labeling
deep ensembles
uncertainty quantification

Abstract: Positive-unlabeled (PU) learning aims at learning a binary classifier from only positive and unlabeled training data. Recent approaches address this problem via cost-sensitive learning by developing unbiased loss functions or via iterative pseudo-labeling solutions to further improve performance. However, two-steps procedures are vulnerable to incorrectly estimated pseudo-labels, as errors are propagated in later iterations when a new model is trained on erroneous predictions. To mitigate this issue we propose \textit{PUUPL}, a new loss-agnostic training procedure for PU learning that incorporates epistemic uncertainty in pseudo-labeling. Using an ensemble of neural networks and assigning pseudo-labels based on high confidence predictions improves the reliability of  pseudo-labels, increasing the predictive performance of our method and leads to new state-of-the-art results in PU learning. With extensive experiments, we show the effectiveness of our method over different datasets, modalities, and learning tasks, as well as improved robustness over mispecifications of hyper-parameters and biased positive data. The source code of the method and all the experiments are available in the supplementary material.

Under review as a conference paper at ICLR 2022
PUUPL: POSITIVE-UNLABELE LEARNING WITH UNCERTAINTY- AWARE PSEUDO-LABEL SELECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Positive-unlabeled (PU) learning aims learning binary classifier from only positive and unlabeled training data. Recent approaches addressed this problem via cost-sensitive learning by developing unbiased loss functions, and their perfor- mance was later improved by iterative pseudo-labeling solutions. However, such two-step procedures are vulnerable incorrectly estimated pseudo-labels, as er- rors are propagated in later iterations when a new model is trained on erroneous predictions. To mitigate this issue we propose PUUPL, new loss-agnostic train- ing procedure for PU learning that incorporates epistemic uncertainty in pseudo- labeling. Using an ensemble of neural networks and assigning pseudo-labels based on high confidence predictions improves the reliability of pscudo-labels, increasing the predictive performance of our method and leading to new state-of- the-art results in PU learning. With extensive experiments, tiveness of our method over different datasets, modalities, and learning tasks, as well improved robustness positive data. The source code of the method and all the experiments are available in the supplementary material.
show the effee-
misspecifications of hyperparameters and biased
INTRODUCTION
Many real-world applications involve positive and unlabeled (PU) datasets in which only some of the data is labeled positive while the majority is unlabeled and contains both positives and nega- tives. PU learning aims to learn a binary classifier in this challenging setting without any labeled negative examples. Learning from PU data reduce deployment costs in many deep learning applications that otherwise require annotations from experts such as medical image diagnosis (Ar- menian and Lilienfeld 1974) and protein function prediction (Gligorijevié al., (022), , and it can even enable applications in settings where the measurement technology itself can not detect negative examples (Purcell al. 2019).
Some recent approaches such as unbiased PU (Du Plessis et al. 2014 uPU) and non-negative PU (Kiryo et al. 2017 nnPU) formulate this problem cost-sensitive learning. Others approach PU learning as a two-step procedure first identifying and labeling some reliable negative examples, and then re-training the model based on this newly constructed labeled dataset (Bekker and Davis, 2020). These approaches show similarities with pseudo-labeling in semi-supervised classification settings (Lee 2013).
Such pseudo-labeling techniques are however especially vulnerable to incorrectly assigned labels of the selected examples as these errors will propagate and magnify in the retrained model, resulting in negative feedback loop. Worse yet, since the true labels are unavailable in PU learning, this situation hard to detect by any metrics computed on the training set. This erroneous selection of unreliable pscudo-labels occurs when wrong model predictions associated with excessive model confidence. Such poor model calibration accompanied by a distortion of the signal for the pscudo label selection (Van Engelen and Hoos 2020).
In recent literature on pseudo-labeling explicitly estimating the prediction uncertainty (Abdar et al.) 2021 Rizve al.) 2021 Arazo al. 2020). While this is the case for semi-supervised classification, there does not yet exist a method that explores the use of uncertainty quantification for pseudo-laheling in PU learning context.
this problem is recognized and successfully addressed by

---

Under review as a conference paper at ICLR 2022
[a] Network Unc.
(b) Ensemble Unc.
Train an ensemble
Predict with
uncertainty
Network Unc.
(c) Ensembbe Unc.
Dentty
Auc: 0.55 MUCi 6.41
Dositive-ullaelldd
Pseudo-labeled
PP*
Select with uncertainty
PUUPL is pscudo-labeling framework for PU learning that uses the epistemic uncertainty of an ensemble to select confident examples to pscudo-label. The ensemble can be trained with any PU loss for PU data while minimizing the cross-entropy loss on the previously assigned pscudo-labels. On a toy example, a single network is not very confident on most of the unlabeled data (a), resulting in many high-confidence incorrect predictions and many low-confidence correct ones (c). The epistemic uncertainty of an ensemble is, the other hand, very low on of the the unlabeled data (b), resulting in correct predictions having low uncertainty and most incorrect predictions having high uncertainty (d). Thus, the uncertainty an ensemble can be used more reliably to rank predictions and select correct ones (e).
Figure 1:
Contributions: Motivated by this, we propose a novel, uncertainty-awarr pseudo-labeling frame- work for PU learning that uses established uncertainty quantification techniques to identify reliable examples to pseudo-label (Fig. [ In particular, our contributions are: (1) We introduce PUUPL (Positie-Unllaeled,, Uncertainty-Aware Pseudo-Labeling), a simple uncertainty-aware pseudo- labeling framework for PU learning. (2) PUUPL can use any loss function for PU learning, im- proving model performance while being robust to the specific data biases that the respective loss considers. (3) We evaluate our methods on wide range of benchmarks and PU datasets, achieving state-of-the-ane performance in PU learning. (4) Our extensive ablation studies provide new insights into uncertainty-aware pseudo-labeling for PU learning. Further, they show that our method is ro- bust to the choices of hyperparameters, with 1% or less variability in test accuracy among different choices well distribution shifts between labeled positives in the train and test datasets. To the best of our knowledge, PUUPL is the first framework for PU learning which leverages uncertainty information during pseudo- beuuoolabeeing.
RELATED WORK
PU Learning PUL introduced a variant of binary classification (Liu et al. 2003) and is related to one-class learning (Ruff et al. 2018 Li et al. 2010), multi-positive learning (Xu et al. 2017]. multi-task learning (Kaji et al. 2018). and semi sume--sperriieed learning (Chapelle et al. 2009) There exist three main research branches for PUL: two-step techniques, class prior incorporation, and biased PUL (Bekker and Davis 2020). this work, we combine Pseudo-Labeling which has similarities to two-step techniques, with biased PUL, also coined as reweighting methods, and refer to Bekker and Davis (200)) for a comprehensive overview of the field. In this context, Du Plessis et al. (2014) introduced the unbiased risk estimator uPU. Kiryo et al. (2017) showed this loss func- tion is prone to overfitting in deep learning contexts as it lacks a lower bound and proposed the non-negative risk estimator nnPU
remedy. Follow-up work on loss functions for PUL has fo- cused on robustness w.r.t biases in the sampling process such as PUSB (Kato et al. 2019). PUbN (Hsieh et al. 2019) PULNS (Luo et al. 2021).
2

---

Under review as a conference paper at ICLR 2022
Uncertainty-aware Pseudo-Labeling Pseudo-labeling follows the rationale that the model lever- ages its own predictions on unlabeled data as pseudo training targets to enable iterative semi- supervised model training. The first such approach for deep learning was introduced by Lee (2133), simply selecting the class with the highest predicted probability as a pseudo label. One weakness of pseudo-labeling is that erroneously selected pseudo-labels can amplify for training, potentially lead- ing to model degradation. This is grounded in poor model calibration distorting the signal for the pseudo label selection (Van Engelen and Hoos 2020). [scen et al. (211)) try to mitigate this issue using confidence and class weights. [Shi et al. 2018) use confident scores based on the geometric neighborhood of the unlabeled samples while Arazo et al. effectively tackle this confirmation bias using Mixup (Zhang et al.) (017). Label Noise (Tanaka et al. 2018), and Entropy Regulariza- tion (Grandvalet et al. 2005). Rizve et al. (202T) introduced a pseudo-labeling framework using a weighting scheme for class balancing and MC dropout (Gal and Ghahramani (016)) for calibration, Beluch et al. (2018) found deep ensembles (Lakshminarayanal et al. 201 7a) to yield the best model calibration an active learning context, especially in low-label regimes. The commonality of these works is the explicit consideration of model uncertainty to improve pseudo-label selection, which motivates us to apply this in the context of PU learning.
Pseudo- Laeuei-a for PU Learning Two-step approaches in PU learning first identify negative sam- ples from the unlabeled dataset, and then train binary classification model on the original dataset augmented with the newly identified negatives (Bekker and Davis, (020). These approaches share similarities with pseudo-labeling but lack an iterative feedback loop after the completion of the second step.
A first attempt combine pseudo-labeling with PU learning was made with Self-PU (Chen al. 20205), where self-paced learning, a confidence-weighiee scheme based on the model predictions and teacher-student distillation approach are combined. Via this complex training scheme, Self-PU was shown to marginally outperform recent baselines. With PUUPL, we propose an alternative PL strategy for PU learning that performs better in a simpler and more principled way using implicitly well-calibrated models to improve the pseudo-label selection.
Uncertainty-aware pseudo-labeling for PU learning To the best of our knowledge, we are the first to introduce uncertainty-awart pseudo-labeling paradigm to PU learning. Although our method shares the same motivation as that from Rizve et al. (202 for semi-supervised classification, we differ in several important aspects: (1) we specifically target PU data with a PU loss, (2) we quantify uncertainty with an ensemble instead of Monte Carlo dropout, (3) we use epistemic uncertainty instead of the predicted class probabilities for the selection, (4) we do not use temperature scaling, and (5) we use soft labels.
3 METHOD
We propose PUUPL (Positive Unlabeled, Uncertainty-aware Pseudo-Labeling), an iterative pseudo- labeling procedure to progressively select and label the most confident examples from unlabeled data. The pseudo-code for PUUPL is shown as Algorithm [ Our method separates the training set x into the sets P. U, and L which contain the initial positives, the currently unlabeled, and the pseudo-labeled samples respectively. The set L is initially empty. At each pseudo-labeling iteration, we first train our model using all samples in P. U, and L until some convergence condition is met (Section 3.2) Then, samples in U are predicted and ranked w.r.t their predictive uncertainty (Section 3.3) and samples with the most confident score are assigned the predicted label and moved into the set L. (Section B.4p. Similarly, samples in L are also predicted and the most uncertain samples moved back to the unlabeled set U (Section 3.5). Next, the model is re-initialized to the same initial weights and new pseudo-labeling iteration starts.
In the following, we first describe the notation used in this paper and then explain in detail the training procedure of PUUPL.
3.1 NOTATION
Consider input samples X with label y and superscripts ""r _ee and te for training, validation, and test data respectively. The initial training labels y are set to one for all samples in P and zero for all others in U. We group the indices of original positives, unlabeled, and pabude-labeled samples
3

---

Under review as a conference paper at ICLR 2022
Algorithm Pseudocode for the PUUPL Training Procedure
Input
Train, validation and test data X yy" , .2" xt=, y
Number K of networks in the ensemble (suggested K =
Maximum number 7' of pseudo-labels to assign at each round (suggested ' 1000)
Maximum uncertainty threshold 4 to assign pseudo-labels (suggested 0.05)
Minimum uncertainty threshold tu to remove pseudo-labele (suggested t. = 0.35)
Output Model parameters 0*
1: P <- indices of positive samples in Xt
2: U f- indices of unlabeled samples in X
> Indices of pseudo-labeled samples
4: 8 f Random weight initialization
5: while not converged do
6: Initialize model weights 0
>. Training
Train ensemble of K networks X """ using the loss in Eq. Validate on = y update weights 0" if accuracy improved
9:
f ensemble predictions for 8x
Uncertainty
10:
Compute epistemic uncertainty ú with via Eq.
11:
L Lue Balanced set of examples to pseudo-labe via Eq. Jusing úg. T and U Examples pssedo-unlalee
Pseudo-labeling
Eq. 10 using üi
13:
L + LU Lb U new
Update indices
14:
U + U T. U mee
15:
N Pm
D Update pseudo-lahels
16:
end while
18:
Restore the weights 0 that scored highest on the validation set
19:
Compute accuracy the held-out test set X
20:
return 8*
X into the sets P. U, and respectively. Our proposed model an ensemble of K deep neural networks whose random initial weights are collectively denoted as 60. The predictions of the k-th network for sample are indicated with Piu- offx), with a(-) the logistic function and fu the predicted logits. The logits and predictions for sample averaged across the networks in the ensemble are denoted by f; and d respectively. We subseript data and predictions with to index individual samples, and use an index set in the subscript to index all samples in the set (e.g., X = (X e } denotes the features of all unlabeled samples). denote the total, epistemic and aleatoric uncertainty of sample i as úi, úf, and uf, respectively.
3.2 LOSS FUNCTION
We train our proposed model with loss function that convex combination of a loss Lpu for the samples in the positive and unlabeled set (P U U) and loss Li for the samples in the lseudo-labeled set (L):
-= C- Lr. +1 (1- LrU
with À E (0, 1). The loss C, is the binary cross-entropy computed w.r.t the assigned pseudo-labels y. Our model is agnostic to the specific PU loss Lre used. This allows our method to be easily adapted to different scenarios for which a PU loss was proposed and improve over its performance, for example when coping with selection bias in the positive examples (Kato et al. (019) the
availability of a biased negative set (Hsieh et al. 2019). For the standard setting of PU learning, we the non-negative correction nn PU of the PU loss (Kiryo et al. 2017):
Cru ((P11) max (0, (U,
11)
With the prior probability of sample being positive, which assume known and be esti- mated from PU data (du Plessis et al. 2016), and f(S. y) the expected sigmoid loss of samples in the set S with label y: ((S.v)
(3)
S
I exply P.)
es
4

---

Under review as a conference paper at ICLR 2022
3.3 MODEL UNCERTAINTY
To quantify the predictive uncertainty, we utilize deep ensemble with K networks with the same architecture, each trained on the full training dataset (Lakshminarayana) et al., 2017b). Given the predictions Pa1.... Pik for sample Z3, we associate three types of uncertainties x; predictions (Hüllermeier and Waegeman 202T): the aleatoric uncertainty as the mean of the entropy of the predictions (Eq. 4 the total uncertainty as the entropy of the mean prediction (Eq. [5) and the epistemic uncertainty formulated as the difference between the two (Eq. 6).
8. Σ k=1
Litx log Pak + (1 Pssk) log(1 - pa))
(4)
of = P. log P. (1 - log(( P.)
(5)
" i,
(6)
Epistemic uncertainty corresponds to the mutual information between the parameters of the model and the true label of the sample. Low epistemic uncertainty thus means that that the model param- eters would not change significantly if trained on the true label, suggesting that the prediction is indeed correct. Using such prediction target the cross entropy loss would in turn provide stronger, more explicit learning signal to the model, so that correctly pseudo-labeles example provides a larger decrease in risk compared using the same example without any label within the positive-unlabeled loss.
3.4 PSEUDO-LABELING
The estimated epistemic uncertainty (Eq. 66 is used to rank samples the unlabeled set U and to select reliable samples for pseudo-labeling. Next, the predictions of the unlabeled samples U are ranked according to their epistemic uncertainty (Eq, 6). Let p(i) denote the rank of sample i, then the L new of newly pseudo-labeled samples formed by taking the samples with lowest uncertainty from U, ensuring that it is lower than the threshold ty:
new = E ( STAU, st)
Previous works have shown that balancing the pseudo-label selection between the two classes, i.c., ensuring that the ratio of newly labeled positives and negatives is close given target ratio r, beneficial (Rizve et al. 2021 In this case, the set nne should be partitioned according to the model's predictions into set Lee of predicted positives and Leew of predicted negatives, and the most uncertain samples in the larger set should be discarded to reach the desired ratio r, which we fix to 1. We then assign soft pseudo-labels, i.e., the average prediction in the open interval (0, 1), to these samples:
y: Dj ViE new
L
new
(8)
3.5 PSEUDO-UNLABELING
Similar to the way that low uncertainty on an unlabeled example indicates that the prediction can be trusted, high uncertainty on pseudo-labeled example indicates that the assigned pseudo-label might not be correct after all. To avoid training on such possibly incorrect pseudo-labels, we move the lseudo-labeled examples with uncertainty above threshold f, back into the unlabeled set:
Tnew = (i E L 2 twp Vi Tnew
(9)
yi
(10)
5

---

Under review as a conference paper at ICLR 2022
Mor
0.30
Validation Accuracy
0.85
0.25
Calibration Calbbattii
0.80
0.20
0.75
Expected
0.70
0.05
200
400
1000
1200
1400
1600
Epoch since start
Validation accuracy (left, blue) and expected calibration enoor (ECE, right, green) for on CIFAR-IO. The ensemble is trained for fixed number of 100 epochs before pseudo-labeling, visible the periodic spikes in both curves, Note substantial reduction in ECE the second and third pseudo-labeling iterations, when the ensemble is trained on soft labels. The best validation accuracy of 90.76% is indicated by the orange dot and corresponds a test accuracy of 90.35%.
Figure 2:
4 EXPERIMENTS
4.1 EXPERIMENTAL PROTOCOL
To empirically compare proposed framework existing state-of-the-art losses and models,
followed standard protocols for PU learning (Chen et al. 2020b Kiryo et al. 2017 Kato et al. 2019).
Datasets: We evaluated our method in the standard setting of MNIST (Dnng
2012) and CIFAR-
10 KKrzhevsky et al., 2009 datasets, as well as Fashion MNIST (F-MNIST) Xiao et 2017
STL-10 (Coates et al. 2011) and IMDb (Maas et al.) (201T) datasets to show the applicability to different data modalities. Similar to previous studies (Chen et al. 2020b) Kiryo et al., 2017 Kato eet al., 2019), positives were defined as odd digits in MNIST, vehicles in CIFAR-10 and STL-10, and we used trousers, dress, sandals, sneaker, and ankle boots for F-MNIST and positive reviews for IMDb. For STL-10 used all available labeled and unlabeled data and the official ten cross- validation folds. For all other datasets, we reserved a validation set of 5,000 samples and use all other samples for training with 1,000 randomly chosen labeled positives, is common practice, and evaluated on the canonical test set of each dataset. More details are provided Appendix B
Network architectures: To ensure fair comparison with other works in PU learning (Chen al. 2020b Kiryo et al. 2017) we used the same architectures on the same datasets, namely a 13-layers convolutional neural network for the experiments CIFAR-10 (Table A.3) and MLP with four fully connected hidden layers of 300 neurons each and ReLU activation for MNIST and F-MNIST. For IMDb we used a bidirectional LSTM network with MLP head whose architecture was opti- mized as part of the hyperparameter search.
Training: We trained all models with the Adam optimizer (Kingma and Ba 2015) with 81 0.9 and By 0.999, and an exponential learning rate decay with 0.99. We further used the nnPU loss (Kiryo al. 2017) as LPU (Eq. [T unless otherwise stated. As is common in the pseudo- labeling literature (Chen et al. 2020b; Rizze et al. 2021 Kato et al. 2019 Tanaka et al. 2018 Hu jt al. 202T), we assume that positive and negative labeled validation set available and use this validation set for early stopping, i.c., stop the pseudo-labeling loop when the model's accuracy this set has stopped improving, and use the parameters that achieved the highest validation accuracy to compute the test performance. An experiment will show how this requirement be relaxed in practice.
Hyperparameter tuning: We used the Hyperband algorithm (Li et al., 2017) optimize all hy- perparameters on the CIFAR-10 dataset with 77 = 3 and 4, using the validation accuracy as the criterion be optimized. The configuration that achieved the highest validation accuracy was

---

Under review as a conference paper at ICLR 2022
MNIST
F-MNIST
CIFAR-10
CIFAR-10 (3,000 Lab.)
IMDb
STL-10
(1,000 Lab.)
nnPU"
93.41 (0.20)
88.60 (0.40)
SelIPU"
94.21 (0.54)
89.68 (0.22)
90.77 (0.21)
DAN
89.70
PAN
89.70
78.
UnPU
95.26 (0.55)
95. 70 (0.18)
89.20 (0.29)
90.91 (0.24)
78.57 (0.68)
93.09 (0.54)
+PL
95.78 (0.30)
95.84 (0.16)
89.74 74 (0.46)
91.01 (0.18)
80.64 (0.99)
+PUUPL
96.01 (0.29)
95.91 (0.20)
90.18 (0.15)
91.44 (0.29)
81.83 (0.24)
93.30 (0.42)
Table I: Test accuracy and standard deviation over five (ten for STL-10) runs in brackets on various datascts of recent methods for PU lcarning. The row "+PL" refers uncertainty- unaware pscudo-labeling baseline that uses the average prediction of an ensemble networks as selection criterion, while the bottom row refers to our uncertainty- we also report the performance with 3,000 labeled positives, and STL-10, where we used the official ten cross- validation folds. Bold font indicates highest performance. "scores as reported by Chen et al. (2020b) and Hu
solution. We used 1,000 labeled positives for training except for CIFAR-10, where
al. 202T due unavailability of source code.
then used as basis for the ablation studies and fine-tuned on the remaining datasets show that the pseudo-labeling hyperparameters (Table I do not require tuning when transferred to other datasets and data modalities. Specifically, on the other datasets we only tuned hyperparameters re- lated to network training such batch size, learning rate, weight decay, number of training epochs, dropout probability and other details of the network architecture by running only the first bracket of Hyperband with 1) = 3 and S = 3.
Evaluation: the best configuration found by Hyperband was trained five times with different random training/validdaten splits and evaluated the test set to produce the final results, except for STL-10 where we used the official ten cross- coos-validotionn folds.
4.2 RESULTS
The best performance achieved by our method is shown in Table and compared with SelfPU (Chen iet al. (0200b) DAN (Liu et al. (219 and PAN (Hu et al. 2022). PUUPL was al always able to improve over the baseline nnPU loss, with larger gap for more difficult datasets such CIFAR- 10 (+0.98%) and IMDb (+1.899) 1.9%% as well as over SelfPU (Chen et al. 2020b) and DAN (Liu et al. 2019), setting a new state-of- ofite--o-theaan art for PU learning. Moreover, PUUPL is naturally very well calibrated despite the absence of explicit calibration on labeled data (Fig. D. making its predictions inherently reliable. The best pseudo-labeling hyperparameters constitute the defaults we suggested in Algorithm I and are K = 2, T 1000, I 0.05 and 4y = 0.35. Note that the baseline nnPU scores reported in Table [ were also obtained by training an ensemble of two networks with the nnPU loss, thus possibly explaining the discrepancy observed with SelIPU, The best network architecture for IMDb is shown in Table
4.3 ABLATION STUDIES
We performed ablation studies on the CIFAR-I0 dataset by changing one parameter at a time of the best configuration found by Hyperband, training and evaluating with five different splits and reporting the test accuracy corresponding to the best validation score for each run. To limit the computational resources needed, we used at most 15 aseudo-labellng iterations.
Weights initialization: We confirmed the observation that it is beneficial to re-initialize the weights after each pseudo-labeling step (Arazo et al. 2020), with slightly better performance (+0.052%) achieved when the weights are re-initialized to the same values before every pseudo-labeling itera- tion (Fig. D.. We believe this encourages the model to be consistent across pseudo-labeling rounds.
Pseudo-label assignment: Soft pseudo-labels were preferred over hard ones (+0.755 75%).. We found that our model was very well calibrated with ECEs as low as 0.05 on the labeled validation data (Fig. D. indicating that the soft laeudo-labels they estimated were reliable training targets and that post hoc calibration labels at every iteration harmed performance (-0.12%); instead, pscudo-labels should be kept fixed after being assigned for the first time. A possible explanation is that fixed pseudo-labels prevent the
not necessary. Contrary to expectation, however, re-assigning all pseudo-
7

---

Under review as a conference paper at ICLR 2022
(a) Weights init.
(b) Balanced PL
(c) Uncertainty type
(d) Early stop. labels
90.25
90
90.0
90.00
89
89
89.5
Same
Rand.
No r=1
mmbaly
Epi.
Alea.
Tot.
Nos
Pos-Unl
Figure 3: Mcan and standard deviation of the test accuracy obtained over five runs by different variations of our PUUPL algorithm: (a) different weight initialization cach iteration, (b) balanced imbalanced PL selection, (c) type of uncertainty, (d) whether to use positive-negatioo or positive-unlabeles validation set. Note the different scales on y-axes.
(a) Loss class prior
(b) Cross-ent, weight
Num. Labeled
Model PUUPL
89.0
86
nnPU
85
15
250
300
1000
2000
3000
EDDn
Figure 4: Mean and standard deviation of the test accuracy obtained over five runs by different hyperparameter values, Refer to the main text for a discussion. Note the different scales on the y-axes.
model's predictions from drifting too far away from the initial pseudo-labeling towards incorrect assignment. It was also beneficial to assign the same number of positive and negative pseudo-labels (Fig. Bb) compared to keeping the same ratio of positives and negatives found in the whole dataset (-0.20%) or not balancing the selection at all (-0.55%).
Uncertainty: Ranking predictions by aleatoric performance was almost as good ranking by epis- temic uncertainty (-0.08%), while total uncertainty produced moderately worse rankings (-0.37% Fig. 3D.. An ensemble with only networks achieved the best performance, while larger sembles performed worse, and Monte Carlo dropout (-0.85%) was better than ensembles of five (-1.00%) and ten networks (-1.5%%)
Early stopping: Finally, performing early stopping on the validation PU loss resulted in worse accuracy (-1.12%) compared using the accuracy on positive-negative labels (Fig. 3 Although considerable when compared to the impact of other algorithmic choices, such performance drop indicates that PUUPL. can be used effectively in real-world scenarios with no labeled validation data available.
4.4 ROBUSTNESS
Following the same protocol as the ablation studies in Section 4.3 method with respect misspecifications of the continuous hyperparameters (Fig.
we tested the robustness of our
Pseudo-labeling: Our method was fairly robust to the maximum number T' of assigned pseudo- labels and the maximum uncertainty threshold for the pseudo-labels, with almost constant per- formance up to T 1000 and t 0.1. The best performance was achieved by the combination having T 1000 and 0.05, but both of these experiments were performed while disabling the other constraint (i.e., setting T = inf when testing I and vice-versa). Using only a constraint T resulted in a reduction of -0.11% while constraining 4 alone resulted in reduction of -1.04%. The results for tu were less conclusive for the general trend, possibly because values lower than 0.35 require more than the 15 pseudo-labeling iterations we used for the experiment, and values above 0.4 did not show significant differences.
Misspecification of the class prior: The performance of our framework slowly degraded as the prior moved further from the true value of 0.4 with performance reduction of less than 2.5% in the range [0.3, 0.6] (Fig. 44) Furthermore, the performance gap between PUUPL and nnPU widens is more severly misspecified. Modern losses for PU learning such uPU and nnPU rely on the
8

---

Under review as a conference paper at ICLR 2022
nnPU
nnPUSB
Only PU loss PU los++PUUPL
87.05 (0.71)
87.31 (0.60)
87.70 (0.69)
87.91 (0.71)
Table 2: Test accuracy of our framework on the CIFAR-I0 dataset with a selection bias on the positive labels when using the nnPU and nnPUSB losses. Our framework improves over the base PU loss in both cases; in particular, PUUPL with nnPU loss achieved better performance than the nnPUSB loss alone.
correct estimation of the positive class prior π from domain knowledge or a priori estimation of n, which constitutes a whole research branch in PU learning (Chen et al., 2020a) and is significant challenge in any practical PU application (Bekker and Davis [020] Chen et al. 2020a). We believe that the inclusion of epistemic uncertainty, the usage of soft labels and the convex combination of two losses enables PUUPL to be considerably more robust to significant misspecification of the class prior
Loss combination: The best performing combination had À = 0. 0.15, with modest performance re- duction until À = 0.5 -0.25, Fig. 4b). Values of 0.05 and below resulted in the same performance reduction of -0.5%, similarly to A = 0.75, and performance was 1.09% worst at À = 0.9. Too large À might facilitate the emergence of harmful confirmation bias, but it is nonetheless important to train on the pseudo-labels, too, to avoid losing the information contained therein.
Number of training labeled positives: The performance of our method steadily increased and seemed to plateau at 91.4% between 3,000 and 6,000 labeled positives. The gap between nnPU and PUUPL is largest in the low labeled data region with a 1.44% gap at 250 labels, where achieved 87.59% accuracy, shrinking to a gap of 0.52% with 3,000 labels, where our performance was 91. 91.44% (Fig. 4b). This supports our intuition about the importance of uncertainty because, as the amount of labeled data decreases, uncertainty becomes more important to detect overfitting and to prevent the model from assigning incorrect pscudo-labels.
The most general assumption of PU learning is that the labeled examples are a biased sample from the positive distribution (Bekker and Davis 2020). We tested PUUPL in such biased setting where positives in the training and validation sets were with 50% chance an airplane, 30% chance an automobile, 15% chance ship and 5% chance truck, while in previous experiments the positives were evenly composed of airplanes, automobiles, ships and trucks. The test distribution was unchanged, meaning that test samples are half as likely to be airplanes compared to the training set, and five times more likely to be to be truck images. We also fixed all hyperparameters to the values identified previously, except for the loss Lru where we used the nnPUSB loss (Kato et al., 2019) to handle the positive bias. The baseline with nnPUSB loss performed better than the nnPD loss (+0.26%), but worse than PUUPL with the nnPU loss (-0.39%), highlighting the benefit of our uncertainty-aware approach. The best performance was however achieved with PUUPL on top of the nnPUSB loss (+0.21% % compared to nnPU and -2.27% compared to the unbiased setting). showing that PUUPL can leverage the advantages of different PU losses and further improve on them (Table
Positive bias:
CONCLUSIONS
In this paper, we proposed an uncertainty-awan pseudo-labeling framework for PU learning which quantifies the epistemic uncertainty of an ensemble of networks and selects which examples to pseudo-label based on their predictive uncertainty. We demonstrated the benefits of our approach on different data modalities and biased settings, achieving state-of-the-aa performance in all our benchmarks. We further conducted extensive ablation studies and investigated the robustness of our approach, showing it to be reliable in settings that are likely to be encountered in the real world, such bias in the positive data, the unavailability of labeled negatives validation data and the misspecification of the class prior π.
Most of the ethical concerns stem from the specific application and dataset. Here we have shown a certain robustness towards biased positive labels without providing a comprehen- sive assessment, therefore practitioners should always ensure, insofar possible, that the obtained
Ethics statement:

---

Under review as a conference paper at ICLR 2022
predictions are "fair" (with "faimess"" defined appropriately w.r.t the target application) and do not systematically affect particular subsets of the population of interest.
Reproducibility statement: The source code for the framework is available in the supplementary material.
REFERENCES
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 2021.
Eric Arazo, Diego Ortego, P. Albert, N. O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep con-ssuper learning. 2020 International Joint Conference on Neu- ral Networks (IJCNN, pages 1-8, 2020.
Harotune K Armenian and Abraham M Lilienfeld. The distribution of incubation periods of neo- plastic diseases. American journal of epidemiology, 99(2):92-100 1974.
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109(4):719-76)) 2020.
William H Beluch, Tim Genewein, Andreas Nürbberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 9368-9377, 2018.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2book reviews]. IEEE Transactions Neural Nenworks, 20(3):542-544 542, 2009.
Hui Chen, Fangqing Liu, Yin Wang. Liyue Zhao, and Hao Wu. A variational approach for learning from positive and unlabeled data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14844- 14854. Curran Associates, Inc., 2020a.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Confer- ence on Muchine Learning, pages 1510-1519. PMLR, 2020b.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourleenth International Conference Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Reseurch, pages 215-223, Fort Lauderdale, FL., USA, 11-13 html
Apr 2011. PMLR. http:ttpss//ppcceeeiigoo mlr wtt.::ss.....p.....emm.m wttppp/ssww/..aeeeeeessesee
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-)2,,,212,,2000
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. Advances in neural information processing systems, 27:703-711, 2014.
Marthinus C. du Plessis, Gang Niu, and Masashi Sugiyama. Class-prior estimation for learning from positive and unlabeled data. 106(4):463-492, nov 2016. doi: 10.1007/119994401-6666---
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 114 2021.
10

---

Under review as a conference paper at ICLR 2022
Yves Grandvalet, Yoshua Bengio, al. Semi-supervised learning by entropy minimization. NeurIPS, 367:281-296, 2005.
Yu-Guan Hsich, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and biased negative data. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages //proceedings. mll. wwt.:.aww.........e.....
PMLR, 09-15 Jun 2019. URL http:
html
Wenpeng Hu, Ran Le, Bing Liu, Feng Ji, Jinwen Ma, Dongyan Zhao, and Rui Yan. Predictive adversarial learning from positive and unlabeled data. In AAAI, 2021.
E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction concepts and methods. Much. Learn., 1 10:457-550, 2021.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5070-5079, 2019.
Hirotaka Kaji, Hayato Yamaguchi, and Masashi Sugiyama. Multi task learning with positive and unlabeled data and its application to mental state prediction. In ICASSP, pages 2301-2305, 2018.
Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with selection bias. International conference on learning representations, 2019.
Diederik P. Kingma and Jimmy Ba. abs/1412.69800, 2015.
Adam: A method for stochastic optimization. CoRR,
Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positive- unlabeled learning with non-negative risk estimator. CoRR, abs/1703.00593, 2017. URL
http://arxiv org abs/ aaa.//171..05555
Alex Krizhevsky, Geoffrey Hinton, et al. tiny images. Citeseer, 2009. URL
Learning multiple layers of features from https::/wwet tt//t/t.
Seariing-featureen 2009-0000-- TR. pdf
Balaji Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017a.
Balaji Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017b.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.
Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18:185:1-185:2. 2017.
Wenkai Li, Qinghua Guo, and Charles Elkan. positive and unlabeled learning algorithm for one- class classification of remote-sensing data. IEEE transactions on geoscience and remote sensing, 49(2):717-725, 2010.
Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip $ Yu. Building text classifiers using positive and unlabeled examples. In Third IEEE International Conference on Data Mining, pages 179- 186. IEEE, 2003.
Fangqing Liu, H. Chen, Liyue Zhao, and Hao Wu. Discriminative adversarial networks for positive- unlabeled learning. ArXiv, abs/ hass/996.00042.. 2019.
Chuan Luo, Pu Zhao, Chen Chen, Bo Qiao, Chao Du, Hongyu Zhang, Wei Wu, Shaowei Cai, Bing He, Saravanakumar Rajmohan, al. Pulns: Positive-unlabele unlabeled learning with effective negative sample selector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages $784-8792 2021.
11

---

Under review as a conference paper at ICLR 2022
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher
Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150
Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/www. web. http:/aatooooooooooo//0//
Anthony W. Purcell, Sri H. Ramarathinam, and Nicola Ternette Mass spectrome- based identification of MHC-bound peptides for immunopeptidomics. 1476):687-1177, may 2019. doi: 1000////1199-019-011101 2F SFs41596-019-0133
URL.
M. N. Rizve, Kevin Duarte, Y. Rawat, and M. Shah. In defense of pseudo-labeling: An uncertainty- aware pseudo-label selection framework for semi-supervised learning. ArXiv, abs/2101.06329, 2021.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pages 4393 4402. PMLR, 2018.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 299-315, 2018.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In Proceedings the IEEE Conference on Computer Vision and Puttern Recognition, pages 5552-5560, 2018.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learn- ing, 109(2):373 440, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv:1708.077477 2017
Yixing Xu, Chang Xu, Chao Xu, and Dacheng Tao. Multi-positive and unlabeled learning. In IJCAI, pages 3182-3188, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz Paz. mixup: Beyond empirical risk minimization. arXiv preprint 171.0017. 2017.
12

---

Under review as a conference paper at ICLR 2022
Hyper-parameter
Value range
Estimator
Ensemble MC Dropout
Number samples
[2. 25]
Uncertainty type
Aleatoric, epistemic, total [100, 5000
Max. new labels
Max. new label uncertainty
0, log 2
Min. unlabel uncertainty tu
00 log 2
Reassign all pseudo-labels
Yes
Re-initialize to same weights Cross-entropy weight X
Yes or no [0, 1]
Table A.1: Pseudo-labeling hyperparameters
Layer type
Layer parameters
LSTM
impus size=200,, hidden size=128, num layers=2, dropout=0.25, bidirectional-True p=0.2
Dropout
Linear
in fcatures=256, out fcatures=196, bias= True
Batch Norm.
eps=le-05, momentum=0.11
ReLU
Dropout
p=0.2
Linear in features=196, out features=196, bias=True
Batch Norm.
eps= eps=le-05, 1e-05, momentum=0.1
ReLU
Linear
in features=196, out features=1, bias=Trrue
Table A.2: Network architecture used for the IMDb experiments
APPENDIX
A NETWORK ARCHITECTURE AND HYPERPARAMETERS
Table repports the hyperparameters related to pscudo-labeling and their ranges. (epport the network architecture used in the IMDb experiments, while Table A.3 reports the network used with CIFAR-10.
B DATASET INFORMATION
Table B.T reports the number of samples for each split and each dataset. For the image datasets, we subtracted the mean pixel intensity in the training set and divided by the standard deviation. For IMDb we used pre-trained Glo Ve embeddings of size 200 on a corpus of six billion tokens.
13

---

Under review as a conference paper at ICLR 2022
Layer type
Layer parameters
Conv. 2D
in channels=3, out channels=96, kernel size=3 stride=1, padding=l p-0.15 eps=le-05, momentum=0.1
Dropout
Batch Norm.
ReLU
Conv. 2D
in channels=96, out channels=96, kernel size=3, stride=1, padding=1
Dropout
p=0.15
Batch Norm.
eps=le-05, momentum=0.1
ReLU
Conv. 2D
in channels=96, out channels=96, kernel size=3, stride=2, padding=1
Dropout
p-0.15
Batch Norm.
eps=le-05, momentum=0.1
ReLU
Conv. 2D
in channels=96, out channels=192, kernel size=3, stride=1, padding=1
Dropout
p=0.15
Batch Norm.
eps=le-05, momentum=0.1
ReLU
Conv. 2D
in channels=192, out channels=192, kernel size=3, stride=1, padding=1
Dropout
p=0.15
Batch Norm.
eps=1e-05, momentum=0.1
ReLU
Conv. 2D
channels=192, channels=192, kernel size=3, stride=2, padding=1
Dropout
p=0.15
Batch Norm.
eps=le-05, momentum=0.1
ReLU
Conv. 2D
in 19a out channels=192, kernel size=3, stride=1, padding=1
Dropout
p=0.15
Batch Norm.
eps=1e-05, momentum=0.1
ReLU
Conv. 2D
in 192n out channels=192, kemel size=1, stride11, padding=0
Dropout
p=0.15
Batch Norm.
eps=le-05, momentum=0.1
ReLU
Conv. 2D
in channels=192, channels=10, kerel size=1, stride=1, padding=0
Dropout
p=0.15
Batch Norm.
eps=le-05, momentum-0.1
ReLU
Flatten
Linear
in features=640,, out fcatures=1000, bias=True
ReLU
Linear
in Seatures=1000, out features= 1000, bias=True
ReLU
Linear
in features=1000, out features=l, bias=True
Table A.3: Network architecture used for the CIFAR-10 experiments
Dataset
Tot. Pos.
Tot. Neg.
Train Lab.
Train Unlab.
Val. size
Test Size
MNIST
30,508
29,492
1,000
54,000
5,000
10,000
F-MNIST
30,000
30,000
1,000
54,000
5,000
10,000
CIFAR-10
20.000
30,000
1,000
44,000
5,000
10,000
STL-10 ?
?
3,600
105,400
5,000
8,000
IMDb
12,500
12,500
1,000
19,000
5,000
25,000
Table B.1: Total size of the datasets and the data splits. As for STL-10,, used the entire set of examples, incuding the truly unlabeled ones.
14


Reviews:
[{"review_id": "uPKG6qtkV4O", "reviewer": "Reviewer_kfAD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies the PU learning problem. It proposes a two-step approach that can estimate the pseudo-label uncertainty so that more reliable pseudo-labels can be assigned, which improves the predictive performance. The proposed estimation method is different from previous methods. \n\nMy main concern is that two key assumptions are made in the paper, which are not realistic. The first one is that a positive and negative labeled validation set is available and used as the validation set for early stopping. The second is that the proportion of positive and negative samples is known. Both are problematic. The paper justifies the assumptions by listing some prior work that made one or the other assumption. However, I don’t think the fact that some prior work using these assumptions makes the assumptions right. In a real-life situation, these numbers are not available. There are many approaches that do not make any of the assumptions, but they are not compared, e.g., the recent system PAN, “Predictive adversarial learning from positive and unlabeled data,” AAAI-2021. \n\nPrior work has used several techniques to score the unlabeled set and then label them. Since the paper that proposed the two-step approach, “partially supervised classification of text documents,” ICML-2002, several approaches have been proposed to pseudo-label the unlabeled set. I understand that the proposed approach is different and probably more advanced than those older approaches, but it will be more complete to have an experimental comparison with some earlier approaches to show the advantage of the proposed method. \n\nThe paper is easy to understand. \n", "summary_of_the_paper": "This paper studies the PU learning problem. It proposes a two-step approach that can estimate the pseudo-label uncertainty so that more reliable pseudo-labels can be assigned, which improves the predictive performance. The proposed estimation method is different from previous methods. ", "summary_of_the_review": "Although the proposed estimation method is different from existing ones, my main concern is that the two assumptions are unrealistic. They actually run against the very setting of PU learning. Without the two assumptions, the proposed method would have problems. The paper should also compare with some earlier methods.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "migx_57__ns", "reviewer": "Reviewer_3wif", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using the epistemic uncertainty in PU learning is interesting. However, it is not clear the technical novelty and it lacks a simple baseline against uncertainty-aware pseudo-labeling.\n\n---\n\nWhat is the difficulty to introduce uncertainty-aware pseudo-labeling techniques to PU learning? It seems that simply applying uncertainty-aware pseudo-labeling technique results in the proposed method. That is, the technical novelty of the proposed method seems not strong. If the authors can clarify non-trivial technical contributions, it would help understand this paper well.\n\nComparison with uncertainty-unaware pseudo-labeling techniques helps understand the superiority of the proposed method. It is expected to illustrate that considering uncertainty is important when we use pseudo-labeling in PU learning. For example, nnPU with ordinary pseudo-labeling techniques could be compared with nnPU with the proposed uncertainty-aware pseudo-labeling.\n\nIn Section 3.2, there is the sentence \"we assume cleanly labeled (positives and negatives) validation and test sets (see Section 3.6) as usual in literature.\" I checked the cited papers quickly, but could not find such a setting. If negative samples are available, we can use them (e.g., a half of them) in training. And, it might result in better performance thanks to negative samples. So, the assumption sounds not practical. Thus, the results reported in the experiments are not good evidence of the effectiveness of the proposed method. But, there is a chance that I miss the sentences supporting the authors' assumption. If the authors can show the sentences describing the assumption about the validation set, I will confirm it.\n\n=== After rebuttal ===  \nThank you for your responses.  \nThe idea of using the epistemic uncertainty in PU learning is interesting. So, I hope that the authors will resolve my and the others' concerns in the next venue.", "summary_of_the_paper": "This paper proposes PUUPL, an uncertainty-aware pseudo-label selection method for positive-unlabeled (PU) learning. To improve the performance of pseudo-labeling, the authors suggest using the epistemic uncertainty (the difference between the entropy of the mean prediction and the mean of entropies of each prediction). In the experiments, PUUPL outperformed the existing state-of-the-art PU learning methods.", "summary_of_the_review": "This paper introduces uncertainty-aware pseudo-labeling techniques to PU learning. Although the idea is interesting, the technical novelty and the performance improvement seem marginal. In addition, according to the paper, it assumes cleanly-labeled validation data (we can access negative samples) are available, which is a strange assumption in PU learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NcCwVgCauke", "reviewer": "Reviewer_9jfD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Trivial motivation:\n\nIn Abstract, the authors say that \"two-steps procedures are vulnerable to incorrectly estimated pseudo-labels\" and to mitigate this issue they propose this method. It is well known that many important PU learning methods are not two-step, such as nnPU.\n\nLimited novelty:\n\nFirst, I am not convinced that the proposed method is novel. The key techniques used are transferred from some existing methods (from Sec.3.2-3.6).\nSecond, I don't understand the meaning of designing an algorithm like this. For example, it uses an unbiased risk (Kiryo et al., 2017), which can already guarantee the consistency of learning. But complicating the learning procedure makes this loss lose its original advantage, and an additional regularization method has to be added. And it labels the instance according to the epistemic uncertainty (Eq.4-6). It is unclear why this measure is used and what the advantages are. The algorithm design is almost at random, at least from the current description. In addition, I think the complex algorithm design does not result in significant performance improvements.\n\nMinor concerns:\n\n- What is Eq.(3) for?\n- The writing of the paper can be improved, e.g., \"two-steps procedures\"->\"two-step procedures\"\n", "summary_of_the_paper": "This paper proposed a method to learn with PU data which quantifies the epistemic uncertainty of an ensemble of networks and selects which examples to pseudo-label based on their predictive uncertainty. The authors propose to use the pseudo-labeling technique based on the uncertainty of the prediction, combined with early stopping.", "summary_of_the_review": "The novelty is limited and the contributions are incremental. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H-euD9RxY5H", "reviewer": "Reviewer_G93N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ A simple uncertainty-aware pseudo-labeling framework for PU learning is proposed.\n+ The proposed method is general and can be integrated into any loss function for PU learning.\n+ This paper is easy to follow and the logic is clear. \n\nWeaknesses\n+ The novelty of this paper may not be enough. Specifically, the pseudo-labeling techniques have been well studied in a lot of weakly surprised learning scenarios such as label noise learning and semi-supervised learning. \n+ The motivation of the metric used to quantify the uncertainty is not clear. The authors should give more explanations of the advantage of the proposed quantified metric. \n+ The notations in Section 3.3 (which is the most important section) may cause confusion. For example, is it a sample x_i or example x_i? Is \\hat{p}_{i} a vector or scalar? How about \\hat{p}_{ik}?\n+ Only four baseline methods are used in experiments. I think it is better to add more.", "summary_of_the_paper": "The authors proposed to use pseudo-labels based on high confidence predictions to improve the classification performance of PU learning. Experiments show the effectiveness of the proposed method.", "summary_of_the_review": "I think the research problem is interesting. However, I think the technical novelty is marginally significant, and the motivation of the proposed metric may not be clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new loss-agnostic PU learning method based on uncertainty-aware pseudo-label selection.
I would like to thank the authors for their feedback to the initial reviews, which clarified many uncertain issues and improved our understanding of the current paper.
Nevertheless, even if the pseudo-labeling technique was applied to PU learning for the first time, given that it is a common practice in many weakly supervised learning tasks, the technical novelty is rather limited.

Therefore I cannot recommend acceptance of this paper.
Decision: Reject