Improving State-of-the-Art in One-Class Classification by Leveraging Unlabeled Data
Authors: Farid Bagirov
Dmitry Ivanov
Aleksei Shpilman
Venue: ICLR 2022 Submitted (2022)

Abstract: Recent advances in One-Class (OC) classification combine the ability to learn exclusively from positive examples with the expressive power of deep neural networks. A cornerstone of OC methods is to make assumptions regarding negative distribution, e.g., that negative data are scattered uniformly or concentrated in the origin. An alternative approach employed in Positive-Unlabeled (PU) learning is to additionally leverage unlabeled data to approximate negative distribution more precisely. In this paper, our goal is to find the best ways to utilize unlabeled data on top of positive data in different settings. While it is reasonable to expect that PU algorithms outperform OC algorithms due to access to more data, we find that the opposite can be true if unlabeled data is unreliable, i.e. contain negative examples that are either too few or sampled from a different distribution. As an alternative to using existing PU algorithms, we propose to modify OC algorithms to incorporate unlabeled data. We find that such PU modifications can consistently benefit even from unreliable unlabeled data if they satisfy a crucial property: when unlabeled data consists exclusively of positive examples, the PU modification becomes equivalent to the original OC algorithm. Our main practical recommendation is to use state-of-the-art PU algorithms when unlabeled data is reliable and to use PU modifications of state-of-the-art OC algorithms that satisfy the formulated property otherwise. Additionally, we make a progress towards distinguishing the cases of reliable and unreliable unlabeled data using statistical tests.

Under review as a conference paper at ICLR 2022
IMPROVING STATE-OF-THE-ART IN ONE-CLASS CLASSIFICATION BY LEVERAGING UNLABELED DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent advances in One-Class (OC) classification combine the ability to learn exclusively from positive examples with the expressive power of deep neural net- works. A cornerstone of OC methods is to make assumptions regarding negative distribution, e.g., that negative data are scattered uniformly or concentrated in the origin, An alternative approach employed in Positive- Unlabeled (PU) learning is to additionally leverage unlabeled data to approximate negative distribution more precisely. In this paper, our goal is find the best ways to utilize unlabeled data top of positive data in different settings. While it is reasonable to expect that PU algorithms outperform OC algorithms due access more data, we find that the opposite can be true if unlabeled data is unreliable, i.e. contain negative examples that are either too few or sampled from different distribution. As an alternative to using existing PU algorithms, we propose to modify OC algorithms to incorpo- rate unlabeled data. We find that such PU modifications can consistently benefit even from unreliable unlabeled data if they satisfy crucial property: when unla- beled data consists exclusively of positive examples, the PU modification becomes equivalent to the original OC algorithm. Our main practical recommendation is to use state-of-the-ar PU algorithms when unlabeled data is reliable and to use PU modifications of state-of-the-art OC algorithms that satisfy the formulated prop- erty otherwise. Additionally, we make a progress towards distinguishing the cases of reliable and unreliable unlabeled data using statistical tests.
INTRODUCTION
An input of supervised binary classifier consists of two sets of examples: positive and negative. However, the access to clean samples from both classes can be obstructed in many realistic scenar- ios. A particularly well-studied restriction is the absence of clean negative examples. One of the approaches to deal with this restriction is One-Class (OC) classification (Moya et al., 1993). Typ- ically, OC algorithms treat available positive examples normal data and try separate it from previously unseen data, referred to as anomalies or outliers (Grubbs) 1969 Hodge and Austin 2004 Chandola et al.) 2009| Chalapathy al., 2018 Chalapathy and Chawla 2019). As noted in other studies (Scott and Blanchard 2009 Ruff et al. 2020a|b) b). OC algorithms necessarily make prior as- sumptions about negative distribution. For example, some methods assume negative distribution to be uniform (Vert et al. 2006), Scott and Nowak 2006) or concentrated where positive data are rare (Tax and Duin] [204] Ruff et al. (2018), model negative distribution as a Gaussian (Oza and Patel] 2018), or separate positive data from the origin (Schölkopf et al., 2000).
[004]
Another approach classification the absence of negative data is Positive- Unlabeled (PU) lear- ing (Denis) 1998 Denis et al. 2005 Li and Liu 2005). In addition to a clean positive sample, PU algorithms leverage an unlabeled set of mixed positive and negative examples. In contrast to OC
methods, PU methods approximate negative distribution (Elkan and Noto 2008 Ivanov]
2019). its
statistics (Du Plessis et al. 2014: 2015 Kiryo et al., 2017), or samples from Yuu et al. 2002 Liu
al. 2002 Li and Liu| 2003] Xu et al. 2019), by comparing positive and unlabeled data. This
approach can even outperform supervised classification, given a sufficient amount of unlabeled data (Niu et al. 2016 Kiryo et al. (2077).
Because PU methods make fewer assumptions and have access to more data, they might seem fa- vorable to OC methods whenever unlabeled data hand. However, our experiments show that

---

Under review as a conference paper at ICLR 2022
pos labeled pos
pos Inbeled
labeled dasciller
DOs lazeled
dassiller
(a) OC (no shift) ROC AUC=0.92
(b) PU (no shift) ROC AUC=0.97
(c) OC (negative shift) ROC AUC=0.91
(d) PU (negative shift)
ROC AUC=0.08
Figure 1: Performance of OC and PU models on synthetic data. Subfigures (a) and (b) show the performance of the algorithms in the standard case when negative distribution does not change from train to test. Subfigures (c) and (d) show the effects of negative distribution shift. The OC and PU models are OC-SVM and PU-SVM, respectively (section 3.2.1p. The OC model is trained on labeled positive examples. The PU model is trained on labeled positive as well as unlabeled examples. i.c. mixture of positive and negative examples. Black dashed lines represent the decision boundaries of the optimal Bayesian classifier. Overdependence on unreliable unlabeled data causes the PU method misclassify all anomalies when shift of negative distribution occurs.
dependence on unlabeled data may hinder PU methods in particularly extreme cases, which we re- fer to as cases of unreliable unlabeled data. We identify several such cases, including distributional shifts in unlabeled data, scarcity of unlabeled data, and scarcity of latent negative examples in un- labeled data. We present motivational example of a possible effect of distributional shifts on PU models in Figure In subplots (a, b) a PU model approximates the separating line more accurately than an OC model. Conversely, in subplots (c, d) shift of negative distribution causes the PU model to misclassify all negative examples, whereas the OC model unaffected by the shift.
One possible conclusion is to opt for using OC methods when unlabeled data is unreliable, but we aim to find a principled way to construct robust PU methods that can learn even from unreliable unlabeled data. To this end, we propose modifications of several modern OC algorithms that lever- age unlabeled data. These modifications either are based on risk estimation techniques where the negative risk is approximated using positive and unlabeled data (Kiryo et al. 2017), or simply place positive data with unlabeled in Algorithm-spectie that all modifications benefit from reliable unlabeled data, but only modifications from the second group are safe to apply to unreliable unlabeled data (i.e. either improve upon or perform on par with the original OC methods). We pinpoint this result crucial property that only the modifications from the second group possess: in the absence of latent negative examples, the PU modification becomes equivalent the original OC algorithm. Our main practical recommendation is use ofate-of-the-at PU algorithms when reliable unlabeled data is available and to use modifications of state-of-the-art OC algorithms that satisfy the property when robustness a concern.
routines (see Sections 3.2.4
We find
A question remains how to identify the cases of unreliable unlabeled data. We find that scarcity of latent negatives can be statistically tested by comparing predictions of OC or PU models for positive and unlabeled samples. Similarly, a shift of negative distribution can be tested by comparing predictions for unlabeled data from training and testing distributions, providing the latter is available.
Related Work There is a line of work in the OC literature that investigates ways to augment OC methods with additional data. Several studies find that exposure to small and possibly biased sample of outliers can improve the performance of OC classifiers (Hendrycks et al., 2018 Ruff et al. 2019 2020b). Scott and Blanchard (2009) show both thcoretically and empirically that unlabeled data help a classic machine learning algorithm to detect novelties. Nevertheless, there is a lack of modern literature that views OC and PU learning different solutions to the same problem and gives practical recommendations for different scenarios. Our study attempts to fill this gap.
The problem of unreliable unlabeled data is also underexplored in PU literature. One existing di- rection is concerned with robustness to class prior shift (Charoenphakdee and Sugiyamal 2019). While this is useful property, we study more severe case of arbitrary negative distribution shifts.
2

---

Under review as a conference paper at ICLR 2022
Furthermore, an increase in proportions of positive class be seen special case of negative distribution shift. We implement the latest PU method robust to class prior shifts as a baseline (Naka- jima and Sugiyama, 202Th. Other papers and Lowd, (020) and covariate shifts (Sakai and Shimizu 2019), but unlike us, they assume access unbiased unlabeled data from the testing distribution. To the best of our knowledge, we address the problem of PU learning with shifted or scarce latent negatives for the first time.
concerned with arbitrary positive shifts (Hammoudch
Our study is conceptually similar to the field of safe semi-supervised learning where the focus is on making unlabeled data never hurt while retaining performance (Li and Zhou 2014 Li and Liang,
2019
Guo et al. 2020). The difference is that we do not assume access labeled negative data.
PROBLEM SETUP
Let E Fd be data point and let y labeled and 0 otherwise. We assume that only positive data can be labeled, i.c. p(s = 1ly = 0) = O. We view æ, y, random variables with some joint distribution f(x, U,,, The probability density functions of positive or negative distributions are given by:
{0,1}
1} be a binary label. Let 1 if the data point is
fx(x) = f y 1,8 0)
(1a)
f=(r)== = 0, =)
(1b)
Let (x(x)) be the probability density function of unlabeled distribution, i.e. a mixture of positive and negative distributions, and a ply 1|s 0) be the mixture proportion. We assume that the probability of being labeled p(s = 1 | 1) is constant and independent of æ, which is known as Selected Completely At Random (SCAR) setting (Elkan and Noto 2008). In this case:
fu(x) = f(z|s 0) =ff(())))))))))))==))))<))) + (1-
(2)
In OC methods we assume that only sample X, from fn(x)) is available. In PU methods we sume the case-control scenario (Bekker and Davis 2020 two sets of data Xp and X, are sampled independently from fp(x)) and fz)... Both OC and PU methods output some score function pro- portional to p(x) == fly = 1 ar, 8 = 0) that separates positive and negative data. PU methods often additionally estimate a. Since is generally unidentifiable (Blanchard al.) 2010), we will focus on estimation of its upper bound a", as proposed in (Jain et al. 2016).
3 METHODS
In this section, we first describe PU methods used in this study. We begin by describing Risk Estimation approach, which some of our PU modifications of OC algorithms are based on. Then, we describe DRPU and PAN, which we use for comparison as the most recent and state-of-the-art PU algorithms. In Appendix A.5), we elaborate on this choice state-of-the-an by comparing these algorithms with other existing PÚ algorithms. Finally, we describe OC methods used in this study and propose their PU modifications. Additional details reported in Appendix B
3.1 POSITIVE-UNLBBLER METHODS
3.1.1 RISK ESTIMATION
Let h(x) be an arbitrary decision function that estimates y. ((t,v) y) be loss function, i.c. the loss incurred for predicting when the ground truth is y. Define R. (h) = Ea-s, (1((x)), 1)) and RT (h) = Ez-,, (I(h(z),0) 0)) as positive and negative risks. If both positive and negative data are available, the risk of the decision function
be estimated negative risks (eq. Ba). In PU case, (h) is unavailable but can be estimated as a difference between risks on positive and unlabeled data (eq. 3bb, as shown in (Du Plessis et al. 2014 2015).
weighted
of positive and
Rpm(h) = (1)a (h) (3a)
Rmu(h) aR2 (h) aR (h) R (h) (3b)
3

---

Under review as a conference paper at ICLR 2022
Estimator (3b) can be improved by introducing non-negativity constraint to reduce overfitting Kiryo et al. 2017):
Ryo (h) = a ( +
(h))
(4)
Estimator 44 is called non-negative risk estimator. In practice, the decision function h is parame- terized by 6, which can represent weights of neural network some other model. The parameters are trained to minimize Ron ( (h(x 0)) for some loss function like double hinge (Du Plessis et al., 2015) or sigmoid (Kiryo et al., 2017). We use the latter. Notice that a is assumed to be identified in this method, so in experiments, we additionally estimate it with TIcE (Bekker and Davis 2018).
be applied modify any OC model to leverage unlabeled data, providing this OC model is based on or can be generalized to the supervised (PN) setting. This can be done by first replacing the OC objective with the PN objective, and then applying risk estimation the PN objectíve, i.e. evaluating negative risk using positive and unlabeled samples.
Risk estimation
3.1.2 DRPU
DRPU (Nakajima and Sugiyama 2021 is a state-of-the-art extension of Risk Estimation technique that relaxes the assumption identified mixture proportions by leveraging density ratio estimation. Specifically, it minimizes the Bregman divergence between the posterior probability p(z) and its parametric estimate, As result, DRPU is stable to the shifts of class proportions in unlabeled data.
3.1.3 PAN
Predictive Adversarial Network (PAN) (Hu et 2021) is another state-of-the-art PU algorithm based on a GANs (Goodfellow et al. 2014). In this algorithm, the generator is replaced with a classifier trained to identify unlabeled examples that are likely to be considered positive by the dis- criminator. The discriminator then tries to maximize KL-divergence between its and the generator's predictions while correctly classifying labeled positive examples.
3.2 ONE-CLASS METHODS
3.2.1 OC-SVM
OC-SVM (Schölkopf al. 2000 one of the most well-known OC methods. OC-SVM modi- fies the classical objective of Support Vector Machines and tries to separate the nominal data from the origin. In the case of RBF kernel (and other mansllatio-invariinn kernels (Schölkopf et al. 2001h, OC-SVM tries to enclose positive data while minimizing enclosed volume and is equivalent to SVDD (David 2001 Tax and Duin, 2004). Similarly to SVM, OC-SVM can optimized by solving the dual problem and specifying only dot product in the feature space. OC-SVM solves the following optimization problem:
min W..
max(0, r- w ((x)) -
(5)
PN
where O is feature map in some Hilbert space, // both regularization parameter and a correction for possible data contamination, and (fx,)) - is a decision function. Unlike most OC papers, we use linear kernel instead of RBF kernel in our experiments. During preliminary experiments, we discovered that such replacement improves ROC AUC of OC-SVM on benchmark datasets.
PU As Plessis et al. 2015)) show, the linear models can be trained PU setting by minimizing risk defined in (3b). Since SVM is linear model that minimizes empirical risk, one use it as h(x)) and apply risk estimator techniques. In Appendix 1.1.2 we show that similarly to SVM and OC-SVM it can be optimized by solving the dual problem, but that all labeled positive examples are support vectors in PU-SVM. As a faster alternative. we optimize non-negative risk 4 with Stochastic Gradient Decent (SGD), the details of which reported in Appendix B.I.
4

---

Under review as a conference paper at ICLR 2022
3.2.2 OC-CNN
OC-CNN (Oza and Patel 2018) uses pretrained model like ResNet frozen feature extractor. In the feature space, cero-centered Gaussian is used as the pseudo-negativy distribution. After that, a supervised classifier is trained to distinguish positive and pseudo- examples. As in the original paper, we use ResNet pretrained on ImageNet as feature extractor. Since pretraining done supervised manner and classes from CIFAR10 10 overlap with classes from ImageNet, OC-CNN should not be directly compared with other OC algorithms that do not have access such additional data. The same logic applies to the proposed PU modification.
PU Since OC-CNN assumes particular negative distribution, it is easy to modify for PU setup. We replace the pseudo-negatio examples in the latent space with unlabeled examples processed through the same feature extractor and then perform the standard risk estimation procedure, i.c. minimize non-negative risk in the latent space (4).
3.2.3 OC-LSTM
OC-LSTM (Ergen et al. 2017) combines Long Short-Term Memory (Hochreiter and Schmidhuber 1997) and OC-SVM. First, the data points are processed through LSTM block, after which OC- SVM is applied to the last hidden state of LSTM. One way to optimize such model is to freeze its components iteratively. Alternatively, OC-SVM can be optimized with SGD and trained end-to-end with LSTM. We employ the latter approach.
PU
adapt OC-LSTM unlabeled data, we replace OC-SVM with our PU-SVM.
3.2.4 DROCC
DROCC (Goyal et al, 2020 achieves state-of-the-art performance on several real-world datasets across different domains. The key idea behind DROCC is that positive data lie low-dimensional locally Euclidean manifold. Based on this assumption, the pseudo-negative examples can be gen- erated from available positives via gradient ascent, similarly to adversarial attacks. After that, classifier is trained to minimize cross-entropy loss on positive and pseudo-negative examples.
PU In PU modification of DROCC, we generate pseudo-negative points from unlabeled rather than positive examples. This way, the pseudo-negative examples are closer to the real negatives unlabeled data contain some negative points, i.e. if 1. the extreme case when unlabeled data contain only positives, our modification is equivalent to the original algorithm.
3.2.5
CSI
CSI (Tack et al. 2020 is an OC method based on contrastive learning. In addition to contrasting given example with other instances as in conventional contrastive learning, CSI contrasts the exam- ple with distributionalll-shiifii
augmentations of itself, such rotations or permutations of parts of an image. The original study uses ResNet as neural architecture, but we train a small convolutional network instead for fair comparison with other algorithms.
PU In our PU modification of CSI, we additionally contrast positive points with examples from unlabeled data. Like our PU-DROCC, such modification will be close to the original model in the extreme case when unlabeled data has no negative examples, ie. 1.
4 DATASETS
Here we briefly describe the datasets used in this study. For details, please refer to Appendix
CIFAR-10 CIFAR-10 is a standard for both OC and PU methods benchmark dataset with images from animal and vehicle classes (Krizhevsky al. 2009). There are around 6000 images for each class and the proportion of negatives and positives depends on the particular experiment.
This dataset consists of abnormal images from six classes, including Chair, Car, Airplane, Boat, Sofa, and Motorbike (Saleh et al. 2013). Normal images come from the respective
Abnormal1001
5

---

Under review as a conference paper at ICLR 2022
classes from the PASCAL VOC dataset (Everingham et al. 2010). An example of images from this We perform experiments only with Car class since has the most examples, with 110 abnormal and 1315 normal images, 0.87. This dataset is challenging for PU methods since the negative data are very scattered and only a few negative examples are available.
dataset is presented in Figure 2
Pendigits Pendigits dataset consists of consecutive pixels sampled from digits handwritten on a tablet (Asuncion and Newman 2007). The task is to classify the digits based on these sequences of pixels. We use processed dataset from ODDS with 6870 examples total and 156 anomalies, 0.97.
Text Datasets We use two different datasets with text data. The first is standard dataset of decep- tive reviews (Ott et al. 2011). It consists of 400 deceptive hotel reviews acquired with crowdsoure- ing service and 400 truthful reviews from TripAdvisor, 0.33. The second is SMS Spam dataset (Asuncion and Newman 2007). It contains 5572 examples with 747 spam messages considered as anomalies, 0.76.
EXPERIMENTS
this section, we describe experimental settings and report results. We repeat each experiment 10 times. Statistical significance is verified via paired Wilcoxon signed-rank test with 0.05 P-value threshold. We indicate a significant difference in favor of PU or OC method with a line under the higher ROC AUC value. As noted before (Section 3.2.2.. the performance of OC-CNN and PU- CNN should not be directly compared to other models. Thee selected hyperparameters and tuning procedure are reported Appendix D Additional experiments are reported Appendix
5.1 ONE-VS-ALL
common experimental setting in OC papers for datasets with multiple classes one-vs-all clas- sification. In this setting, one class is treated as positive and all other classes constitute negative examples. We fix a in this setting as 0.5, label half of the positive points, and conduct experiments with each class selected as positive. Since OC methods have access to only half of all positive points, their performance cannot be directly compared to the results reported in their original papers. ROC AUC metrics for OC and PU methods for all classes are presented in Table [ PU modifications consistently significantly outperform their OC counterparts with few exceptions where methods per- form par. Furthermore, DRPU and PAN outperform other algorithms most classes, although on classes 7 through 9 our PU-CSI performs exceptionally well.
Table I: ROC AUC in one-vs-all setting
Pos class
0
2
4
5
6
8
9
avg
OC-SVM
0.78
0.67
0.60
0.67
0.63
0.64
0.74
0.66
0.68
0.75
0.68
PU-SVM
0.79
0.70
0.62
0.68
0.65
0.72
0.74
0.70
0.77
0.82
0.72
CSI
0.70
0.87
0.63
0.62
0.60
0.83
0.73
0.80
0.86
0.89
0.76
PU-CSI
0.71
0.90
0.61
0.68
0.62
0.83
0.79
0.92
0.89
0.91
0.79
DROCC
0.76
0.74
0.64
0.61
0.71
0.67
0.75
0.71
0.78
0.79
0.72
PU-DROCC
0.80
0.85
0.75
0.74
0.76
0.77
0.81
0.79
0.84
0.83
0.79
OC-CNN
0.74
0.80
0.64
0.69
0.77
0.76
0.81
0.73
0.78
0.83
0.76
PU-CNN
0.91
0.94
0.87
0.86
0.90
0.90
0.94
0.92
0.95
0.93
0.91
DRPU
0.85
0.89
0.78
0.78
0.81
0.82
0.88
0.83
0.88
0.87
0.84
PAN
0.85
0.89
0.76
0.77
0.80
0.80
0.88
0.82
0.88
0.87
0.83
5.2 SHIFT OF THE NEGATIVE DISTRIBUTION
While in the previous setting PU algorithms have shined, that setting is convenient in that unlabeled data contain reliable latent negative examples. In the next setting, we investigate the effect neg- ative distribution shift, i.e. when latent negative examples sampled from different distributions

---

Under review as a conference paper at ICLR 2022
Table 2: ROC AUC for different modalities of negative distribution under random negative shifts on CIFAR-10
Neg modality
2
4
OC-SVM
0.76
0.78
0.80
0.748
PU-SVM
0.70
0.78
0.80
0.752
CST
0.68
0.74
0.69
0.69
PU-CSI
0.73
0.76
0.70
0.70
DROCC
0.73
0.74
0.73
0.72
PU-DROCC
0.73
0.77
0.77
0.75
OC-CNN
0.72
0.71
0.74
0.74
PU-CNN
0.80
0.78
0.85
0.83
DRPU
0.74
0.77
0.80
0.81
Examples of abnormal car (left) and normal car (right) from Abnormal1001.
Figure 2:
PAN
0.74
0.79
0.81
0.81
at train and test times. For example, this can relevant if the negative distribution constantly shifts over time and the model cannot be retrained at each time step, or if a large but biased unlabeled sample available for training. whereas an unlabeled sample used for inference is either too small or for some reason unavailable. Since negative shift only affects unlabeled data, only PU methods are expected to be sensitive to it (Fig. []. Further, we suspect that the effect of negative shift on PU methods might decrease with the increased modality of the negative distribution due negative data covering more latent space, so we additionally vary the number of negative classes.
For this setting, we consider class from CIFAR-IO as positive, random classes as negative train, and n other random classes as negative test, where n E {1, 2, 2,3, 3,4.. The results are presented in Table Despite the distribution shifts that negatively affect PU methods, most PU modifications either significantly outperform or perform on par with their OC counterparts. The exception is OC-SVM, which outperforms all other algorithms when n Evidently, even incorrect but data- driven estimates of the negative distribution used by PU algorithm are average more helpful than the generic assumptions made by OC algorithms. Furthermore, for state-of-the-an PU algorithms increasing the number of negative classes partially mitigates the effect of negative shifts.
Although PU models still outperform their OC counterparts when the shift random, there are particular shifts that can significantly harm the performance of PU algorithms. We present results of experiments with non-random negative shifts, such as animal- animal, animal-vehicle, vehicle- animal, and vehicle-vehice in Table 3 The results vary from algorithm to algorithm. OC-SVM and in some cases OC-CNN outperform their PU modifications, highlighting that modifications based on risk estimation are not robust negative distribution shifts. PU-CSI shows the same performance as regular CSI, but both perform poorly compared to other models. This is likely due to CSI performing poorly on some classes chosen positive (Table I. we fix the positive class as in the current setting. On the other hand, PU-DROCC significantly outperforms its OC counterpart in most cases and at least performs on par in other cases. also outperforms both offe-of-thheart PU algorithms in all cases but one. Indeed, this algorithm shows robustness to shifts in latent negative distribution. Experiments with a different class chosen as positive are reported in Appendix
5.3 SIZE AND CONTAMINATION OF UNLABELED DATA
Other cases when PU algorithms may struggle are when negative examples are scarce, i.c. when the positive class proportion is high or the sample size of unlabeled data is small. We study the perfor- mance of OC and PU methods with respect a and the size of the unlabeled sample. We focus on state-of-the--ar algorithms as well as some baselines, leaving other experiments to Appendix [A] We choose all vehicles as positive classes and all animals as negative classes. The results are presented in Figure 3 Subpiot (b) shows that even as few 50 unlabeled examples are sufficient for PU models to outperform DROCC, but that both versions of CSI outperform other algorithms until the unlabeled sample becomes bigger than 1000 examples. Subplot (a) shows that ofate-of-the-att PU methods struggle when unlabeled examples are mostly positive. contrast, PU-DROCC consis- tently outperforms its OC analog. While both versions of CSI perform similarly, they outperform all

---

Under review as a conference paper at ICLR 2022
Table 3: ROC AUC for particular shifts with Airplane class (0) chosen as positive. We test shifts for unimodal and multimodal negative distributions. We consider following multimodal negative distributions: A f2,3, 3, 4), A2 (5. 6, 7), and V [1,8,9}
Shift
1 2
1
1
8
3
V1 A
A
V1
A + A2
OC-SVM
0.80
0.76
0.51
0.86
0.84
0.64
0.86
PU-SVM
0.77
0.63
0.55
0.79
0.78
0.57
0.85
CSI
0.75
0.54
0.46
0.54
0.75
0.49
0.80
PU-CSI
0.76
0.56
0.47
0.56
0.74
0.49
0.79
DROCC
0.74
0.76
0.60
0.80
0.75
0.68
0.79
PU-DROCC
0.75
0.79
0.66
0.85
0.77
0.69
0.87
OC-CNN
0.80
0.60
0.50
0.80
0.81
0.56
0.82
PU-CNN
0.63
0.66
0.65
0.92
0.69
0.71
0.96
DRPU
0.63
0.59
0.67
0.73
0.92
0.59
0.61
PAN
0.61
0.58
0.66
0.72
0.93
0.62
0.63
0.5
0.90
0.90
0.85
ROC AUC
ROC AUC
0.75
BROCC
0.75
BBOCE
0.70
OC-SVM
PU-DROCC
070
PAN
PU-OROCC
PAN
0.65
DAPU
CSI
MUACSS
0.00
055
0.50
50
250
1000
2000
5000 13000 2000
(a)
(b)
Figure 3: ROC AUC with respect to the proportion of positive examples in unlabeled data (a) and the size unlabeled data (b).
other algorithms when ar is high. Note that when a = 0.05, the difference between CSI and PU-CST is statistically significant in favor of the latter. Also note that CSI generally performs much better than in the previous settings, as its performance varies with the selected positive classes. As a final note, investigate the effect of modality of positive data in more detail in Appendix[A.I]
5.4 ABNORMALION
The results of the experiments Abnormal are presented in Table 4 Although this dataset is chal- lenging for PU methods due to the scarcity of anomalies and similarity of nominal and anomalous data, PU modifications still outperform OC counterparts across all models. Furthermore, PAN and our PU-DROCC perform the best among all methods.
Table ROC AUC on Abnormal 1001
Base
OC
PU
Table 5: ROC AUC on sequential data
SVM
0.60
0.64
Model
PenDigits
Deceptive
SMS Spam
CSI
0.55
0.64
DROCC
0.60
0.84
OC-LSTM PU-LSTM
0.96 0.97
0.52
0.69 0.92
CNN
0.53
0.76
0.77
DRPU
0.78
PAN
0.84
8

---

Under review as a conference paper at ICLR 2022
0.5
AU-SMM PU-OROCC DROCC
PU-SVM
PLOOROCC
0,4
110 shilt
0.3
0.3
D-valle
p-valuee
02
0.2
0.1
0.1
0.6
3.5
(a)
(b)
Figure 4: P-values of Mann- Hann-Whitney U-test for identification of a) high or b) negative shift. We show results for OC-SVM and DROCC as OC models and PU-SVM and PU-DROCC as PU models.
5.5 SEQUENTIAL DATA
Comparison LSTM-based models on datasets with sequential data can be found in Table 5 PU models achieve better scores than OC algorithms all three datasets. Furthermore, OC-LSTM performs poorly on the Deceptive Reviews dataset, which has very few labeled positive examples.
5.6 IDENTIFYING UNRELIABLE UNLABELED DATA
As our experiments show, some PU algorithms perform poorly when available unlabeled data are unreliable. Here, we propose a method to distinguish such cases. Empirically, PU models can struggle when is too high (a 0.9), when negative shifts occur, when only few unlabeled ex- amples are at hand. We discover that statistical tests can help with the first two problems. Applying Mann-Whinn Whitney U-test the outputs of OC PU models for unlabeled and positive samples can identify a is high (fig. a).. Similarly, applying Mann-Whithe for train and unlabeled samples can help detect distribution shifts. Fig. (b) shows p-values for samples with and without negative shifts, which only overlap when a is high.
U-test on outputs of PU models
CONCLUSION
this paper, we propose new PU algorithms based on existing OC algorithms, compare these and state-of-the-aa -frth--rrre- PU algorithms in different settings, and formulate general guideline for construct- ing a PU algorithm robust to unreliable unlabeled data, i.c. that should become an OC algorithm when unlabeled data close being purely positive. We find that algorithms that follow this guideline never underperform their OC counterparts, but that safety can come at a cost as PU SO- TAs generally outperform our algorithms when unlabeled data reliable. Furthermore, that our PU modifications based on risk estimation consistently improve upon OC algorithms when unlabeled data are reliable. This approach is applicable any OC algorithm that has a supervised analog and therefore presents another principled way to leverage unlabeled data that does not re- quire drastic changes of the algorithm, albeit does not guarantee improvement if unlabeled data are unreliable. We hope that our findings will motivate future researchers to investigate augmentations of their OC algorithms that leverage unlabeled data, possibly by following our guideline.
find
On the other hand, to the best of our knowledge, our paper is the first to explicitly formulate the cases of unreliable unlabeled data and to investigate the potential vulnerability of PU algorithms to such cases. While we make progress towards identifying such cases via statistical tests, more desirable alternative a PU algorithm that achieves ofate-of-the-art performance regardless of whether unlabeled data reliable. We hope that future researchers will consider testing the robustness of their PU algorithms in the settings unreliable unlabeled data, as well as proposing safe alternatives to their PU algorithms that should be in such settings.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Details about all studied datasets can be found in Section 4 and Appendix C In supplementary material, we provide code for data preprocessing, as well as all for all methods and experiments described in the paper. Hyperparameters and their tuning procedure are reported in Appendix D
REFERENCES
A. Asuncion and D. Newman. Uei machine learning repository, 2007.
J. Bekker and J. Davis. Estimating the class prior in positive and unlabeled data through decision tree induction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Bekker and J. Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109(4):719-760, 2020.
G. Blanchard, G. Lee, and Scott. Semi-supervised novelty detection. The Journal of Machine Learning Research, 11:2973-3009, 2010.
R. Chalapathy and Chawla. Deep learning for anomaly detection: survey. arXiv preprint 17Xiv:1901.03407, 2019.
R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv: (rXi:::802.06660. 2018.
V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey, ACM computing surveys (CSUR), 1-58),1-58, 2009.
N. Charoenphakdee and M. Sugiyama. Positive-unlabeled classification under class prior shift and asymmetric error. In Proceedings of the 2019 SIAM International Conference on Data Mining, pages 271-279. SIAM, 2019.
H. Chen, F. Liu, Y. Wang, L. Zhao, and H. Wu. variational approach for learning from positive and unlabeled data. arXiv preprint arXiv:9906 1x00:1906.00642, 2019.
Cresci, R. Di Pietro, M. Petrocchi, A. Spognardi, and M. Tesconi. Dna-inspired online behavioral modeling and its application spambot detection. IEEE Intelligent Systems, 31(5):58-66) 04,5,,66,77 2016.
M. David. Tax. One-class classification. PhD thesis, PhD thesis, Technische Universiteit Delft, 2001.
Denis. Pac learning from positive statistical queries. International Conference on Algorithmic Learning Theory, pages 112-126. Springer, 1998.
Denis, R. Gilleron, and F. Letouzey. Learning from positive and unlabeled examples. Theoretical Computer Science, (48(1):70-55 2005.
M. Du Plessis, G. Niu, and M. Sugiyama. Convex formulation for learning from positive and unla- beled data. In International conference machine learning, pages 1386-1394, 2015.
M. C. Du Plessis, G. Niu, and M. Sugiyama. Analysis of learning from positive and unlabeled data. In Advances in neural information processing systems, pages 703-711, 2014.
C. Blkan and K. Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 213-220. 2008
Ergen, A. H. Mirza, and S. S. Kozat. Unsupervised and semi-supervised anomaly detection with istm neural networks. arXiv preprint arXiv: 2017.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
10

---

Under review as a conference paper at ICLR 2022
Z. Ghafoori and C. Leckie. Deep multi-sphere support vector data description. In Proceedings of the 2020 SIAM International Conference on Data Mining, pages 109-117. SIAM, 2020.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
S. Goyal, A. Raghunathan, M. Jain, H. V. Simhadri, and P. Jain. Droce: Deep robust one-class classification. arXiv preprini arXiv:2002. 12718, 2020.
F. E. Grubbs. Procedures for detecting outlying observations in samples. Technometrics, 11(1): 1-21, 1969.
L.-Z. Guo, Z.-Y. Zhang, Y. Jiang, Y.FF. Li, and Z.-H. Zhou. Safe deep semi-supervised learning for unseen-class unlabeled data. In International Conference on Machine Learning, pages 3897- 3906. PMLR, 2020.
Z. Hammoudeh and D. Lowd. Learning from positive and unlabeled data with arbitrary positive shift. arXiv preprint arXiv:2002.10261, 2020.
D. Hendrycks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 046606, 2018.
Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 988):1335-1788 1997.
V. Hodge and J. Austin. survey of outlier detection methodologies. Artificial intelligence review, 22(2))85-126, 126, 2004.
W. Hu, M. Wang, Q- Qin, J. Ma, and B. Liu. Hm: A holistic approach to class learning. Advances in Neural Information Processing Systems, 33:19111-19124, 2020.
W. Hu, R. Le. B. Liu, Ji, J. Ma, D. Zhao, and R. Yan. Predictive adversarial learning from positive and unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7806-7814, 2021.
D. Ivanov. Dedpul: Difference-of wtt.://eee-esstttaaadddddettttso positive-umlabeled learning. arXiv preprint arXiv:1902.08 06965,, 2019.
S. Jain, M. White, M. W. Trosset, and P. Radivojac. Nonparametric semi-supervised learning of class proportions. arXiv preprint arXiv: 009944, 2016.
W. Karush. Minima of functions of several variables with inequalities as side constraints. M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago, 1939.
R. Kiryo, G. Niu, M. C. Du Plessis, and M. Sugiyama. Positive-unlabeled learning with non non-negative risk estimator. In Advances in neural information processing systems, pages 1675-1685, 2017.
A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
H. W. Kuhn and A. W. Tucker. Nonlinear programming. Traces and emergence of nonlinear programming, pages 247-258. Springer, 2014.
X. Li and B. Liu. Learning to classify texts using positive and unlabeled data. In IJCAI, volume 3, pages 587-592, 2003.
X.-L. Li and B. Liu. Learning from positive and unlabeled examples with different data distributions. In European conference on machine learning, pages 218-229. Springer, 2005.
Y.-F. Li and D.-M. Liang. Safe semi-supervised learning: brief introduction. Frontiers Comput. Sci., 13(4)6669-6) 676, 2019.
Y.-F. Li and Z.HH. Zhou. Towards making unlabeled data never hurt. IEEE transactions on pattern analysis and machine intelligence, 175-81155-188 2014.

---

Under review as a conference paper at ICLR 2022
B. Liu, W. S. Lee, P. S. Yu, and X. Li. Partially supervised classification of text documents. In ICML, volume 2. pages 387-394. Citeseer, 2002.
M. M. Moya, M. W. Koch, and L. D. Hostetler. One-class classifier networks for target recognition applications. NASA ST1/Recon Technical Report N, 93:24043, 1993.
S. Nakajima and M. Sugiyama. Positive-unlaheled classification under class-prior shift: A prior- invariant approach based on density ratio estimation, arXiv preprint arXiv:2 2021.
G. Niu, M. C. du Plessis, T. Sakai, Y. Ma, and M. Sugiyama. Theoretical comparisons of positive- unlabeled learning against positive-negative learning. Advances neural information processing systems, 29: 1 199-1207, 2016.
M. Ott, Y. Choi, C. Cardie, and J. T. Hancock. Finding deceptive opinion spam by any stretch of the imagination. arXiv preprint arXiv:1107.ce 4757077745555 2011.
Oza and V. M. Patel. One-class convolutional neural network. IEEE Signal Processing Letters, 26(2):277-28) 2018.
J. Rodríguez-Ruiz J. I. Mata-Sánchez, R. Monroy, O. Loyola-González, and A. Lóppz--uuevas A one-class classification approach for bot detection on twitter. Computers Security, 91:101715, 2020.
L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International conference on machine learning, pages 4393 4402, 2018.
L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02663 2019.
L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. unifying review of deep and shallow anomaly detection. arXiv preprint vrXiv:2009.177:3 2020a.
L. Ruff, R. A. Vandermeulen, B. J. Franks, K.-R. Müller, and M. Kloft. Rethinking assumptions in deep anomaly detection. arXiv preprint arXiv:2006. 200.000300000 2020b.
T. Sakai and N. Shimizu. Covariate shift adaptation learning from positive and unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4838-4845, 2019.
B. Saleh, A. Farhadi, and A. Elgammal. Object-centric anomaly detection by attribute-basee rea- soning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 787-794, 2013.
B. Schölkopf, Williamson, A. Smola, J. Shawe-Taylor, and C. Platt. Support vector method for novelty detection. In Advances in neural information processing systems, pages 582- 588, 2000.
B. Schölkopf, C. Platt, J. Shawe-Taylor, J. Smola, and R. C. Williamssn. Estimating the support of high-dimensional distribution. Neural computation, 13(7): 2001.
C. Scott and G. Blanchard. Novelty detection: Unlabeled data definitely help. In Artificial intelli- gence and statistics, pages 464-471. PMLR, 2009.
C. D. Scott and R. D. Nowak. Learning minimum volume sets. The Journal of Machine Learning Research. 7:665-704, 2006.
J. Tack, S. Mo, J. Jeong, and J. Shin. Csi: Novelty detection via contrastive learning distribution- ally shifted instances. arXiv preprint arXiv:2007.08/ 2020.
D. M. Tax and R. P. Duin. Support vector data description. Machine learning, 54(1):45-66 2004,
R. Vert, J.PP. Vert, and B. Schölkopf. Consistency and convergence rates one-class svms and related algorithms. Journal of Machine Learning Research, 7(5), 2006.
12

---

Under review as a conference paper at ICLR 2022
M. Xu, B. Li. G. Niu, B. Han, and M. Sugiyama. Revisiting sample selection approach to positive- unlabeled learning: Turning unlabeled data into positive rather than negative. arXiv preprint 19X1::1901.10155. 2019.
H. Yu, J. Han, and K. C.-C. Chang. Pebl: Positive example based learning for web page classification using svm. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '02, pages 239-248, New York, NY, USA, 2002. ACM. ISBN 1-58113-567-X, doi: 10. 10.1145775044.775503,
URL http: acm org/10.1145)
775047. 775083
13

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL EXPERIMENTS
A.1 NUMBER OF POSITIVE MODES
In the standard one-vs-all setting the positive distribution is close unimodal. Many OC meth- ods try to envelope positive data and thus greatly benefit from unimodality (Ghafoori and Leckie 2020). However, distributions of real-world data are often more complex. Therefore, the one-vs-all setting might produce overly optimistic estimates of the performance of OC methods. We present synthetic example that highlights the potential struggle of OC methods with multimodal positive distributions in the case when the negative points are concentrated between the modes of the positive distribution (Fig. a). In this example, the OC method attempts to enclose all positive modes in single envelope and assigns high probability of being positive negative examples. In contrast, PU methods are robust to multimodality and can accurately estimate the decision boundary (Fig. (b)).
In this setting, we consider a random subset from vehicle (animal) CIFAR- 10 classes as positive data and all animal (vehicle) classes as negative data. We study the performance of models with respect to the number classes that form a positive sample. The results are reported in Tables 6 7 the one-vs-all setting, PU modifications consistently outperform their OC counterparts. Further, the performance of all OC methods decreases with the increased number of modes. OC-SVM suffers from multimodality the most, which is expected from a method that tries to enclose all positive data in single envelope. The performance of some PU modifications also drops due data becoming more complex, but these drops are slight compared to OC algorithms. The exception is PU-CSI that performs exactly well as the original CSI on this task. Interestingly, the performance of SOTA PU methods generally increases with the modality. As a result, DRPU and PAN outperform all other algorithms when the modality high (as in other experiments, we exclude CNN-based models from this comparison due to leveraging ResNet pretrained on ImageNet). However, when the modality low, PU-DROCC achieves even better performance.
Like in
Table 6: ROC AUC for different modalities of positive distribution on CIFAR-10 with vehicle classes chosen as positive
Pos modality
2
4
OC-SVM
0.74
0.60
0.58
0.57
PU-SVM
0.80
0.80
0.77
0.80
CSI
0.85
0.82
0.83
0.88
PU-CSI
0.85
0.82
0.83
0.88
DROCC
0.78
0.79
0.77
0.73
PU-DROCC
0.92
0.87
0.86
0.83
OC-CNN
0.90
0.90
0.81
0.82
PU-CNN
0.98
0.98
0.96
0.96
DRPU
0.88
0.86
0.88
0.93
PAN
0.87
0.85
0.88
0.93
Table 7: ROC AUC for different modalities of positive distribution on CIFAR-10 with animal classes chosen as positive
Pos modality
1
3
5
SV-
0.60
0.64
0.69
0.66
0.64
0.67
PU-SVM
0.63
0.65
0.72
0.68
0.66
0.69
CSI
0.64
0.60
0.59
0.55
0.53
0.51
PU-CSI
0.66
0.60
0.59
0.53
0.52
0.51
DROCC
0.73
0.75
0.74
0.69
0.72
0.69
PU-DROCC
0.90
0.91
0.90
0.89
0.89
0.90
OC-CNN
0.76
0.81
0.70
0.78
0.74
0.74
PU-CNN
0.97
0.97
0.95
0.97
0.96
0.96
DRPU
0.82
0.78
0.77
0.82
0.86
0.93
PAN
0.82
0.77
0.77
0.81
0.85
0.93
14

---

Under review as a conference paper at ICLR 2022
pos labeled
pos labeled
unlabeled
unlabeled
neg unlabeled
neg unlabeled
optimal classifier
optimal classifier
(a) OC ROC AUC
(b) PU ROC AUC 0.93
Figure 6: Separating lines of OC and PU algorithms on multimodal synthetic data. Positive distribu- tion is mixture Gaussians with four different centers. The OC and PU models are OC-SVM and PU-SVM, respectively. OC-SVM tries to enclose all modes in single envelope, whereas PU-SVM correctly identifies the location negative data.
Table 8: ROC AUC for different modalities of negative distribution under random negative shifts CIFAR- 10 with Bird (2) class chosen positive
Neg modality
3
OC-SVM
0.54
0.59
0.62
0.61
PU-SVM
0.54
0.61
0.64
0.60
CSI
0.59
0.61
0.58
0.60
PU-CSI
0.54
0.59
0.56
0.57
DROCC
0.69
0.68
0.64
0.64
PU-DROCC
0.72
0.73
0.71
0.70
OC-CNN
0.63
0.58
0.58
0.64
PU-CNN
0.60
0.73
0.76
0.81
DRPU
0.72
0.70
0.70
0.72
PAN
0.64
0.65
0.70
0.69
A.2 SHIFT OF THE NEGATIVE DISTRIBUTION
Table g and 9 show results of experiments under random and specific negative distribution shifts with bird class chosen as positive. The results are similar those the main text. These exper- iments show that while some shifts are not too detrimental for PU methods, there exist particular bad shifts that are. We also see that performance of OC methods varies drastically depending on the negative class, meaning that some classes are more difficult to distinguish than others regardless of shifts. Still, on average PU modifications can provide significant improvements. Moreover, our PU- DROCC is robust to shifts it consistently outperforms the original DROCC and also outperforms on average other methdods, including PU SOTAs, in cases when modality of negative distribution is not too high. Finally, note that consider rather drastic variants of negative shifts. In the real world, distribution might shift over time more smoothly. also can be possible for PU methods to adapt shifts by incorporating new reliable unlabeled data into training.
15

---

Under review as a conference paper at ICLR 2022
Table 9: ROC AUC for particular shifts with Bird class (2) chosen as positive. We shifts for unimodal and multimodal negative distribution. We consider following multimodal negative distri- butions: A3 [3, 4, 5), A4 {6, 7). V/2 {0. 1). V/3
{8. 9}
Shift 0-+3 3
1
+ 4
A3
A3 + V/2
+ V/3
A3 +
A
OC-SVM
0.53
0.77
0.66
0.47
0.75
0,47
0.52
0.72
PU-SVM
0.68
0.59
0.72
0.47
0.68
0.57
0.60
0.66
CSI
0.70
0.52
0.66
0.63
0.66
0.59
0.51
0.59
PU-CSI
0.69
0.55
0.71
0.60
0.69
0.60
0.50
0.58
DROCC
0.67
0.70
0.78
0.53
0.73
0.63
0.60
0.72
PU-DROCC
0.63
0.69
0.80
0.58
0.88
0.67
0.59
0.68
OC-CNN
0.61
0.77
0.66
0.55
0.78
0.56
0.57
0.71
PU-CNN
0.58
0.58
0.83
0.64
0.97
0.79
0.58
0.71
DRPU
0.53
0.54
0.89
0.54
0.51
0.62
0.91
0.73
PAN
0.53
0.54
0.83
0.54
0.52
0.62
0.91
0.73
A.3 SIZE AND CONTAMINATION OF UNLABELED DATA
In the main text, we report performance of the few selected OC and PU methods with respect to the proportions a and the size of unlabeled data |.u). Here, we additionally report results for all studied OC models and their PU modifications. The results are presented in Figures 7 and 8 Figure shows that modifications usually outperform original models perform on par, even when a is particularly high. The result, however, is compounded with multimodality of positive distribution, which might have negative effect on some OC models (see Appendix A. Figure shows that for most models few unlabeled examples Moreover, PU modifications of SVM and PU-DROCC always outperform their OC analogues, even when unlabeled sample is extremely small.
250 are enough to significantly improve performance.
A.4 TWITTER BOTS
Results for experiments Twitter dataset shown in fig. 10 For the description of the dataset, Appendix C Since the dataset is tabular, it is difficult to apply CNN-based and CSI-based models, only apply the rest. In general, els. especially in more realistic case when new bots are added to the already existing. In most cases, state-of-the-art PU models achieve the best performance. It is worth noting that the results of KRodriguez-Ruii et al. 2020) for OC-SVM model are different for some classes. We have discov- ered that with some seeds it is possible replicate results of the original paper, and that the authors appear to have used one random seed in their experiments. However, when results are averaged by seed, the quality of OC-SVM model can drop significantly. Another possible explanation the slight differences in the datasets that we and Rodríguez- RuriiguezzRRiiz-e et al. (020)) use.
find that PU modifications outperform original mod-
Table 10: ROC AUC for different negative test distributions Twitter dataset.
Negatives
all
sociall
social2
social3
traditionall
OC-SVM
0.61
0.98
0.57
0.90
0.20
PU-SVM
0.86
0.71
0.89
0.77
0.98
DROCC
0.72
0.93
0.74
0.95
0.28
PU-DROCC
0.81
0.96
0.88
0.96
0.28
DRPU
0.94
0.94
0.96
0.93
0.89
PAN
0.83
0.99
0.87
0.99
0.45
A.S PU MODELS
Some modern PU methods moved from ROC AUC the main metric to accuracy. Therefore, it hard to compare different methods since in each paper they are optimized for their respective metrics. In this experiment, try to perform fair comparison between modern and classic PU
16

---

Under review as a conference paper at ICLR 2022
0.88
08850
0.80
0,8825
0,8000
0.75
ROC AUC
ROC AUG
0,8775
0.70
0,8750
0.66
0.8330
0.00
OC-SVM
C8I
0.56
0.8630
0.50
Q
(a) SVM-hased models.
(h) CSI-based models.
3,800
0.875
3.825
0.850
ROC AUG
ROC AUC
0.825
3.850
0.000
3.85
0.775
3.800
0.750
3.75
CC-CN RU-ONN
DROCC PUCROOO
(c) CNN-based models,
(d) DROCC- basec models.
Figure 7: ROC AUC with respect proportion of positive examples in unlabeled data.
Table 11: ROC AUC of PU models for various a
0.05
0.25
0.5
0.75
0.95
VPU
0.95
0.94
0.92
0.86
0.58
DRPU
0.90
0.93
0.93
0.91
0.81
PAN
0.95
0.94
0.93
0.91
0.69
EN
0.94
0.93
0.93
0.88
0.66
mnPU
0.64
0.74
0.91
0.91
0.78
unPU*
0.91
0.92
0.93
0.85
0.72
DEDPUL
0.94
0.93
0.92
0.88
0.68
methods. We choose vehicle classes from CIFAR-10 as positive and animal classes as negatives. The results are reported in Table I1 All methods are described in Appendix BrrrendxxB..7aaae and Section 3.1 In Table nnPU row represent nnPU with a estimated from DEDPUL, and nnPU* uses real value of It can be that DRPU outperforms other models when <= 0.75, while PAN outperforms other models when a >= 0.5. Since these two models cover the best performance in all cases, we them tfat-oof-the-an for comparison in the main text.
A.6 OC MODELS
We perform additional comparison with HRN, another OC model that was published concurrently with CSI. The results are reported in Table 12 Despite using the code provided by the authors, HRN performed poorly in our experiments and significantly differs from the results reported in the original paper (reported as HRN"). Our hypothesis to why this could have happened is that, according to the code, the authors have reported the highest ROC-AUC achieved during training, while evaluate ROC-AUC after the training is complete for all methods. Still, even if
17

---

Under review as a conference paper at ICLR 2022
0.8
0.880
0.80
0.873
0.75
ROC AUI
RDC AUC
0.870
0.888
DC-SVM
PU-SMM
2000
100GO
Xu
Xy
(a) SVM-based models.
(b) CSI-based models.
0.86
ROC AUC
0.8
RDC AUC
0.00
0.7
0.78
0.6
OC: CMN
EROCC
0.2
53
250
1300
2000
5000 10000
20000
50
250
1000
2000
5000
18000 23000
Xu
Xu
(c) CNN-based models.
(d) DROCC-based models.
Figure ROC AUC for different sizes of available unlaheled data.
use the results reported in the original paper for comparison, CSI and DROCC generally outperform HRN, so we do not include it for comparison in the main text.
Table 12: ROC AUC in one-vs-all setting for OC models
Pos class
0
2
3
4
5
6
7
8
9
OC-SVM
0.78
0.67
0.60
0.67
0.63
0.64
0.74
0.66
0.68
0.75
CSI
0.70
0.87
0.63
0.62
0.60
0.83
0.73
0.80
0.86
0.89
DROCC
0.76
0.74
0.64
0.61
0.71
0.67
0.75
0.71
0.78
0.79
HRN
0.74
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.67
0.55
HRN
0.73
0.69
0.57
0.63
0.71
0.67
0.77
0.65
0.78
0.77
B METHODS
B.I PU-SVM
This subsection organized as follows. First, we construct PU-SVM from classic SVM model and describe our implementation that is based on optimization with SGD. Second, we prove that PU- SVM without non-negativity constraint can also be solved via dual problem, albeit less efficiently.
B.
OUR IMPLEMENTATION
The soft-margin SVM objective be formulated in the following way:
18

---

Under review as a conference paper at ICLR 2022
min X ||wl + F. (y w- D(x) 6)
(6)
wib
where (v, x)) = (0, yz) is hinge loss, À is regularization parameter, and w 2(x) is the decision function. Since the objective (6) is risk with regularization term, estimation techniques to it. However, several changes are required to that end. First, we replace hinge loss with double hinge loss lan (y, ar) max -2yx (y, æ)). As shown in (Du Plessis al. 2015), the unbiased estimate the objective with double hinge-loss a convex function (7)
apply risk
min A ||wl2 2 + Exn -o fa (-, (xx - b)+ 2/ Ho lan (1, (xxpp) b) aFipp Pa (-1, (1.p)
As we show in Section B.I.2. the objective 7 can be solved via dual problem, but the solution is computationally inefficient. As an alternative, we add an additional non-negative constraint to and solve the resulting objective with the stochastic gradient decent (SGD):
min > ||w1 + SeFiyyrs, (d, (.xpp)) b)+
wib + ma Ea La (-1, (x.)) - b) a So Lan (-1, (Zxp)) - b))
(8)
The non-negativity constraint is motivated by (Kiryo et al., 2017). Since we solve (8) with SGD, we need to explicitly construct feature space for x, i.e. (xx) which again is inefficient. Instead, we make our second change and replace the dot-product in the feature space
((x) with the kernel function K(w, z). Note that now w has same dimension as ø, rather than the dimension of the feature space (z). Similar trick is applied in OC-NN (Chalapathy al. 2018) and DeepSVDD
(Ghafoori and Leckie, 2020), where the dot-product is replaced with inference of neural network. Finally, obtain the PU-SVM objective that use in our study:
min À ||wl + or CxEap lan K (w, x)) - b)+
+ max Ezurs (- (-
K K( ( x)) - b) - a. lan k(,,)) - 6))
B.1.2 DUAL PROBLEM
Here we show how the objective ( can be solved via dual problem. To this end, we use the following property of double-hinge loss:
Lan (1, x) - Lan (-1, x) = max( 2x, 0, - x) max (2x, 0, 11++)) = 2x
(10)
After replacing the expectations in 7 with their empirical estimates and applying the property (10) we get the following optimization problem:
1 Σ (=, - 2xb +
2a
min L(w, b) = min w.b
A ||w0112
+ nu iEN
(11)
web
np
iEP
Similarly to classic SVM, for each unlabeled point we introduce an additional slack variable & Lan (x. b), such that:
0 & 2 1 - b+ 2 (a:) b)
(12)
(13)
(14)
19

---

Under review as a conference paper at ICLR 2022
Or, in another form:
50
(15)
5 + w
x4) < 0
(16)
- 2 w (x:))
< 0
(17)
According to Karush Kuhn-Tucker conditions Karush
1939 Kuhn and Tucker 2014), the opti-
mization problem (11) can be solved with the following Lagrangian function:
= A ||011
a
Σ
((x;) + 2ab - 1 -Es
St E6-EAs A&C
w
T2p
Thu
VEP
iEa
FEN
Σ
B,(b B - 1 + Ev)
Ci(ls - (x, b))
(18)
sEu
iEu
As KKT suggests, the optimal vector for the problem above satisfies the following conditions:
A
(19)
(b - (xxx) - 1 + {:) = 0
(20)
C5 ((w (I,) b))
(21)
00 20 00 = 0
(22)
Ow
Ōb
Tacking closer look at each derivative yields:
30
2w
(1:)) + Σ
(B: + (x,)
(23)
Ow
n2
iEP
iEu
00
2a
Σ
B₁ C
(24)
bb
00
1
A;
B C₁
(25)
96,
Define 73 as:
.ipp
T/
(26)
3 Bi
Ci
1 E
Then, w can be rewritten as:
w Σ Tイ;))
(27)
27 is substituted into the Lagrangian, we finally get:
2 タ=ン Σ TiT
(x:)
T. (1,))(((
up VEN.P jEP
DEP.M.EPp.
+2 Σ B, Σ
C.)+ [C.+
Σ
A
B - Ci
1h
iEu
0
Σ B Σ Σ
(B) +2 ΣV (CC
(28)
iEW
iEP.M JEa
jER.ajER
20

---

Under review as a conference paper at ICLR 2022
Subject to:
A, Bi, 2:
(29)
20
Σ B: - _ =0
(30)
A B₁ C = 0
(31)
74u
,2 p B
Ti
2nh
(32)
C2 , ,iEu
We can apply the kernel trick to
28
and get a quadratic optimization problem with linear constraints
29
32
(r,,BCC)A = Σ Σ
TiT (Ti,
a M
(((1,)))))
Thp iEN, JEP
iEp.a JEp.a
B + Σ Σ Σ IEa
2/) (,,,
Σ Σ (C,K(Z),Z3)
(33)
dEP,u JEu
AEP.u jEu
Similarly to the soft-margin SVM, each point with non-zero т; is support vector. Since, all la- beled positive examples have positive т; they are all support vectors. Because of large number of support vectors, this approach computationally demanding and time consuming, so we opt for optimization with SGD described in Appendix B.1.1
B.2 OC-SVM
We use scikit-learn implementation of OC-SVM in our work. We train all SVM models on features extracted with encoder, which trained as part of autoencoder only positive data. We solve OC-SVM via dual problem, while PU-SVM is trained with gradient decent.
B.3 OC-CNN
We implement OC-CNN according to the original paper (Oza and Patel 2018
B.4 DROCC
For DROCC, we slightly modify the authors' implementatior"" Originally, the adversarial search is performed in the input space. Because of that, negative examples are often original positive images with some noise. Instead, we search for negative examples in the feature space obtained from the output of the middle hidden layer of the network. In our experiments, we find that this small change improves ROC AUC.
B.5 CSI
We use the official implementation of CSI
B.6
HRN
HRN (Hu 2020 trains neural classifier on the log-likelihood loss with special regular- ization term w The proposed term is referred as holistic regularization or H-regularization and
essentially is an analogue of lasso or ridge regularization of higher degree. The paper proposes to
https://ithua.commmiccooofffffff
21

---

Under review as a conference paper at ICLR 2022
set 12. Holistic regularization helps the model to remove feature bias and prevent it from collapsing into a constant solution. We adapt the official implementation of HRN
B.7 PU METHODS
B.7.1
VPU
VPU Chen al. 2019) based variational inference that allows estimate a variational upper bound of the -i-dieereeecc between the real posterior distribution and its estimate, parameterized by neural network. A major advantage of VPU that variational upper bound can be computed without knowing prior probability a, which most modern PU models rely on. Our implementation based on the pseudo-algorithm provided in the original paper.
B.7.2 DRPU
We adapt official implementation of DRPU
B.7.3 PAN
We use our implementation based the original paper.
B.7.4 EN
EN Elkan and Noto 2008 is classic PU algorithm. It consists of two steps. At the first step, biased classifier is trained naively distinguish positive data from unlabeled. At the second step, the output of the trained classifier is calibrated in order to make unbiased predictions.
B.7.5 DEDPUL
DEDPUL [Ivanov] 2019) is a two-stage algorithm. At the first stage, it trains biased classifier to distinguish positive data from unlabeled and obtains predictions of this classifier for all examples. This similar to the first stage of EN. At the second stage, estimates the probability density functions of positive and unlabeled data in the space of predictions. Using the Bayes rule, these densities can estimate the posterior probability p(z). whereas the prior probability a is chosen such that it equals the expected posteriors. DEDPUL achieves state-of-the-ard performance in mixture proportion estimation and can improve accuracy of any PU algorithm.
We use the original implementation of DEDPUI without any changes.
B.8 TICE
In all our experiments, use TIcE estimator of a for PU-SVM and PU-CNN models. We borrow TIEE implementation from DEDPUL repository 5
C DATASETS
Twitter bots In Appendix A.4, we additionally compare models on a dataset of Twitter accounts (Cresci et al.,, 2016; Rodríguez-Ruiz Ruiz et al., (020). This dataset provides information about around 7000 accounts, which are divided into genuine users and four different types bots. On this dataset, treat only type of bot known during training. On inference,
have four different options for negative data: the remaining three bot classes and all bot classes together. We acquire the data from the repository [ which has differences in the number of classes and the account features with the dataset described in the original paper.
22

---

Under review as a conference paper at ICLR 2022
Table 13: Datasets' details for different settings in the main text
Setting
Dataset
Pos class
Neg class
Pos lab
Pos unl
Neg unl
all other
One-vs-all
CIFAR-10
any class
2500
2500
2500
0,5
classes
rnd subset
Neg shift
CIFAR-10
{0}
from other
2500
2500
2500
0.5
classes
subset from {0, 8, 9)
all other
Pos modes
CIFAR-10
2500
2500
2500
0.5
classes
Dependency
all other classes
on Xxx
CIFAR-10
{0, 1, 8, 9)
2500
vary
vary
0.5
Dependency
all other
on
CIFAR-10
{0, 8,
classes
2500
2500
2500
vary
Abnormai
Car
576
0.87
PenDigits
2533
2509
110
0.97
Dec. Reviews
170
154
316
0.33
SMS Spam
1910
1937
609
0.76
Table 14: Datasets' details for new settings in Appendix
Setting
Dataset
Pos class
Neg class
Pos lab
Pos unl
Neg unl
see Table
Neg shift
CIFAR-10
{0} or {2)
2500
2500
2500
0.5
Table
subset from (, 4, 5. 6.
all other classes
Pos modes
CIFAR-10
2500
2500
2500
0.5
Twitter
Genuine
Sociall bot
423
424
818
0.33
Abnormal We use preprocessed Abnormall001 dataset, where all images are resized to 200 x 200 resolution. Additionally for SVM-based models, preprocess both CIFAR-10 and Abnormall001 with encoders trained only on positive data. For CNN-based models, we normalize images with standard mean and std to apply ResNet.
Text Datasets In order to preprocess text datasets, characters to lower case, and apply Snowball Stemmer from nltk package. Additionally, we use pretrained Google News embeddings
stop words and punctuation, cast
Train-test split For CIFAR-I1 10 standard train-test split. Other datasets split randomly into train and test in proportion 4:
Dataset details Tables 13 and 14 presents dataset details for each experimental setting. OC meth- ods use only positive labeled data. PU methods also use unlabeled data but do know which is positive and which is negative.
D HYPERPARAMETERS
Hyperprameters for all models can be found in Tables 16 17 18 20
First, we tune hyperpa- rameters on one train-test split via grid search for all models except DROCC. Initial hyperparameters for DROCC we take from (Goyal et al. 2020). After that, we manually fine-tune hyperparameters in the neighborhood of the initial parameters found with grid search. For SVM-based methods we search on the following grid: kernel E {rbf, linear]. v E {0.1, 0.5, 0.9). E {0.01, 0.1, 1), {0.01, 0.1, 1). Ir E {10-3 10 10-4). num epochs {50, 100, 200). For LSTM-based methods we search on the following grid: Istm dim e (4, 16), v E {0.1, 0.5, 0.9), batch sizee {128, 256), 10-1,11-)),mm num epochs E (50, 100, 200). For CNN-based methods search
Ir (10-2, 10-3 10-1).000mmmmm -nmm
23

---

Under review as a conference paper at ICLR 2022
20) For CSI- based methods we search on the following grid: À (0.01, 0.5), lr E 11008.10- 10 temperaturee {0.01, 0.05, 0.1, 0.5), batch sizee {32, 64, 128). y E {0.9, 0.96, 0.99). For DRPU we search on the following grid: a E 0.000,,001,10 0.1), Ir E (10-1, 1013,,, E (0.96, 0.99), num epochs (10, 20, 50). For PAN search the following grid: À (0.001,0.01.0.1) Ir E {10-², 10000-0 " {0.96, 0.99}, num epochs E 20,,20,50 50).
the following grid: y {0.9, 17. Ir E {10-3, 11003,100"1 1000) num epochs
Table 15: Hyperparameters for SVM- models
Hyperparameter
kernel
1/
A
num epochs
Ir
Ir decay
OC-SVM
linear 0.5
2e-3
PU-SVM
linear
0.01
100
5e-3
0.995
Table 16: Hyperparameters for CNN-based models
Hyperparameter epochs
Ir
batch size
OC-CNN
10
le-4
256
PU-CNN
10
le-4
256
Table 17: Hyperparameters for LSTM-based models
Hyperparameter
num epochs
batch size
Istm dim
OC-LSTM
200
le-4
128
0.5
16
PU-LSTM
100
5e-3
256
0.5
16
24

---

Under review as a conference paper at ICLR 2022
Table 18: Hyperparameters for state-of-the-art PU models
Hyperparameter
Q
X
num epochs
Ir
DRPU
0.001
le-3
0.99
PAN
0.001
20
lc-3
0.99
Table 19: DROCC Hyperparameters
Model
DROCC
PU-DROCC
Pos data
vehicles
Abnormal
All
À
0.5
0.5
0.3
0.3
0.3
0.2
0.3
0.3
0.3
0.3
0.5
0.5
0.3
radius
28
30
1.5
1.1
1,5
1.1
1.5
1.5
1.5
1.1
1.1
1.1
1.1
1.1
Jearning rate
5e-3
1c-3
5e-4
ascent step size
le-2
le-2
le-2
le:3
Te-4
1e:3
le-2
le-2
1e-2
1e-2
le-3
lc-3
le:5
ascent num steps
40
60
60
60
20
40
60
50
50
50
60
50
10
Thr
0.99
num epochs
30
30
30
30
30
30
30
30
30
30
30
15
20
batch size
128
256
Table 20: Hyperparameters for CSI-based models
model
Parameter
vehicles
Abnormal
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.5
0.5
0.1
0.1
batch size
64
32
32
32
32
32
32
32
32
32
64
64
temp
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.1
0.5
0.1
CSI
0.99
0.99
0.99
0.99
0.99
b
le-4
le-3
le-3
1c:3
le-4
1c-3
le-3
1e:3
le-3
1c:3
le-4
le-3
0.1
0.1
0.1
0.1
0.1
0.5
0.1
0.1
batch
32
32
32
32
96
64
PU-CSI
batch size unl
16
16
16
16
16
16
16
16
16
32
32
temp
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.1
0.5
0.07
0.01
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
le-3
1c-3
1c-3
1c-3
1c-3
1c-3
5e-3
1c-4
25


Reviews:
[{"review_id": "jtf6bcUsMhY", "reviewer": "Reviewer_kMPP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "positives:\n- The authors study a critical problem of leveraging unlabeled data to improve the performance of one-class classification tasks. The proposed PU modification of each one-class classification approach is easy to complete. Experiments show the improvements of the proposed approaches.\n\nnegatives:\n- Although the experiments show the effectiveness of using PU loss in the classical one-class SVM, there lack theoretical analysis of the proposed approach, such as the consistency or the convergence rate.\n- Section 3 lacks descriptions of each PU modified algorithm. The details of each method are unclear.\n- The novelty seems limited. The proposed methods are straightforward combinations of one-class classification algorithms and PU loss. The authors report the experimental results on different PU modified algorithms, but it is still short of new insights for using unlabeled data in the one-class classification tasks.", "summary_of_the_paper": "This work empirically investigates using the PU loss in Positive-Unlabeled learning in the one-class classification task. The authors propose several PU modifications of the original classification algorithms that can leverage the unlabeled data. Experiments show improvements after the modifications.", "summary_of_the_review": "The authors propose several modified one-class classification algorithms, where part of the original loss is replayed by the PU loss in Positive-Unlabeled learning. Although the experiments show the effectiveness, there lack theoretical analysis of the proposed approaches.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cwTaU7rNhAK", "reviewer": "Reviewer_R2RK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The empirical motivation provided in the paper clearly demonstrate the need for such study. By considering state-of-the-art OC algorithms, the authors perform PU modifications ranging from simply replacing the OC part with an existing PU part (in the case of OC-LSTM) or introduce appropriate  data into the unlabeled data part during model training. Such simple modifications is shown to have good have good empirical support for claiming PU modifications are better in 1-All scenario and random negative distributional shift. However, as the experiment reveals, non-random negative shifts still affect PU. So the primary question remains: even if distributional shifts are detected using Mann-Whitney test, how to know if this is a random or non-random shift? Furthermore, how to determine if unlabeled data is imbalanced and contain mostly positive data? Here too, the empirical results shown in table 3 shows mixed results. Surprisingly, the opposite seems true when sequential data is considered when the dataset with fewer positive examples exists. This is not clearly explained in the paper.\n\nAnother primary result in the paper is on the use of statistical tests to identify unreliability in unlabeled data. Though the authors mention 3 extremely important questions that provide recommendations on the methodology to be used, the test falls short in addressing when only a few unlabeled data is available. The primary question is how much is few? This may depend on variations in the data, model complexity etc. Unfortunately, the paper does not address this primary questions.", "summary_of_the_paper": "The paper describes an empirical study performed on one-class (OC) and positive-unlabeled (PU) learning  settings. Here, the authors compare the OC and PU versions of multiple existing models under an un-reliable unlabeled data scenarios (such as scarcity and distributional shift of unlabeled data)  and provide practical recommendations to use PU algorithms, or modified OC algorithms for PU appropriately depending on data properties so as to leverage the superior classification efficiency of PU methods typically observed over reliable unlabeled data. ", "summary_of_the_review": "As the paper focussed on empirical evaluation of existing mechanism for providing practical recommendations, the paper's brings out the need for understanding unlabeled data distributions before blind application of OC or PU methods. However, there still exists key unanswered questions needed for practitioners to incorporate the recommendations in practical settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8bLM6sqU80A", "reviewer": "Reviewer_9LXT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.\n- The study problem, exploring large unlabeled data with some positive examples, is important\n- The idea of extending OCC models to PU learning settings is interesting\n- The empirical results show the effectiveness of adapted OCC models in some PU settings.\n\nWeaknesses.\n- PU-OCC models have advantages over PU models only in some extreme cases, such as when the unlabeled data contains mostly positive examples, when there are distribution shifts, etc. These are not surprising as OCC models are specifically designed for such settings. Further, for those unusual settings, there are more one-class classification or anomaly detection problems than PU learning. From this perspective, these settings are somehow ill-posed\n- The empirical results are not convincing enough, as most of the empirical justification are based on one single dataset CIFAR-10.\n- The performance of different methods is dependent heavily on the proportion of positive examples in the unlabeled data. The parameter \\alpha is pre-fixed for each specific dataset. I wonder what would be the empirical observations when the pre-fixed parameter \\alpha is changed in different data.\n- The PU-OCC models are developed based on existing studies; I cannot find any major technical novelties", "summary_of_the_paper": "The work presents an empirical study of extending one-class (OC) classification to PU learning in order to leverage unlabeled data more effectively, especially when the unlabeled data contains only a small proportion of positive samples. Five OC classification models are used and evaluated on a few classification benchmarks.", "summary_of_the_review": "The studied problem is important, but there are a number of major issues in the main claims, the empirical justification and technical novelty. So, I do not recommend it for publication at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "16Zf0kzJ4le", "reviewer": "Reviewer_99kf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper conducts a good set of experiments with the goal of identifying cases when unlabeled data might not help\n\nCons:\n1. Results are not surprising, lacks fresh insights, does not present concrete actionable steps\n2. Alternate semi-supervised ways of using the unlabeled data (such as label propagation) have not been discussed.\n\n\nMain Comments\n\n1. Overall, the approach parallels semi-supervised algorithms where often adding unlabeled data helps and sometimes does not. This is well-known in general. The paper focuses on a specific class of algorithms to demonstrate this.\n\n2. The paper tries to construct a few pathological cases (by setting up the data in certain manner) to bring the issues of introducing unlabeled data into focus, but since the results do not change dramatically (PU still wins in most cases), the examples look weak and not very illustrative.\n\n3. The most important contribution would have been Section 5.6. However, this section is rather small. It would be better to relate \\alpha to results discussed in (say) Section 5.2 in the context of specific datasets. I.e., some explanation along the lines of \"Dataset X has \\alpha = 0.10 and hence we see PU performing better/worse than OC and if we set \\alpha=0.05, then we can achieve the opposite of this behavior...\"\n\n4. The paper's conclusion is rather fuzzy in the end (true \\alpha probably is not known)... It would be good to list out concrete steps to identify dataset characteristics (such as unreliability of the unlabeled data), and offer prescriptions.\n\n5. It would also be good to discuss how we can discard data that looks obviously unreliable and retain what is reliable.", "summary_of_the_paper": "The paper suggests ways to extend purely supervised one-class classifier algorithms using techniques borrowed from PU algorithms in order to make use of unlabeled instances. The paper also presents situations where the unlabeled data (in addition to positive-labeled data) might not be helpful.", "summary_of_the_review": "The paper attempts to address an issue of fundamental interest but falls short. The experimental results might still be useful to researchers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides empirical results for one-class classification problems.
The studied problem is important and the reviewers admire the challenge of this paper.
However, the empirical results are not still insightful enough to provide practical recommendations.

Some of the questions raised by the reviewers could potentially be answered by the authors, but we did not receive any feedback unfortunately.

Given that there is essentially no technical novelty in this paper, it cannot be accepted for ICLR.
Decision: Reject