Year: 2021

Abstract:
Recent advances in One-Class (OC) classification combine the ability to learn exclusively from positive examples with the expressive power of deep neural networks. A cornerstone of OC methods is to make assumptions regarding negative distribution, e.g., that negative data are scattered uniformly or concentrated in the origin. An alternative approach employed in Positive-Unlabeled (PU) learning is to additionally leverage unlabeled data to approximate negative distribution more precisely. In this paper, our goal is to find the best ways to utilize unlabeled data on top of positive data in different settings. While it is reasonable to expect that PU algorithms outperform OC algorithms due to access to more data, we find that the opposite can be true if unlabeled data is unreliable, i.e. contain negative examples that are either too few or sampled from a different distribution. As an alternative to using existing PU algorithms, we propose to modify OC algorithms to incorporate unlabeled data. We find that such PU modifications can consistently benefit even from unreliable unlabeled data if they satisfy a crucial property: when unlabeled data consists exclusively of positive examples, the PU modification becomes equivalent to the original OC algorithm. Our main practical recommendation is to use state-of-the-art PU algorithms when unlabeled data is reliable and to use PU modifications of state-of-the-art OC algorithms that satisfy the formulated property otherwise. Additionally, we make a progress towards distinguishing the cases of reliable and unreliable unlabeled data using statistical tests.

heading: 1 INTRODUCTION
text: An input of a supervised binary classifier consists of two sets of examples: positive and negative. However, the access to clean samples from both classes can be obstructed in many realistic scenarios. A particularly well-studied restriction is the absence of clean negative examples. One of the approaches to deal with this restriction is One-Class (OC) classification (Moya et al., 1993). Typically, OC algorithms treat available positive examples as normal data and try to separate it from previously unseen data, referred to as anomalies or outliers (Grubbs, 1969; Hodge and Austin, 2004; Chandola et al., 2009; Chalapathy et al., 2018; Chalapathy and Chawla, 2019). As noted in other studies (Scott and Blanchard, 2009; Ruff et al., 2020a;b), OC algorithms necessarily make prior assumptions about negative distribution. For example, some methods assume negative distribution to be uniform (Vert et al., 2006; Scott and Nowak, 2006) or concentrated where positive data are rare (Tax and Duin, 2004; Ruff et al., 2018), model negative distribution as a Gaussian (Oza and Patel, 2018), or separate positive data from the origin (Schölkopf et al., 2000).
Another approach to classification in the absence of negative data is Positive-Unlabeled (PU) learning (Denis, 1998; Denis et al., 2005; Li and Liu, 2005). In addition to a clean positive sample, PU algorithms leverage an unlabeled set of mixed positive and negative examples. In contrast to OC methods, PU methods approximate negative distribution (Elkan and Noto, 2008; Ivanov, 2019), its statistics (Du Plessis et al., 2014; 2015; Kiryo et al., 2017), or samples from it (Yu et al., 2002; Liu et al., 2002; Li and Liu, 2003; Xu et al., 2019), by comparing positive and unlabeled data. This approach can even outperform supervised classification, given a sufficient amount of unlabeled data (Niu et al., 2016; Kiryo et al., 2017).
Because PU methods make fewer assumptions and have access to more data, they might seem favorable to OC methods whenever unlabeled data are at hand. However, our experiments show that
dependence on unlabeled data may hinder PU methods in particularly extreme cases, which we refer to as cases of unreliable unlabeled data. We identify several such cases, including distributional shifts in unlabeled data, scarcity of unlabeled data, and scarcity of latent negative examples in unlabeled data. We present a motivational example of a possible effect of distributional shifts on PU models in Figure 1. In subplots (a, b) a PU model approximates the separating line more accurately than an OC model. Conversely, in subplots (c, d) a shift of negative distribution causes the PU model to misclassify all negative examples, whereas the OC model is unaffected by the shift.
One possible conclusion is to opt for using OC methods when unlabeled data is unreliable, but we aim to find a principled way to construct robust PU methods that can learn even from unreliable unlabeled data. To this end, we propose modifications of several modern OC algorithms that leverage unlabeled data. These modifications either are based on risk estimation techniques where the negative risk is approximated using positive and unlabeled data (Kiryo et al., 2017), or simply replace positive data with unlabeled in algorithm-specific routines (see Sections 3.2.4, 3.2.5). We find that all modifications benefit from reliable unlabeled data, but only modifications from the second group are safe to apply to unreliable unlabeled data (i.e. either improve upon or perform on par with the original OC methods). We pinpoint this result to a crucial property that only the modifications from the second group possess: in the absence of latent negative examples, the PU modification becomes equivalent to the original OC algorithm. Our main practical recommendation is to use state-of-the-art PU algorithms when reliable unlabeled data is available and to use modifications of state-of-the-art OC algorithms that satisfy the property when robustness is a concern.
A question remains how to identify the cases of unreliable unlabeled data. We find that scarcity of latent negatives can be statistically tested by comparing predictions of OC or PU models for positive and unlabeled samples. Similarly, a shift of negative distribution can be tested by comparing predictions for unlabeled data from training and testing distributions, providing the latter is available.
Related Work There is a line of work in the OC literature that investigates ways to augment OC methods with additional data. Several studies find that exposure to a small and possibly biased sample of outliers can improve the performance of OC classifiers (Hendrycks et al., 2018; Ruff et al., 2019; 2020b). Scott and Blanchard (2009) show both theoretically and empirically that unlabeled data help a classic machine learning algorithm to detect novelties. Nevertheless, there is a lack of modern literature that views OC and PU learning as different solutions to the same problem and gives practical recommendations for different scenarios. Our study attempts to fill this gap.
The problem of unreliable unlabeled data is also underexplored in PU literature. One existing direction is concerned with robustness to class prior shift (Charoenphakdee and Sugiyama, 2019). While this is a useful property, we study a more severe case of arbitrary negative distribution shifts.
Furthermore, an increase in proportions of positive class can be seen as a special case of negative distribution shift. We implement the latest PU method robust to class prior shifts as a baseline (Nakajima and Sugiyama, 2021). Other papers are concerned with arbitrary positive shifts (Hammoudeh and Lowd, 2020) and covariate shifts (Sakai and Shimizu, 2019), but unlike us, they assume access to unbiased unlabeled data from the testing distribution. To the best of our knowledge, we address the problem of PU learning with shifted or scarce latent negatives for the first time.
Our study is conceptually similar to the field of safe semi-supervised learning where the focus is on making unlabeled data never hurt while retaining performance (Li and Zhou, 2014; Li and Liang, 2019; Guo et al., 2020). The difference is that we do not assume access to labeled negative data.

heading: 2 PROBLEM SETUP
text: Let x ∈ Rd be a data point and let y ∈ {0, 1} be a binary label. Let s = 1 if the data point is labeled and 0 otherwise. We assume that only positive data can be labeled, i.e. p(s = 1|y = 0) = 0. We view x, y, s as random variables with some joint distribution f(x, y, s). The probability density functions of positive or negative distributions are given by:
fp(x) := f(x | y = 1, s = 0) (1a) fn(x) := f(x | y = 0, s = 0) (1b)
Let fu(x) be the probability density function of unlabeled distribution, i.e. a mixture of positive and negative distributions, and α = p(y = 1|s = 0) be the mixture proportion. We assume that the probability of being labeled p(s = 1 | y = 1) is constant and independent of x, which is known as Selected Completely At Random (SCAR) setting (Elkan and Noto, 2008). In this case:
fu(x) := f(x|s = 0) = αfp(x) + (1− α)fn(x) (2)
In OC methods we assume that only a sample Xp from fp(x) is available. In PU methods we assume the case-control scenario (Bekker and Davis, 2020): two sets of data Xp and Xu are sampled independently from fp(x) and fu(x). Both OC and PU methods output some score function proportional to p(x) := f(y = 1|x, s = 0) that separates positive and negative data. PU methods often additionally estimate α. Since α is generally unidentifiable (Blanchard et al., 2010), we will focus on estimation of its upper bound α∗, as proposed in (Jain et al., 2016).

heading: 3 METHODS
text: In this section, we first describe PU methods used in this study. We begin by describing Risk Estimation approach, which some of our PU modifications of OC algorithms are based on. Then, we describe DRPU and PAN, which we use for comparison as the most recent and state-of-the-art PU algorithms. In Appendix A.5, we elaborate on this choice as state-of-the-art by comparing these algorithms with other existing PU algorithms. Finally, we describe OC methods used in this study and propose their PU modifications. Additional details are reported in Appendix B.

heading: 3.1 POSITIVE-UNLABELED METHODS

heading: 3.1.1 RISK ESTIMATION
text: Let h(x) be an arbitrary decision function that estimates y, l(t, y) be loss function, i.e. the loss incurred for predicting t when the ground truth is y. Define R+p (h) = Ex∼fp(l(h(x), 1)) and R−n (h) = Ex∼fn(l(h(x), 0)) as positive and negative risks. If both positive and negative data are available, the risk of the decision function can be estimated as a weighted sum of positive and negative risks (eq. 3a). In PU case, R−n (h) is unavailable but can be estimated as a difference between risks on positive and unlabeled data (eq. 3b), as shown in (Du Plessis et al., 2014; 2015).
Rpn(h) = αR + p (h)+(1−α)R−n (h) (3a) Rpu(h) = αR+p (h)− αR−p (h) +R−u (h) (3b)
Estimator (3b) can be improved by introducing a non-negativity constraint to reduce overfitting (Kiryo et al., 2017):
Rnn(h) = αR + p (h) + max(0,−αR−p (h) +R−u (h)) (4)
Estimator (4) is called non-negative risk estimator. In practice, the decision function h is parameterized by θ, which can represent weights of a neural network or some other model. The parameters are trained to minimize Rnn(h(x | θ)) for some loss function like double hinge (Du Plessis et al., 2015) or sigmoid (Kiryo et al., 2017). We use the latter. Notice that α is assumed to be identified in this method, so in experiments, we additionally estimate it with TIcE (Bekker and Davis, 2018).
Risk estimation can be applied to modify any OC model to leverage unlabeled data, providing this OC model is based on or can be generalized to the supervised (PN) setting. This can be done by first replacing the OC objective with the PN objective, and then applying risk estimation to the PN objective, i.e. evaluating negative risk using positive and unlabeled samples.

heading: 3.1.2 DRPU
text: DRPU (Nakajima and Sugiyama, 2021) is a state-of-the-art extension of Risk Estimation technique that relaxes the assumption of identified mixture proportions by leveraging density ratio estimation. Specifically, it minimizes the Bregman divergence between the posterior probability p(x) and its parametric estimate. As a result, DRPU is stable to the shifts of class proportions in unlabeled data.

heading: 3.1.3 PAN
text: Predictive Adversarial Network (PAN) (Hu et al., 2021) is another state-of-the-art PU algorithm based on a GANs (Goodfellow et al., 2014). In this algorithm, the generator is replaced with a classifier trained to identify unlabeled examples that are likely to be considered positive by the discriminator. The discriminator then tries to maximize KL-divergence between its and the generator’s predictions while correctly classifying labeled positive examples.

heading: 3.2 ONE-CLASS METHODS

heading: 3.2.1 OC-SVM
text: OC-SVM (Schölkopf et al., 2000) is one of the most well-known OC methods. OC-SVM modifies the classical objective of Support Vector Machines and tries to separate the nominal data from the origin. In the case of RBF kernel (and other translation-invariant kernels (Schölkopf et al., 2001)), OC-SVM tries to enclose positive data while minimizing enclosed volume and is equivalent to SVDD (David, 2001; Tax and Duin, 2004). Similarly to SVM, OC-SVM can be optimized by solving the dual problem and specifying only dot product in the feature space. OC-SVM solves the following optimization problem:
min w,r
1 2 ‖w‖2 − r + 1 νN N∑ i=0 max(0, r − w · Φ(xi))− r (5)
where Φ is a feature map in some Hilbert space, ν is both a regularization parameter and a correction for possible data contamination, and w · Φ(xi) − r is a decision function. Unlike most OC papers, we use linear kernel instead of RBF kernel in our experiments. During preliminary experiments, we discovered that such replacement improves ROC AUC of OC-SVM on benchmark datasets.
PU As Du Plessis et al. (2015) show, the linear models can be trained in PU setting by minimizing risk defined in (3b). Since SVM is a linear model that minimizes empirical risk, one can use it as h(x) and apply risk estimator techniques. In Appendix B.1.2, we show that similarly to SVM and OC-SVM it can be optimized by solving the dual problem, but that all labeled positive examples are support vectors in PU-SVM. As a faster alternative, we optimize non-negative risk (4) with Stochastic Gradient Decent (SGD), the details of which are reported in Appendix B.1.1.

heading: 3.2.2 OC-CNN
text: OC-CNN (Oza and Patel, 2018) uses a pretrained model like ResNet as a frozen feature extractor. In the feature space, a zero-centered Gaussian is used as the pseudo-negative distribution. After that, a supervised classifier is trained to distinguish positive and pseudo-negative examples. As in the original paper, we use ResNet pretrained on ImageNet as a feature extractor. Since pretraining is done in a supervised manner and classes from CIFAR10 overlap with classes from ImageNet, OC-CNN should not be directly compared with other OC algorithms that do not have access to such additional data. The same logic applies to the proposed PU modification.
PU Since OC-CNN assumes a particular negative distribution, it is easy to modify for PU setup. We replace the pseudo-negative examples in the latent space with unlabeled examples processed through the same feature extractor and then perform the standard risk estimation procedure, i.e. minimize non-negative risk in the latent space (4).

heading: 3.2.3 OC-LSTM
text: OC-LSTM (Ergen et al., 2017) combines Long Short-Term Memory (Hochreiter and Schmidhuber, 1997) and OC-SVM. First, the data points are processed through an LSTM block, after which OCSVM is applied to the last hidden state of LSTM. One way to optimize such model is to freeze its components iteratively. Alternatively, OC-SVM can be optimized with SGD and trained end-to-end with LSTM. We employ the latter approach.
PU To adapt OC-LSTM for unlabeled data, we replace OC-SVM with our PU-SVM.

heading: 3.2.4 DROCC
text: DROCC (Goyal et al., 2020) achieves state-of-the-art performance on several real-world datasets across different domains. The key idea behind DROCC is that positive data lie on a low-dimensional locally Euclidean manifold. Based on this assumption, the pseudo-negative examples can be generated from available positives via gradient ascent, similarly to adversarial attacks. After that, a classifier is trained to minimize cross-entropy loss on positive and pseudo-negative examples.
PU In our PU modification of DROCC, we generate pseudo-negative points from unlabeled rather than positive examples. This way, the pseudo-negative examples are closer to the real negatives if unlabeled data contain some negative points, i.e. if α < 1. In the extreme case when unlabeled data contain only positives, our modification is equivalent to the original algorithm.

heading: 3.2.5 CSI
text: CSI (Tack et al., 2020) is an OC method based on contrastive learning. In addition to contrasting a given example with other instances as in conventional contrastive learning, CSI contrasts the example with distributionally-shifted augmentations of itself, such as rotations or permutations of parts of an image. The original study uses ResNet as a neural architecture, but we train a small convolutional network instead for a fair comparison with other algorithms.
PU In our PU modification of CSI, we additionally contrast positive points with examples from unlabeled data. Like our PU-DROCC, such modification will be close to the original model in the extreme case when unlabeled data has no negative examples, i.e. α ∼ 1.

heading: 4 DATASETS
text: Here we briefly describe the datasets used in this study. For details, please refer to Appendix C.
CIFAR-10 CIFAR-10 is a standard for both OC and PU methods benchmark dataset with images from ten animal and vehicle classes (Krizhevsky et al., 2009). There are around 6000 images for each class and the proportion of negatives and positives depends on the particular experiment.
Abnormal1001 This dataset consists of abnormal images from six classes, including Chair, Car, Airplane, Boat, Sofa, and Motorbike (Saleh et al., 2013). Normal images come from the respective
classes from the PASCAL VOC dataset (Everingham et al., 2010). An example of images from this dataset is presented in Figure 2. We perform experiments only with Car class since it has the most examples, with 110 abnormal and 1315 normal images, α = 0.87. This dataset is challenging for PU methods since the negative data are very scattered and only a few negative examples are available.
Pendigits Pendigits dataset consists of consecutive pixels sampled from digits handwritten on a tablet (Asuncion and Newman, 2007). The task is to classify the digits based on these sequences of pixels. We use a processed dataset from ODDS with 6870 examples in total and 156 anomalies, α = 0.97.
Text Datasets We use two different datasets with text data. The first is a standard dataset of deceptive reviews (Ott et al., 2011). It consists of 400 deceptive hotel reviews acquired with crowdsourcing service and 400 truthful reviews from TripAdvisor, α = 0.33. The second is SMS Spam dataset (Asuncion and Newman, 2007). It contains 5572 examples with 747 spam messages considered as anomalies, α = 0.76.

heading: 5 EXPERIMENTS
text: In this section, we describe experimental settings and report results. We repeat each experiment 10 times. Statistical significance is verified via paired Wilcoxon signed-rank test with a 0.05 P-value threshold. We indicate a significant difference in favor of PU or OC method with a line under the higher ROC AUC value. As noted before (Section 3.2.2), the performance of OC-CNN and PUCNN should not be directly compared to other models. The selected hyperparameters and tuning procedure are reported in Appendix D. Additional experiments are reported in Appendix A.

heading: 5.1 ONE-VS-ALL
text: A common experimental setting in OC papers for datasets with multiple classes is one-vs-all classification. In this setting, one class is treated as positive and all other classes constitute negative examples. We fix α in this setting as 0.5, label half of the positive points, and conduct experiments with each class selected as positive. Since OC methods have access to only half of all positive points, their performance cannot be directly compared to the results reported in their original papers. ROC AUC metrics for OC and PU methods for all classes are presented in Table 1. PU modifications consistently significantly outperform their OC counterparts with few exceptions where methods perform on par. Furthermore, DRPU and PAN outperform other algorithms on most classes, although on classes 7 through 9 our PU-CSI performs exceptionally well.

heading: 5.2 SHIFT OF THE NEGATIVE DISTRIBUTION
text: While in the previous setting PU algorithms have shined, that setting is convenient in that unlabeled data contain reliable latent negative examples. In the next setting, we investigate the effect of negative distribution shift, i.e. when latent negative examples are sampled from different distributions
at train and test times. For example, this can be relevant if the negative distribution constantly shifts over time and the model cannot be retrained at each time step, or if a large but biased unlabeled sample is available for training, whereas an unlabeled sample used for inference is either too small or for some reason unavailable. Since negative shift only affects unlabeled data, only PU methods are expected to be sensitive to it (Fig. 1). Further, we suspect that the effect of negative shift on PU methods might decrease with the increased modality of the negative distribution due to negative data covering more latent space, so we additionally vary the number of negative classes.
For this setting, we consider 0 class from CIFAR-10 as positive, n random classes as negative train, and n other random classes as negative test, where n ∈ {1, 2, 3, 4}. The results are presented in Table 2. Despite the distribution shifts that negatively affect PU methods, most PU modifications either significantly outperform or perform on par with their OC counterparts. The exception is OC-SVM, which outperforms all other algorithms when n = 1. Evidently, even incorrect but datadriven estimates of the negative distribution used by PU algorithm are on average more helpful than the generic assumptions made by OC algorithms. Furthermore, for state-of-the-art PU algorithms increasing the number of negative classes partially mitigates the effect of negative shifts.
Although PU models still outperform their OC counterparts when the shift is random, there are particular shifts that can significantly harm the performance of PU algorithms. We present results of experiments with non-random negative shifts, such as animal-animal, animal-vehicle, vehicleanimal, and vehicle-vehicle, in Table 3. The results vary from algorithm to algorithm. OC-SVM and in some cases OC-CNN outperform their PU modifications, highlighting that modifications based on risk estimation are not robust to negative distribution shifts. PU-CSI shows the same performance as regular CSI, but both perform poorly compared to other models. This is likely due to CSI performing poorly on some classes chosen as positive (Table 1), as we fix the positive class as 0 in the current setting. On the other hand, PU-DROCC significantly outperforms its OC counterpart in most cases and at least performs on par in other cases. It also outperforms both state-of-the-art PU algorithms in all cases but one. Indeed, this algorithm shows robustness to shifts in latent negative distribution. Experiments with a different class chosen as positive are reported in Appendix A.

heading: 5.3 SIZE AND CONTAMINATION OF UNLABELED DATA
text: Other cases when PU algorithms may struggle are when negative examples are scarce, i.e. when the positive class proportion is high or the sample size of unlabeled data is small. We study the performance of OC and PU methods with respect to α and the size of the unlabeled sample. We focus on state-of-the-art algorithms as well as some baselines, leaving other experiments to Appendix A. We choose all vehicles as positive classes and all animals as negative classes. The results are presented in Figure 3. Subplot (b) shows that even as few as 50 unlabeled examples are sufficient for PU models to outperform DROCC, but that both versions of CSI outperform other algorithms until the unlabeled sample becomes bigger than 1000 examples. Subplot (a) shows that state-of-the-art PU methods struggle when unlabeled examples are mostly positive. In contrast, PU-DROCC consistently outperforms its OC analog. While both versions of CSI perform similarly, they outperform all
other algorithms when α is high. Note that when α = 0.05, the difference between CSI and PU-CSI is statistically significant in favor of the latter. Also note that CSI generally performs much better than in the previous settings, as its performance varies with the selected positive classes. As a final note, we investigate the effect of modality of positive data in more detail in Appendix A.1.

heading: 5.4 ABNORMAL1001
text: The results of the experiments on Abnormal are presented in Table 4. Although this dataset is challenging for PU methods due to the scarcity of anomalies and similarity of nominal and anomalous data, PU modifications still outperform OC counterparts across all models. Furthermore, PAN and our PU-DROCC perform the best among all methods.
Table 4: ROC AUC on Abnormal1001
Base OC PU
SVM 0.60 0.64 CSI 0.55 0.64 DROCC 0.60 0.84 CNN 0.53 0.76 DRPU – 0.78 PAN – 0.84
Table 5: ROC AUC on sequential data
Model PenDigits Deceptive SMS Spam
OC-LSTM 0.96 0.52 0.69 PU-LSTM 0.97 0.77 0.92

heading: 5.5 SEQUENTIAL DATA
text: Comparison of LSTM-based models on datasets with sequential data can be found in Table 5. PU models achieve better scores than OC algorithms on all three datasets. Furthermore, OC-LSTM performs poorly on the Deceptive Reviews dataset, which has very few labeled positive examples.

heading: 5.6 IDENTIFYING UNRELIABLE UNLABELED DATA
text: As our experiments show, some PU algorithms can perform poorly when available unlabeled data are unreliable. Here, we propose a method to distinguish such cases. Empirically, PU models can struggle when α is too high (α > 0.9), when negative shifts occur, or when only a few unlabeled examples are at hand. We discover that statistical tests can help with the first two problems. Applying Mann-Whitney U-test to the outputs of OC or PU models for unlabeled and positive samples can identify if α is high (fig. 4 a). Similarly, applying Mann-Whitney U-test on outputs of PU models for train and test unlabeled samples can help to detect distribution shifts. Fig. 4 (b) shows p-values for samples with and without negative shifts, which only overlap when α is high.

heading: 6 CONCLUSION
text: In this paper, we propose new PU algorithms based on existing OC algorithms, compare these and state-of-the-art PU algorithms in different settings, and formulate a general guideline for constructing a PU algorithm robust to unreliable unlabeled data, i.e. that it should become an OC algorithm when unlabeled data are close to being purely positive. We find that our algorithms that follow this guideline never underperform their OC counterparts, but that safety can come at a cost as PU SOTAs generally outperform our algorithms when unlabeled data are reliable. Furthermore, we find that our PU modifications based on risk estimation consistently improve upon OC algorithms when unlabeled data are reliable. This approach is applicable to any OC algorithm that has a supervised analog and therefore presents another principled way to leverage unlabeled data that does not require drastic changes of the algorithm, albeit does not guarantee improvement if unlabeled data are unreliable. We hope that our findings will motivate future researchers to investigate augmentations of their OC algorithms that leverage unlabeled data, possibly by following our guideline.
On the other hand, to the best of our knowledge, our paper is the first to explicitly formulate the cases of unreliable unlabeled data and to investigate the potential vulnerability of PU algorithms to such cases. While we make a progress towards identifying such cases via statistical tests, a more desirable alternative is a PU algorithm that achieves state-of-the-art performance regardless of whether unlabeled data are reliable. We hope that future researchers will consider testing the robustness of their PU algorithms in the settings of unreliable unlabeled data, as well as proposing safe alternatives to their PU algorithms that should be in such settings.

heading: REPRODUCIBILITY STATEMENT
text: Details about all studied datasets can be found in Section 4 and Appendix C. In supplementary material, we provide code for data preprocessing, as well as all for all methods and experiments described in the paper. Hyperparameters and their tuning procedure are reported in Appendix D.

heading: A ADDITIONAL EXPERIMENTS

heading: A.1 NUMBER OF POSITIVE MODES
text: In the standard one-vs-all setting the positive distribution is close to unimodal. Many OC methods try to envelope positive data and thus greatly benefit from unimodality (Ghafoori and Leckie, 2020). However, distributions of real-world data are often more complex. Therefore, the one-vs-all setting might produce overly optimistic estimates of the performance of OC methods. We present a synthetic example that highlights the potential struggle of OC methods with multimodal positive distributions in the case when the negative points are concentrated between the modes of the positive distribution (Fig. 6 a). In this example, the OC method attempts to enclose all positive modes in a single envelope and assigns high probability of being positive to negative examples. In contrast, PU methods are robust to multimodality and can accurately estimate the decision boundary (Fig. 6 b).
In this setting, we consider a random subset from vehicle (animal) CIFAR-10 classes as positive data and all animal (vehicle) classes as negative data. We study the performance of models with respect to the number classes that form a positive sample. The results are reported in Tables 6, 7. Like in the one-vs-all setting, PU modifications consistently outperform their OC counterparts. Further, the performance of all OC methods decreases with the increased number of modes. OC-SVM suffers from multimodality the most, which is expected from a method that tries to enclose all positive data in a single envelope. The performance of some PU modifications also drops due to data becoming more complex, but these drops are slight compared to OC algorithms. The exception is PU-CSI that performs exactly as well as the original CSI on this task. Interestingly, the performance of SOTA PU methods generally increases with the modality. As a result, DRPU and PAN outperform all other algorithms when the modality is high (as in other experiments, we exclude CNN-based models from this comparison due to leveraging a ResNet pretrained on ImageNet). However, when the modality is low, PU-DROCC achieves even better performance.

heading: A.2 SHIFT OF THE NEGATIVE DISTRIBUTION
text: Table 8 and 9 show results of experiments under random and specific negative distribution shifts with bird class chosen as positive. The results are similar to those in the main text. These experiments show that while some shifts are not too detrimental for PU methods, there exist particular bad shifts that are. We also see that performance of OC methods varies drastically depending on the negative class, meaning that some classes are more difficult to distinguish than others regardless of shifts. Still, on average PU modifications can provide significant improvements. Moreover, our PUDROCC is robust to shifts as it consistently outperforms the original DROCC and also outperforms on average other methdods, including PU SOTAs, in cases when modality of negative distribution is not too high. Finally, note that we consider rather drastic variants of negative shifts. In the real world, distribution might shift over time more smoothly. It also can be possible for PU methods to adapt to shifts by incorporating new reliable unlabeled data into training.

heading: A.3 SIZE AND CONTAMINATION OF UNLABELED DATA
text: In the main text, we report performance of the few selected OC and PU methods with respect to the proportions α and the size of unlabeled data |Xu|. Here, we additionally report results for all studied OC models and their PU modifications. The results are presented in Figures 7 and 8. Figure 7 shows that PU modifications usually outperform original models or perform on par, even when α is particularly high. The result, however, is compounded with multimodality of positive distribution, which might have a negative effect on some OC models (see Appendix A.1). Figure 8 shows that for most models as few unlabeled examples as 250 are enough to significantly improve performance. Moreover, PU modifications of SVM and PU-DROCC always outperform their OC analogues, even when unlabeled sample is extremely small.

heading: A.4 TWITTER BOTS
text: Results for experiments on Twitter dataset shown in fig. 10. For the description of the dataset, see Appendix C. Since the dataset is tabular, it is difficult to apply CNN-based and CSI-based models, so we only apply the rest. In general, we find that PU modifications outperform original models, especially in a more realistic case when new bots are added to the already existing. In most cases, state-of-the-art PU models achieve the best performance. It is worth noting that the results of (Rodrı́guez-Ruiz et al., 2020) for OC-SVM model are different for some classes. We have discovered that with some seeds it is possible to replicate results of the original paper, and that the authors appear to have used one random seed in their experiments. However, when results are averaged by seed, the quality of OC-SVM model can drop significantly. Another possible explanation is the slight differences in the datasets that we and Rodrı́guez-Ruiz et al. (2020) use.

heading: A.5 PU MODELS
text: Some modern PU methods moved from ROC AUC as the main metric to accuracy. Therefore, it is hard to compare different methods since in each paper they are optimized for their respective metrics. In this experiment, we try to perform a fair comparison between modern and classic PU
methods. We choose vehicle classes from CIFAR-10 as positive and animal classes as negatives. The results are reported in Table 11. All methods are described in Appendix B.7 and Section 3.1. In Table 11, nnPU row represent nnPU with α estimated from DEDPUL, and nnPU∗ uses real value of α. It can be seen that DRPU outperforms other models when α <= 0.75, while PAN outperforms other models when α >= 0.5. Since these two models cover the best performance in all cases, we use them as state-of-the-art for comparison in the main text.

heading: A.6 OC MODELS
text: We perform additional comparison with HRN, another OC model that was published concurrently with CSI. The results are reported in Table 12. Despite using the code provided by the authors, HRN performed poorly in our experiments and significantly differs from the results reported in the original paper (reported as HRN∗). Our hypothesis to why this could have happened is that, according to the code, the authors have reported the highest ROC-AUC achieved during training, while we evaluate ROC-AUC after the training is complete for all our methods. Still, even if we
use the results reported in the original paper for comparison, CSI and DROCC generally outperform HRN, so we do not include it for comparison in the main text.

heading: B METHODS

heading: B.1 PU-SVM
text: This subsection is organized as follows. First, we construct PU-SVM from classic SVM model and describe our implementation that is based on optimization with SGD. Second, we prove that PUSVM without non-negativity constraint can also be solved via dual problem, albeit less efficiently.

heading: B.1.1 OUR IMPLEMENTATION
text: The soft-margin SVM objective can be formulated in the following way:
min ω,b
λ ‖w‖2 + E(x,y)∼fu lh(y, w · Φ(x)− b) (6)
where lh(y, x) = max(0, 1 − yx) is hinge loss, λ is regularization parameter, and ω · Φ(x) − b is the decision function. Since the objective (6) is a risk with regularization term, we can apply risk estimation techniques to it. However, several changes are required to that end. First, we replace hinge loss with double hinge loss ldh(y, x) = max(−2yx, lh(y, x)). As shown in (Du Plessis et al., 2015), the unbiased estimate of the objective (6) with double hinge-loss is a convex function (7).
min ω,b
λ ‖ω‖2 + Exu∼fu ldh(−1, ωΦ(xu)− b)+
+αExp∼fp ldh(1, ωΦ(xp)− b)− αExp∼fp ldh(−1, ωΦ(xp)− b) (7)
As we show in Section B.1.2, the objective (7) can be solved via dual problem, but the solution is computationally inefficient. As an alternative, we add an additional non-negative constraint to (7) and solve the resulting objective with the stochastic gradient decent (SGD):
min ω,b
λ ‖ω‖2 + αExp∼fp ldh(1, ωΦ(xp)− b)+
+ max(0, Exu∼fu ldh(−1, ωΦ(xu)− b)− αExp∼fp ldh(−1, ωΦ(xp)− b)) (8)
The non-negativity constraint is motivated by (Kiryo et al., 2017). Since we solve (8) with SGD, we need to explicitly construct feature space for x, i.e. Φ(x), which again is inefficient. Instead, we make our second change and replace the dot-product in the feature space w · Φ(x) with the kernel function K(w, x). Note that now w has same dimension as x, rather than the dimension of the feature space Φ(x). Similar trick is applied in OC-NN (Chalapathy et al., 2018) and DeepSVDD (Ghafoori and Leckie, 2020), where the dot-product is replaced with inference of neural network. Finally, we obtain the PU-SVM objective that we use in our study:
min ω,b
λ ‖ω‖2 + αExp∼fp ldh(1,K(ω, x)− b)+
+ max(0, Exu∼fu ldh(−1,K(ω, x)− b)− αExp∼fp ldh(−1,K(ω, x)− b)) (9)

heading: B.1.2 DUAL PROBLEM
text: Here we show how the objective (7) can be solved via dual problem. To this end, we use the following property of double-hinge loss:
ldh(1, x)− ldh(−1, x) = max(−2x, 0, 1− x)−max(2x, 0, 1 + x) = −2x (10)
After replacing the expectations in (7) with their empirical estimates and applying the property (10), we get the following optimization problem:
min ω,b L(ω, b) = min ω,b λ ‖ω‖2 + 1 nu ∑ i∈u ldh(−1, ωΦ(xi)− b) + 2αb+ 2α np ∑ i∈p −ωΦ(xi)  (11) Similarly to classic SVM, for each unlabeled point we introduce an additional slack variable ξi = ldh(−1, ωΦ(xi)− b), such that:
ξi ≥ 0 (12) ξi ≥ 1− b+ ωΦ(xi) (13) ξi ≥ 2(ωΦ(xi)− b) (14)
Or, in another form:
−ξi ≤ 0 (15) 1− ξi − b+ ωΦ(xi) ≤ 0 (16) −ξi − 2(b− ωΦ(xi)) ≤ 0 (17)
According to Karush–Kuhn–Tucker conditions (Karush, 1939; Kuhn and Tucker, 2014), the optimization problem (11) can be solved with the following Lagrangian function:
L(ω, b, ξ, A,B,C) = λ ‖ω‖2 − α np w ∑ i∈p Φ(xi) + 2αb+ 1 nu ∑ i∈u ξi − ∑ i∈u Aiξi
− ∑ i∈u Bi(b− ωΦ(xi)− 1 + ξi)− ∑ i∈u Ci(ξi − 2(ωΦ(xi)− b)) (18)
As KKT suggests, the optimal vector for the problem above satisfies the following conditions:
Aiξi = 0 (19) Bi(b− ωΦ(xi)− 1 + ξi) = 0 (20) Ci(ξi − 2(ωΦ(xi)− b)) = 0 (21)
∂L ∂ω = ∂L ∂b = ∂L ∂ξi = 0 (22)
Tacking a closer look at each derivative yields:
∂L ∂ω = 2ω − α
np ∑ i∈p Φ(xi) + ∑ i∈u (Bi + 2Ci)Φ(xi) (23)
∂L ∂b = 2α− ∑ i∈u Bi − 2 ∑ i∈u Ci (24)
∂L ∂ξi = 1 nu −Ai −Bi − Ci (25)
Define τi as:
τi =
{ α
2np , i ∈ p − 12Bi − Ci , i ∈ u (26)
Then, ω can be rewritten as:
ω = ∑ i∈p,u τiΦ(xi) (27)
If (27) is substituted into the Lagrangian, we finally get:
L = λ ∑ i∈p,u ∑ j∈p,u τiτjΦ(xi)Φ(xj)− α np ∑ i∈u,p ∑ j∈p τiΦ(xi)Φ(xj)
+b (2α− ∑ i∈u Bi − 2 ∑ i∈u
Ci)︸ ︷︷ ︸ 0
+ ∑ i∈u ξi ( 1 nu −Ai −Bi − Ci ) ︸ ︷︷ ︸
0 + ∑ i∈u Bi + ∑ i∈p,u ∑ j∈u τiBjΦ(xi)Φ(xj) + 2 ∑ i∈p,u ∑ j∈u τiCjΦ(xi)CjΦ(xj) (28)
Subject to:
Ai, Bi, Ci ≥ 0 (29) 2α− ∑ i∈u Bi − 2 ∑ i∈u Ci = 0 (30)
1
nu −Ai −Bi − Ci = 0 (31)
τi =
{ α
2np , i ∈ p − 12Bi − Ci , i ∈ u (32)
We can apply the kernel trick to (28) and get a quadratic optimization problem with linear constraints (29-32):
L(τ,B,C) = λ ∑ i∈p,u ∑ j∈p,u τiτjK(xi, xj)− α np ∑ i∈u,p ∑ j∈p τiK(xi, xj)
+ ∑ i∈u Bi + ∑ i∈p,u ∑ j∈u τiBjK(xi, xj) + 2 ∑ i∈p,u ∑ j∈u τiCjK(xi, xj) (33)
Similarly to the soft-margin SVM, each point with non-zero τi is a support vector. Since, all labeled positive examples have positive τi, they are all support vectors. Because of large number of support vectors, this approach is too computationally demanding and time consuming, so we opt for optimization with SGD described in Appendix B.1.1.

heading: B.2 OC-SVM
text: We use scikit-learn implementation of OC-SVM in our work. We train all SVM models on features extracted with an encoder, which was trained as a part of an autoencoder only on positive data. We solve OC-SVM via dual problem, while PU-SVM is trained with gradient decent.

heading: B.3 OC-CNN
text: We implement OC-CNN according to the original paper (Oza and Patel, 2018).

heading: B.4 DROCC
text: For DROCC, we slightly modify the authors’ implementation1. Originally, the adversarial search is performed in the input space. Because of that, negative examples are often original positive images with some noise. Instead, we search for negative examples in the feature space obtained from the output of the middle hidden layer of the network. In our experiments, we find that this small change improves ROC AUC.

heading: B.5 CSI
text: We use the official implementation of CSI 2.

heading: B.6 HRN
text: HRN (Hu et al., 2020) trains a neural classifier on the log-likelihood loss with a special regularization term ‖w‖n. The proposed term is referred as holistic regularization or H-regularization and essentially is an analogue of lasso or ridge regularization of a higher degree. The paper proposes to
1https://github.com/microsoft/EdgeML 2https://github.com/alinlab/CSI
set n = 12. Holistic regularization helps the model to remove feature bias and to prevent it from collapsing into a constant solution. We adapt the official implementation of HRN 3.

heading: B.7 PU METHODS
text: B.7.1 VPU
VPU (Chen et al., 2019) is based on variational inference that allows to estimate a variational upper bound of the KL-divergence between the real posterior distribution and its estimate, parameterized by a neural network. A major advantage of VPU is that variational upper bound can be computed without knowing prior probability α, which most modern PU models rely on. Our implementation is based on the pseudo-algorithm provided in the original paper.
B.7.2 DRPU
We adapt official implementation of DRPU 4.
B.7.3 PAN
We use our implementation based on the original paper.
B.7.4 EN
EN (Elkan and Noto, 2008) is a classic PU algorithm. It consists of two steps. At the first step, a biased classifier is trained to naively distinguish positive data from unlabeled. At the second step, the output of the trained classifier is calibrated in order to make unbiased predictions.

heading: B.7.5 DEDPUL
text: DEDPUL (Ivanov, 2019) is a two-stage algorithm. At the first stage, it trains a biased classifier to distinguish positive data from unlabeled and obtains predictions of this classifier for all examples. This is similar to the first stage of EN. At the second stage, it estimates the probability density functions of positive and unlabeled data in the space of predictions. Using the Bayes rule, these densities can estimate the posterior probability p(x), whereas the prior probability α is chosen such that it equals the expected posteriors. DEDPUL achieves state-of-the-art performance in mixture proportion estimation and can improve accuracy of any PU algorithm.
We use the original implementation of DEDPUL5 without any changes.

heading: B.8 TICE
text: In all our experiments, we use TIcE estimator of α for PU-SVM and PU-CNN models. We borrow TIcE implementation from DEDPUL repository5.

heading: C DATASETS
text: Twitter bots In Appendix A.4, we additionally compare models on a dataset of Twitter accounts (Cresci et al., 2016; Rodrı́guez-Ruiz et al., 2020). This dataset provides information about around 7000 accounts, which are divided into genuine users and four different types of bots. On this dataset, we treat only one type of bot as known during training. On inference, we have four different options for negative data: the remaining three bot classes and all bot classes together. We acquire the data from the repository6, which has differences in the number of classes and the account features with the dataset described in the original paper.
3https://github.com/morning-dews/HRN 4https://github.com/csnakajima/pu-learning 5https://github.com/dimonenka/DEDPUL 6https://botometer.osome.iu.edu/bot-repository/datasets.html
Abnormal We use preprocessed Abnormal1001 dataset, where all images are resized to 200×200 resolution. Additionally for SVM-based models, we preprocess both CIFAR-10 and Abnormal1001 with encoders trained only on positive data. For CNN-based models, we normalize images with standard mean and std to apply ResNet.
Text Datasets In order to preprocess text datasets, we remove stop words and punctuation, cast characters to lower case, and apply Snowball Stemmer from nltk package. Additionally, we use pretrained Google News embeddings.
Train-test split For CIFAR-10 we use standard train-test split. Other datasets we split randomly into train and test in proportion 4 : 1.
Dataset details Tables 13 and 14 presents dataset details for each experimental setting. OC methods use only positive labeled data. PU methods also use unlabeled data but do not know which is positive and which is negative.

heading: D HYPERPARAMETERS
text: Hyperprameters for all models can be found in Tables 15, 16, 17, 18, 19, 20. First, we tune hyperparameters on one train-test split via grid search for all models except DROCC. Initial hyperparameters for DROCC we take from (Goyal et al., 2020). After that, we manually fine-tune hyperparameters in the neighborhood of the initial parameters found with grid search. For SVM-based methods we search on the following grid: kernel ∈ {rbf, linear}, ν ∈ {0.1, 0.5, 0.9}, γ ∈ {0.01, 0.1, 1}, λ ∈ {0.01, 0.1, 1}, lr ∈ {10−3, 10−4}, num epochs ∈ {50, 100, 200}. For LSTM-based methods we search on the following grid: lstm dim ∈ {4, 16}, ν ∈ {0.1, 0.5, 0.9}, batch size∈ {128, 256}, lr ∈ {10−2, 10−3, 10−4},num epochs ∈ {50, 100, 200}. For CNN-based methods we search on
the following grid: γ ∈ {0.9, 1}, lr ∈ {10−3, 10−4}, num epochs ∈ {10, 20, 40}. For CSIbased methods we search on the following grid: λ ∈ {0.01, 0.1, 0.5}, lr ∈ {10−5, 10−4, 10−3}, temperature∈ {0.01, 0.05, 0.1, 0.5}, batch size∈ {32, 64, 128}, γ ∈ {0.9, 0.96, 0.99}. For DRPU we search on the following grid: α ∈ {0.001, 0.01, 0.1}, lr ∈ {10−4, 10−3}, γ ∈ {0.96, 0.99}, num epochs ∈ {10, 20, 50}. For PAN we search on the following grid: λ ∈ {0.001, 0.01, 0.1}, lr ∈ {10−4, 10−3}, γ ∈ {0.96, 0.99}, num epochs ∈ {10, 20, 50}.

Review summaries:
This work empirically investigates using the PU loss in Positive-Unlabeled learning in the one-class classification task. The authors propose several PU modifications of the original classification algorithms that can leverage the unlabeled data. Experiments show improvements after the modifications.