Improving State-of-the-Art in One-Class Classification by Leveraging Unlabeled Data

1 INTRODUCTION . An input of a supervised binary classifier consists of two sets of examples : positive and negative . However , the access to clean samples from both classes can be obstructed in many realistic scenarios . A particularly well-studied restriction is the absence of clean negative examples . One of the approaches to deal with this restriction is One-Class ( OC ) classification ( Moya et al. , 1993 ) . Typically , OC algorithms treat available positive examples as normal data and try to separate it from previously unseen data , referred to as anomalies or outliers ( Grubbs , 1969 ; Hodge and Austin , 2004 ; Chandola et al. , 2009 ; Chalapathy et al. , 2018 ; Chalapathy and Chawla , 2019 ) . As noted in other studies ( Scott and Blanchard , 2009 ; Ruff et al. , 2020a ; b ) , OC algorithms necessarily make prior assumptions about negative distribution . For example , some methods assume negative distribution to be uniform ( Vert et al. , 2006 ; Scott and Nowak , 2006 ) or concentrated where positive data are rare ( Tax and Duin , 2004 ; Ruff et al. , 2018 ) , model negative distribution as a Gaussian ( Oza and Patel , 2018 ) , or separate positive data from the origin ( Schölkopf et al. , 2000 ) . Another approach to classification in the absence of negative data is Positive-Unlabeled ( PU ) learning ( Denis , 1998 ; Denis et al. , 2005 ; Li and Liu , 2005 ) . In addition to a clean positive sample , PU algorithms leverage an unlabeled set of mixed positive and negative examples . In contrast to OC methods , PU methods approximate negative distribution ( Elkan and Noto , 2008 ; Ivanov , 2019 ) , its statistics ( Du Plessis et al. , 2014 ; 2015 ; Kiryo et al. , 2017 ) , or samples from it ( Yu et al. , 2002 ; Liu et al. , 2002 ; Li and Liu , 2003 ; Xu et al. , 2019 ) , by comparing positive and unlabeled data . This approach can even outperform supervised classification , given a sufficient amount of unlabeled data ( Niu et al. , 2016 ; Kiryo et al. , 2017 ) . Because PU methods make fewer assumptions and have access to more data , they might seem favorable to OC methods whenever unlabeled data are at hand . However , our experiments show that dependence on unlabeled data may hinder PU methods in particularly extreme cases , which we refer to as cases of unreliable unlabeled data . We identify several such cases , including distributional shifts in unlabeled data , scarcity of unlabeled data , and scarcity of latent negative examples in unlabeled data . We present a motivational example of a possible effect of distributional shifts on PU models in Figure 1 . In subplots ( a , b ) a PU model approximates the separating line more accurately than an OC model . Conversely , in subplots ( c , d ) a shift of negative distribution causes the PU model to misclassify all negative examples , whereas the OC model is unaffected by the shift . One possible conclusion is to opt for using OC methods when unlabeled data is unreliable , but we aim to find a principled way to construct robust PU methods that can learn even from unreliable unlabeled data . To this end , we propose modifications of several modern OC algorithms that leverage unlabeled data . These modifications either are based on risk estimation techniques where the negative risk is approximated using positive and unlabeled data ( Kiryo et al. , 2017 ) , or simply replace positive data with unlabeled in algorithm-specific routines ( see Sections 3.2.4 , 3.2.5 ) . We find that all modifications benefit from reliable unlabeled data , but only modifications from the second group are safe to apply to unreliable unlabeled data ( i.e . either improve upon or perform on par with the original OC methods ) . We pinpoint this result to a crucial property that only the modifications from the second group possess : in the absence of latent negative examples , the PU modification becomes equivalent to the original OC algorithm . Our main practical recommendation is to use state-of-the-art PU algorithms when reliable unlabeled data is available and to use modifications of state-of-the-art OC algorithms that satisfy the property when robustness is a concern . A question remains how to identify the cases of unreliable unlabeled data . We find that scarcity of latent negatives can be statistically tested by comparing predictions of OC or PU models for positive and unlabeled samples . Similarly , a shift of negative distribution can be tested by comparing predictions for unlabeled data from training and testing distributions , providing the latter is available . Related Work There is a line of work in the OC literature that investigates ways to augment OC methods with additional data . Several studies find that exposure to a small and possibly biased sample of outliers can improve the performance of OC classifiers ( Hendrycks et al. , 2018 ; Ruff et al. , 2019 ; 2020b ) . Scott and Blanchard ( 2009 ) show both theoretically and empirically that unlabeled data help a classic machine learning algorithm to detect novelties . Nevertheless , there is a lack of modern literature that views OC and PU learning as different solutions to the same problem and gives practical recommendations for different scenarios . Our study attempts to fill this gap . The problem of unreliable unlabeled data is also underexplored in PU literature . One existing direction is concerned with robustness to class prior shift ( Charoenphakdee and Sugiyama , 2019 ) . While this is a useful property , we study a more severe case of arbitrary negative distribution shifts . Furthermore , an increase in proportions of positive class can be seen as a special case of negative distribution shift . We implement the latest PU method robust to class prior shifts as a baseline ( Nakajima and Sugiyama , 2021 ) . Other papers are concerned with arbitrary positive shifts ( Hammoudeh and Lowd , 2020 ) and covariate shifts ( Sakai and Shimizu , 2019 ) , but unlike us , they assume access to unbiased unlabeled data from the testing distribution . To the best of our knowledge , we address the problem of PU learning with shifted or scarce latent negatives for the first time . Our study is conceptually similar to the field of safe semi-supervised learning where the focus is on making unlabeled data never hurt while retaining performance ( Li and Zhou , 2014 ; Li and Liang , 2019 ; Guo et al. , 2020 ) . The difference is that we do not assume access to labeled negative data . 2 PROBLEM SETUP . Let x ∈ Rd be a data point and let y ∈ { 0 , 1 } be a binary label . Let s = 1 if the data point is labeled and 0 otherwise . We assume that only positive data can be labeled , i.e . p ( s = 1|y = 0 ) = 0 . We view x , y , s as random variables with some joint distribution f ( x , y , s ) . The probability density functions of positive or negative distributions are given by : fp ( x ) : = f ( x | y = 1 , s = 0 ) ( 1a ) fn ( x ) : = f ( x | y = 0 , s = 0 ) ( 1b ) Let fu ( x ) be the probability density function of unlabeled distribution , i.e . a mixture of positive and negative distributions , and α = p ( y = 1|s = 0 ) be the mixture proportion . We assume that the probability of being labeled p ( s = 1 | y = 1 ) is constant and independent of x , which is known as Selected Completely At Random ( SCAR ) setting ( Elkan and Noto , 2008 ) . In this case : fu ( x ) : = f ( x|s = 0 ) = αfp ( x ) + ( 1− α ) fn ( x ) ( 2 ) In OC methods we assume that only a sample Xp from fp ( x ) is available . In PU methods we assume the case-control scenario ( Bekker and Davis , 2020 ) : two sets of data Xp and Xu are sampled independently from fp ( x ) and fu ( x ) . Both OC and PU methods output some score function proportional to p ( x ) : = f ( y = 1|x , s = 0 ) that separates positive and negative data . PU methods often additionally estimate α . Since α is generally unidentifiable ( Blanchard et al. , 2010 ) , we will focus on estimation of its upper bound α∗ , as proposed in ( Jain et al. , 2016 ) . 3 METHODS . In this section , we first describe PU methods used in this study . We begin by describing Risk Estimation approach , which some of our PU modifications of OC algorithms are based on . Then , we describe DRPU and PAN , which we use for comparison as the most recent and state-of-the-art PU algorithms . In Appendix A.5 , we elaborate on this choice as state-of-the-art by comparing these algorithms with other existing PU algorithms . Finally , we describe OC methods used in this study and propose their PU modifications . Additional details are reported in Appendix B . 3.1 POSITIVE-UNLABELED METHODS . 3.1.1 RISK ESTIMATION . Let h ( x ) be an arbitrary decision function that estimates y , l ( t , y ) be loss function , i.e . the loss incurred for predicting t when the ground truth is y . Define R+p ( h ) = Ex∼fp ( l ( h ( x ) , 1 ) ) and R−n ( h ) = Ex∼fn ( l ( h ( x ) , 0 ) ) as positive and negative risks . If both positive and negative data are available , the risk of the decision function can be estimated as a weighted sum of positive and negative risks ( eq . 3a ) . In PU case , R−n ( h ) is unavailable but can be estimated as a difference between risks on positive and unlabeled data ( eq . 3b ) , as shown in ( Du Plessis et al. , 2014 ; 2015 ) . Rpn ( h ) = αR + p ( h ) + ( 1−α ) R−n ( h ) ( 3a ) Rpu ( h ) = αR+p ( h ) − αR−p ( h ) +R−u ( h ) ( 3b ) Estimator ( 3b ) can be improved by introducing a non-negativity constraint to reduce overfitting ( Kiryo et al. , 2017 ) : Rnn ( h ) = αR + p ( h ) + max ( 0 , −αR−p ( h ) +R−u ( h ) ) ( 4 ) Estimator ( 4 ) is called non-negative risk estimator . In practice , the decision function h is parameterized by θ , which can represent weights of a neural network or some other model . The parameters are trained to minimize Rnn ( h ( x | θ ) ) for some loss function like double hinge ( Du Plessis et al. , 2015 ) or sigmoid ( Kiryo et al. , 2017 ) . We use the latter . Notice that α is assumed to be identified in this method , so in experiments , we additionally estimate it with TIcE ( Bekker and Davis , 2018 ) . Risk estimation can be applied to modify any OC model to leverage unlabeled data , providing this OC model is based on or can be generalized to the supervised ( PN ) setting . This can be done by first replacing the OC objective with the PN objective , and then applying risk estimation to the PN objective , i.e . evaluating negative risk using positive and unlabeled samples .

Summary: This work empirically investigates using the PU loss in Positive-Unlabeled learning in the one-class classification task. The authors propose several PU modifications of the original classification algorithms that can leverage the unlabeled data. Experiments show improvements after the modifications.