On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data

1 INTRODUCTION . With some properly chosen loss function ( e.g. , Bartlett et al. , 2006 ; Tewari & Bartlett , 2007 ; Reid & Williamson , 2010 ) and regularization ( e.g. , Tikhonov , 1943 ; Srivastava et al. , 2014 ) , empirical risk minimization ( ERM ) is the common practice of supervised classification ( Vapnik , 1998 ) . Actually , ERM is used in not only supervised learning but also weakly-supervised learning . For example , in semi-supervised learning ( Chapelle et al. , 2006 ) , we have very limited labeled ( L ) data and a lot of unlabeled ( U ) data , where L data share the same form with supervised learning . Thus , it is easy to estimate the risk from only L data in order to carry out ERM , and U data are needed exclusively in regularization ( including but not limited to Grandvalet & Bengio , 2004 ; Belkin et al. , 2006 ; Mann & McCallum , 2007 ; Niu et al. , 2013 ; Miyato et al. , 2016 ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Luo et al. , 2018 ; Kamnitsas et al. , 2018 ) . Nevertheless , L data may differ from supervised learning in not only the amount but also the form . For instance , in positive-unlabeled learning ( Elkan & Noto , 2008 ; Ward et al. , 2009 ) , all L data are from the positive class , and due to the lack of L data from the negative class it becomes impossible to estimate the risk from only L data . To this end , a two-step approach to ERM has been considered ( du Plessis et al. , 2014 ; 2015 ; Niu et al. , 2016 ; Kiryo et al. , 2017 ) . Firstly , the risk is rewritten into an equivalent expression , such that it just involves the same distributions from which L and U data are sampled—this step leads to certain risk estimators . Secondly , the risk is estimated from both L and U data , and the resulted empirical training risk is minimized ( e.g . by Robbins & Monro , 1951 ; Kingma & Ba , 2015 ) . In this two-step approach , U data are needed absolutely in ERM itself . This indicates that risk rewrite ( i.e. , the technique of making the risk estimable from observable data via an equivalent expression ) enables ERM in positive-unlabeled learning and is the key of success . One step further from positive-unlabeled learning is learning from only U data without any L data . This is significantly harder than previous learning problems ( cf . Figure 1 ) . However , we would still like to train arbitrary binary classifier , in particular , deep networks ( Goodfellow et al. , 2016 ) . Note that for this purpose clustering is suboptimal for two major reasons . First , successful translation of clusters into meaningful classes completely relies on the critical assumption that one cluster exactly −4 −2 0 2 4 ( a ) P component −4 −2 0 2 4 −4 −2 0 2 4 ( b ) N component −4 −2 0 2 4 −4 −2 0 2 4 ( c ) U set1 −4 −2 0 2 4 −4 −2 0 2 4 ( d ) U set2 −4 −2 0 2 4 −4 −3 −2 −1 0 1 2 3 4−4 −3 −2 −1 0 1 2 3 4 CCN UU-biased UU Oracle In the left panel , ( a ) and ( b ) show positive ( P ) and negative ( N ) components of the Gaussian mixture ; ( c ) and ( d ) show two distributions ( with class priors 0.9 and 0.4 ) where U training data are drawn ( marked as black points ) . The right panel shows the test distribution ( with class prior 0.3 ) and data ( marked as blue for P and red for N ) , as well as four learned classifiers . In the legend , “ CCN ” refers to Natarajan et al . ( 2013 ) , “ UU-biased ” means supervised learning taking larger-/smaller-class-prior U data as P/N data , “ UU ” is the proposed method , and “ Oracle ” means supervised learning from the same amount of L data . See Appendix B for more information . We can see that UU is almost identical to Oracle and much better than the other two methods . Figure 1 : Illustrative example of classification from a Gaussian mixture dataset . corresponds to one class , and hence even perfect clustering might still result in poor classification . Second , clustering must introduce additional geometric or information-theoretic assumptions upon which the learning objectives of clustering are built ( e.g. , Xu et al. , 2004 ; Gomes et al. , 2010 ) . As a consequence , we prefer ERM to clustering and then no more assumption is required . The difficulty is how to estimate the risk from only U data , and our solution is again ERM-enabling risk rewrite in the aforementioned two-step approach . The first step should lead to an unbiased risk estimator that will be used in the second step . Subsequently , we can evaluate the empirical training and/or validation risk by plugging only U training/validation data into the risk estimator . Thus , this two-step ERM needs no L validation data for hyperparameter tuning , which is a huge advantage in training deep models nowadays . Note that given only U data , by no means could we learn the class priors ( Menon et al. , 2015 ) , so that we assume all necessary class priors are also given . This is the unique type of supervision we will leverage throughout this paper , and hence this learning problem still belongs to weakly-supervised learning rather than unsupervised learning . In this paper , we raise a fundamental question in weakly-supervised learning—how many sets of U data with different class priors are necessary for rewriting the risk ? Our answer has two aspects : • Risk rewrite is impossible given a single set of U data ( see Theorem 2 in Sec . 3 ) ; • Risk rewrite becomes possible given two sets of U data ( see Theorem 4 in Sec . 4 ) . This suggests that three class priors1are all you need to train deep models from only U data , while any two2 should not be enough . The impossibility is a proof by contradiction , and the possibility is a proof by construction , following which we explicitly design an unbiased risk estimator . Therefore , with the help of this risk estimator , we propose an ERM-based learning method from two sets of U data . Thanks to the unbiasedness of our risk estimator , we derive an estimation error bound which certainly guarantees the consistency of learning ( Mohri et al. , 2012 ; Shalev-Shwartz & Ben-David , 2014 ) .3 Experiments demonstrate that the proposed method could train multilayer perceptron , AllConvNet ( Springenberg et al. , 2015 ) and ResNet ( He et al. , 2016 ) from two sets of U data ; it could outperform state-of-the-art methods for learning from two sets of U data . See Figure 1 for how the proposed method works on a Gaussian mixture of two components . 1Two class-prior probabilities are of the training distributions and one is of the test distribution . 2One of the training distribution and one of the test distribution , or two of the training distributions . 3Learning is consistent ( more specifically the learned classifier is asymptotically consistent ) , if and only if as the amount of training data approaches infinity , the risk of the learned classifier converges to the risk of the optimal classifier , where the optimality is defined over a given hypothesis class . 2 PROBLEM SETTING AND RELATED WORK . Consider the binary classification problem . Let X and Y be the input and output random variables such that • p ( x , y ) is the underlying joint density , • pp ( x ) = p ( x | Y = +1 ) and pn ( x ) = p ( x | Y = −1 ) are the class-conditional densities , • p ( x ) is the marginal density , and • πp = p ( Y = +1 ) is the class-prior probability . Data generation process Let θ and θ′ be two valid class priors such that θ 6= θ′ ( here it does not matter if either θ or θ′ equals πp or neither of them equals πp ) , and let ptr ( x ) = θpp ( x ) + ( 1− θ ) pn ( x ) , p′tr ( x ) = θ′pp ( x ) + ( 1− θ′ ) pn ( x ) ( 1 ) be the marginal densities from which U training data are drawn . Eq . ( 1 ) implies there are ptr ( x , y ) and p′tr ( x , y ) , whose class-conditional densities are same and equal to those of p ( x , y ) , and whose class priors are different , i.e. , ptr ( x | y ) = p′tr ( x | y ) = p ( x | y ) , ptr ( Y = +1 ) = θ 6= θ′ = p′tr ( Y = +1 ) . If we could sample L data from ptr ( x , y ) or p′tr ( x , y ) , it would reduce to supervised learning under class-prior change ( Quiñonero-Candela et al. , 2009 ) . Nonetheless , the problem of interest belongs to weakly-supervised learning—U training ( and validation ) data are supposed to be drawn according to ( 1 ) . More specifically , we have Xtr = { x1 , . . . , xn } ∼ ptr ( x ) , X ′tr = { x′1 , . . . , x′n′ } ∼ p′tr ( x ) , ( 2 ) where n and n′ are two natural numbers as the sample sizes of Xtr and X ′tr . This is exactly same as du Plessis et al . ( 2013 ) and Menon et al . ( 2015 ) with some different names . In Menon et al . ( 2015 ) , θ and θ′ are called corruption parameters , and if we assume θ > θ′ , ptr ( x ) is called the corrupted P density and p′tr ( x ) is called the corrupted N density . Despite the same data generation process in ( 2 ) , a vital difference between the problem settings is performance measures to be optimized . Performance measures Let g : Rd → R be an arbitrary decision function , i.e. , g may literally be any binary classifier . Let ` : R→ R be the loss function , such that the value ` ( z ) means the loss by predicting g ( x ) when the ground truth is y where z = yg ( x ) is the margin . The risk of g is R ( g ) = E ( X , Y ) ∼p ( x , y ) [ ` ( Y g ( X ) ) ] = πpEp [ ` ( g ( X ) ) ] + ( 1− πp ) En [ ` ( −g ( X ) ) ] , ( 3 ) where Ep [ · ] means EX∼pp [ · ] and En [ · ] means EX∼pn [ · ] respectively . If ` is the zero-one loss that is defined by ` 01 ( z ) = ( 1 − sign ( z ) ) /2 , the risk is also known as the classification error and it is the standard performance measure in classification . A balanced version of Eq . ( 3 ) is B ( g ) = 1 2 Ep [ ` ( g ( X ) ) ] + 1 2 En [ ` ( −g ( X ) ) ] , ( 4 ) and if ` is ` 01 , ( 4 ) is named the balanced error ( Brodersen et al. , 2010 ) . The vital difference is that ( 3 ) is chosen in the current paper whereas ( 4 ) is chosen in du Plessis et al . ( 2013 ) and Menon et al . ( 2015 ) as the performance measure to be optimized . We argue that ( 3 ) is more natural as the performance measure for binary classification than ( 4 ) . By the phrase “ binary classification ” , we mean πp is neither very large nor very small . Otherwise , due to extreme values of πp ( i.e. , either πp ≈ 0 or πp ≈ 1 ) , the problem under consideration should be retrieval or detection rather than binary classification . Hence , it may be misleading to optimize ( 4 ) , unless πp ≈ 12 which implies that Eqs . ( 3 ) and ( 4 ) are essentially equivalent . Related work Learning from only U data is previously regarded as discriminative clustering ( Xu et al. , 2004 ; Valizadegan & Jin , 2006 ; Li et al. , 2009 ; Gomes et al. , 2010 ; Sugiyama et al. , 2014 ; Hu et al. , 2017 ) . Their goals are to maximize the margin or the mutual information between X and Y . Recall that clustering is suboptimal , since it requires the cluster assumption ( Chapelle et al. , 2002 ) and it is rarely satisfied in practice that one cluster exactly corresponds to one class . As mentioned earlier , learning from two sets of U data is already studied in du Plessis et al . ( 2013 ) and Menon et al . ( 2015 ) . Both of them adopt ( 4 ) as the performance measure . In the former paper , g is learned by estimating sign ( ptr ( x ) − p′tr ( x ) ) . In the latter paper , g is learned by taking noisy L data from ptr ( x ) and p′tr ( x ) as clean L data from pp ( x ) and pn ( x ) , and then its threshold is moved to the correct value by post-processing . In summary , instead of ERM , they evidence the possibility of empirical balanced risk minimization , and no impossibility is proven . Our findings are compatible with learning from label proportions ( Quadrianto et al. , 2009 ; Yu et al. , 2013 ) . Quadrianto et al . ( 2009 ) proves that the minimal number of U sets is equal to the number of classes . However , their finding only holds for the linear model , the logistic loss , and their proposed method based on mean operators . On the other hand , Yu et al . ( 2013 ) is not ERM-based ; it is based on discriminative clustering together with expectation regularization ( Mann & McCallum , 2007 ) . At first glance , our data generation process , using the names from Menon et al . ( 2015 ) , looks quite similar to class-conditional noise ( CCN , Angluin & Laird , 1988 ) in learning with noisy labels ( cf . Natarajan et al. , 2013 ) .4 In fact , Menon et al . ( 2015 ) makes use of mutually contaminated distributions ( MCD , Scott et al. , 2013 ) that is more general than CCN . Denote by ỹ and p̃ ( · ) the corrupted label and distributions . Then , CCN and MCD are defined by ( p̃ ( Ỹ = +1 | x ) p̃ ( Ỹ = −1 | x ) ) = TCCN ( p ( Y = +1 | x ) p ( Y = −1 | x ) ) and ( p̃ ( x | Ỹ = +1 ) p̃ ( x | Ỹ = −1 ) ) = TMCD ( pp ( x ) pn ( x ) ) , where both of TCCN and TMCD are 2-by-2 matrices but TCCN is column normalized and TMCD is row normalized . It has been proven in Menon et al . ( 2015 ) that CCN is a strict special case of MCD . To be clear , p̃ ( ỹ ) is fixed in CCN once p̃ ( ỹ | x ) is specified while p̃ ( ỹ ) is free in MCD after p̃ ( x | ỹ ) is specified . Furthermore , p̃ ( x ) = p ( x ) in CCN but p̃ ( x ) 6= p ( x ) in MCD . Due to this covariate shift , CCN methods do not fit MCD problem setting , though MCD methods fit CCN problem setting . To the best of our knowledge , the proposed method is the first MCD method based on ERM .

Summary: The authors introduce the task of learning from unlabeled data clearly and concisely with sufficient reference to background material. They propose a learning approach, called UU, from two unlabeled datasets with known class priors and prove consistency and convergence rates. Their experiments are insightful to the problem, revealing how the two datasets must be sufficiently separated and how UU learning outperforms state-of-the-art approaches. The writing is clear and the idea is an original refinement of earlier work, justified by its exceeding state-of-the-art approaches. However, the paper needs more experimentation.