Predicting Classification Accuracy When Adding New Unobserved Classes

1 INTRODUCTION . Advances in machine learning and representation learning led to automatic systems that can identify an individual class from very large candidate sets . Examples are abundant in visual object recognition ( Russakovsky et al. , 2015 ; Simonyan & Zisserman , 2014 ) , face identification ( Liu et al. , 2017b ) , and brain-machine interfaces ( Naselaris et al. , 2011 ; Seeliger et al. , 2018 ) . In all of these domains , the possible set of classes is much larger than those observed at training or testing . Acquiring and curating data is often the most expensive component in developing new recognition systems . A practitioner would prefer knowing early in the modeling process whether the datacollection apparatus and the classification algorithm are expected to meet the required accuracy levels . In large multi-class problems , the pilot data may contain considerably fewer classes than would be found when the system is deployed ( consider , for example , the case in which researchers develop a face recognition system that is planned to be used on 10,000 people , but can only collect 1,000 in the initial development phase ) . This increase in the number of classes changes the difficulty of the classification problem and therefore the expected accuracy . The magnitude of change varies depending on the classification algorithm and the interactions between the classes : usually classification accuracy will deteriorate as the number of classes increases , but this deterioration varies across classifiers and data-distributions . For pilot experiments to work , theory and algorithms are needed to estimate how accuracy of multi-class classifiers is expected to change when the number of classes grows . In this work , we develop a prediction algorithm that observes the classification results for a small set of classes , and predicts the accuracy on larger class sets . In large multiclass classification tasks , a representation is often learned on a set of k1 classes , whereas the classifier is eventually used on a new larger class set . On the larger set , classification can be performed by applying simple procedures such as measuring the distances in an embedding space between a new example x ∈ X and labeled examples associated with the classes yi ∈ Y . Such classifiers , where the score assigned to a data point x to belong to a class y is independent of the other classes , are defined as marginal classifiers ( Zheng et al. , 2018 ) . Their performance on the larger set describes how robust the learned representation is . Examples of classifiers that are marginal when used on a larger class set include siamese neural networks ( Koch et al. , 2015 ) , oneshot learning ( Fei-Fei et al. , 2006 ) and approaches that directly optimize the embedding ( Schroff et al. , 2015 ) . Our goal in this work is to estimate how well a given marginal classifier will perform on a large unobserved set of k2 classes , based on its performance on a smaller set of k1 classes . Recent works ( Zheng & Benjamini , 2016 ; Zheng et al. , 2018 ) set a probabilistic model for rigorously studying this problem , assuming that the k1 available classes are sampled from the same distribution as the larger set of k2 classes . Following the framework they propose , we assume that the sets of k1 and k2 classes on which the classifier is trained and evaluated are sampled independently from an infinite continuous set Y according to Yi ∼ PY ( y ) , and for each class , r data points are sampled independently from X according to the conditional distribution PX|Y ( x | y ) . In their work , the authors presented two methods for predicting the expected accuracy , one of them originally due to Kay et al . ( 2008 ) . We cover these methods in Section 2 . As a first contribution of this work ( Section 3 ) , we provide a theoretical analysis that connects the accuracy of marginal classifiers to a variant of the receiver operating characteristic ( ROC ) curve , which is achieved by reversing the roles of classes and data points in the common ROC . We show that the reversed ROC ( rROC ) measures how well a classifier ’ s learned representation separates the correct from the incorrect classes of a given data point . We then prove that the accuracy of marginal classifiers is a function of the rROC , allowing the use of well researched ROC estimation methods ( Gonçalves et al. , 2014 ; Bhattacharya & Hughes , 2015 ) to predict the expected accuracy . Furthermore , the reversed area under the curve ( rAUC ) equals the expected accuracy of a binary classifier , where the expectation is taken over all randomly selected pairs of classes . We use our results regarding the rROC to provide our second contribution ( Section 4 ) : CleaneX ( Classification Expected Accuracy Neural EXtrapolation ) , a new neural-network-based method for predicting the expected accuracy of a given classifier on an arbitrarily large set of classes1 . CleaneX differs from previous methods by using both the raw classification scores and the observed classification accuracies for different class-set sizes to calibrate its predictions . In Section 5 we verify the performance of CleaneX on simulations and real data-sets . We find it achieves better overall predictions of the expected accuracy , and very few “ large ” errors , compared to its competitors . We discuss the implications , and how the method can be used by practitioners , in Section 6 . 1.1 PRELIMINARIES AND NOTATION . In this work x are data points , y are classes , and when referred to as random variables they are denoted by X , Y respectively . We denote by y ( x ) the correct class of x , and use y∗ when x is implicitly understood . Similarly , we denote by y′ an incorrect class of x . We assume that for each x and y the classifier h assigns a score Sy ( x ) , such that the predicted class of x is arg maxy Sy ( x ) . On a given dataset of k classes , { y1 , . . . , yk } , the accuracy of the trained classifier h is the probability that it assigns the highest score to the correct class A ( y1 , . . . , yk ) = PX ( Sy∗ ( x ) ≥ maxki=1Syi ( x ) ) ( 1 ) where PX is the distribution of the data points x in the sample of classes . Since r points are sampled from each class , PX assumes a uniform distribution over the classes within the given sample . An important quantity for a data point x is the probability of the correct class y∗ to outscore a randomly chosen incorrect class Y ′ ∼ PY |Y 6=y∗ , that is Cx = PY ′ ( Sy∗ ( x ) ≥ Sy′ ( x ) ) . This is the cumulative distribution function of the incorrect scores , evaluated at the value of the correct score . We denote the expected accuracy over all possible subsets of k classes from Y by Ek [ A ] and its estimator by Êk [ A ] . We refer to the curve of Ek [ A ] at different values of k ≥ 2 as the accuracy curve . Given a sample of K classes , the average accuracy over all subsets of k ≤ K classes from the sample is denoted by ĀKk . 1Code is publicly available at : https : //github.com/YuliSl/CleaneX 2 RELATED WORK . Learning theory provides bounds of sample complexity in multiclass classification that depend on the number of classes ( Shalev-Shwartz & Ben-David , 2014 ) , and the extension to large mutliclass problems is a topic of much interest ( Kuznetsov et al. , 2014 ; Lei et al. , 2015 ; Li et al. , 2018 ) . However , these bounds can not be used to estimate the expected accuracy . Generalization to out-of-label accuracy includes the work of Jain & Learned-Miller ( 2010 ) . The generalization of classifiers from datasets with few classes to larger class sets include those of Oquab et al . ( 2014 ) and Griffin et al . ( 2007 ) , and are closely related to transfer learning ( Pan et al. , 2010 ) and extreme classification ( Liu et al. , 2017a ) . More specific works include that of Abramovich & Pensky ( 2019 ) , which provides lower and upper bounds for the distance between classes that is required in order to achieve a given accuracy . Kay et al . ( 2008 ) , as adapted by Zheng et al . ( 2018 ) , propose to estimate the accuracy of a marginal classifier on a given set of k classes by averaging over x the probability that its correct class outscores a single random incorrect class , raised to the power of k − 1 ( the number of incorrect classes in the sample ) , that is Ek [ A ] = EX [ PY ′ ( Sy∗ ( x ) ≥ Sy′ ( x ) ) k−1 ] = Ex [ Ck−1x ] . ( 2 ) Therefore , the expected accuracy can be predicted by estimating the values of Cx on the available data . To do so , the authors propose using kernel density estimation ( KDE ) choosing the bandwidth with pseudo-likelihood cross-validation ( Cao et al. , 1994 ) . Zheng et al . ( 2018 ) define a discriminability function D ( u ) = PX ( PY ′ ( Sy∗ ( x ) > Sy′ ( x ) ) ≤ u ) , ( 3 ) and show that for marginal classifiers , the expected accuracy at k classes is given by Ek [ A ] = 1− ( k − 1 ) ∫ 1 0 D ( u ) uk−2du . ( 4 ) The authors assume a non-parametric regression model with pre-chosen basis functions bj , so that D ( u ) = ∑ j βjbj . To obtain β̂ the authors minimize the mean squared error ( MSE ) between the resulting estimation Êk [ A ] and the observed accuracies Āk1k . 3 REVERSED ROC . In this section we show that the expected accuracy , Ek [ A ] , can be better understood by studying an ROC-like curve . To do so , we first recall the definition of the common ROC : for two classes in a setting where one class is considered as the positive class and the other as the negative one , the ROC is defined as the graph of the true-positive rate ( TPR ) against the false-positive rate ( FPR ) ( Fawcett , 2006 ) . The common ROC curve represents the separability that a classifier h achieves between data points of the positive class and those of the negative one . At a working point in which the FPR of the classifier is u , we have ROC ( u ) = TPR ( FPR−1 ( u ) ) . In a multiclass setting , we can define ROCy for each class y by considering y as the positive class , and the union of all other classes as the negative one . An adaptation of the ROC for this setting can be defined as the expectation of ROCy over the classes , that is ROC ( u ) = ∫ Y ROCy ( u ) dP ( y ) . In terms of classification scores , we have TPRy ( t ) = PX ( Sy ( x ) > t | y ( x ) = y ) , FPRy ( t ) = PX ( Sy ( x ) > t | y ( x ) 6= y ) and thus FPR−1y ( u ) = supt { PX ( Sy ( x ) > t | y ( x ) 6= y ) ≥ u } . Here , we single out each time one of the classes y and compare the score of the data points that belong to this class with the score of those that do not . However , when the number of classes is large , we could instead single out a data point x and compare the score that it gets for the correct class with the scores for the incorrect ones . This reverse view is formalized in the following definition , where we exchange the roles of data points x and classes y , to obtain the reversed ROC : Definition 1 . Given a data point x , its corresponding reversed true-positive rate is rTPRx ( t ) = { 1 Sy∗ ( x ) > t 0 Sy∗ ( x ) ≤ t ( 5 ) The reversed false-positive rate is rFPRx ( t ) = PY ′ ( Sy′ ( x ) > t ) ( 6 ) and accordingly rFPR−1x ( u ) = sup t { PY ′ ( Sy′ ( x ) > t ) ≥ u } . ( 7 ) Consequently , the reversed ROC is rROCx ( u ) = rTPRx ( rFPR−1y ( u ) ) = { 1 Sy∗ ( x ) > supt { PY ′ ( Sy′ ( x ) > t ) ≥ u } 0 otherwise ( 8 ) and the average reversed ROC is2 rROC ( u ) = ∫ X rROCx ( u ) dP ( x ) . ( 9 ) Since PY ′ ( Sy′ ( x ) > t ) is a decreasing function of t , it can be seen that rROCx ( u ) = 1 iff u > PY ′ ( Sy′ ( x ) > Sy∗ ) = 1−Cx ( see Proposition 1 in Appendix A ) . However , even though rROCx is a step function , the rROC resembles a common ROC curve , as illustrated in Figure 1 .

Summary: The authors discuss how a classifier’s performance over the initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes by mean of the dual of the ROC function, swapping the roles of classes and samples. Grounded on such function, the authors develop a novel ANN approach learning to estimate the accuracy of classifiers on arbitrarily large sets of classes. Effectiveness of the approach is demonstrated on a suite of benchmark datasets, both synthetic and real-world.