Predicting Classification Accuracy When Adding New Unobserved Classes
Authors: Yuli Slavutsky
Yuval Benjamini
Venue: ICLR 2021 Poster (2021)
Keywords: multiclass classification
classification
extrapolation
accuracy
ROC

Abstract: Multiclass classifiers are often designed and evaluated only on a sample from the classes on which they will eventually be applied. Hence, their final accuracy remains unknown. In this work we study how a classifier’s performance over the initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes. For this, we define a measure of separation between correct and incorrect classes that is independent of the number of classes: the "reversed ROC" (rROC), which is obtained by replacing the roles of classes and data-points in the common ROC. We show that the classification accuracy is a function of the rROC in multiclass classifiers, for which the learned representation of data from the initial class sample remains unchanged when new classes are added. Using these results we formulate a robust neural-network-based algorithm, "CleaneX", which learns to estimate the accuracy of such classifiers on arbitrarily large sets of classes. Unlike previous methods, our method uses both the observed accuracies of the classifier and densities of classification scores, and therefore achieves remarkably better predictions than current state-of-the-art methods on both simulations and real datasets of object detection, face recognition, and brain decoding.

Published as a conference paper at ICLR 2021
PREDICTING CLASSIFICATION ACCURACY WHEN ADDING NEW UNOBSERVED CLASSES
Yuli Slavutsky, Yuval Benjamini
Department of Statistics and Data Science The Hebrew University of Jerusalem Jerusalem, Israel
{yuli. slavutsky,
hujii ac.
ABSTRACT
Multiclass classifiers are often designed and evaluated only on a sample from the classes on which they will eventually be applied. Hence, their final accuracy remains unknown. In this work initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes. For this, we define measure of separation between correct and incorrect classes that is independent of the number of classes: the reversed ROC (rROC), which obtained by replacing the roles classes and data-points in the common ROC. We show that the classification accuracy is function of the rROC in multiclass classifiers, for which the learned representation of data from the initial class sample remains unchanged when new classes are added. Using these results we formulate a robust neural-network- -haee algorithm, CleaneX, which learns to estimate the accuracy of such classifiers arbitrarily large sets of classes. Unlike previous methods, our method uses both the observed accuracies of the classifier and densities of classification scores, and therefore achieves remarkably better predictions than current state-of-the the-art methods on both simulations and real datasets of object detection, face recognition, and brain decoding.
study how a classifier's performance over the
INTRODUCTION
Advances in machine learning and representation learning led to automatic systems that can identify an individual class from very large candidate sets. Examples are abundant in visual object recogni- tion (Russakovsky et al. 2015]] Simonyan & Zisserman 2014), face identification (Liu et al. 2017b), and brain-machine interfaces (Naselaris et 2011 Seeliger al. (018)). In all of these domains, the possible set of classes is much larger than those observed at training or testing.
Acquiring and curating data is often the most expensive component in developing new recognition systems. A practitioner would prefer knowing early in the modeling process whether the data- collection apparatus and the classification algorithm are expected to meet the required accuracy levels. In large multi-class problems, the pilot data may contain considerably fewer classes than would be found when the system is deployed (consider, for example, the case in which researchers develop a face recognition system that is planned to be used on 10,000 people, but can only collect 1,000 the initial development phase). This increase in the number of classes changes the difficulty of the classification problem and therefore the expected accuracy. The magnitude of change varies depending on the classification algorithm and the interactions between the classes: usually classi- fication accuracy will deteriorate as the number of classes increases, but this deterioration varies across classifiers and data-distributions. For pilot experiments to work, theory and algorithms are needed to estimate how accuracy of multi-class classifiers is expected change when the number of classes grows. In this work, we develop a prediction algorithm that observes the classification results for small set of classes, and predicts the accuracy on larger class sets.
In large multiclass classification tasks, a representation is often learned on a set of k; classes, whereas the classifier is eventually used new larger class set. On the larger set, classification

---

Published as a conference paper at ICLR 2021
can be performed by applying simple procedures such measuring the distances in embedding space between a new example x E X' and labeled examples associated with the classes y/ E V Such classifiers, where the score assigned to a data point to belong to class is independent of the other classes, are defined as marginal classifiers (Zheng et al. 2018). Their performance on the larger set describes how robust the learned representation is. Examples of classifiers that marginal when used on a larger class set include siamese neural networks (Koch et al. 2015). one- shot learning (Fei-Fei et al. 2006) and approaches that directly optimize the embedding (Schroff) et al. 2013). Our goal in this work is to estimate how well a given marginal classifier will perform on a large unobserved set of ky classes, based on its performance on a smaller set of L classes.
Recent works (Zheng & Benjamini 2016 Zheng et al. 2018) set a probabilistic model for rigorously studying this problem, assuming that the 1 available classes are sampled from the same distribution as the larger set of 22 classes. Following the framework they propose, we assume that the sets of ky and hi classes on which the classifier trained and evaluated are sampled independently from an infinite continuous set y according to Y; ~ (y)) and for each class, data points are sampled independently from X' according to the conditional distribution Pxxy (2r y). In their work, the authors presented two methods for predicting the expected accuracy, one of them originally due to Kay et al. (2008). We cover these methods in S
As first contribution of this work (Section 3 we provide a theoretical analysis that connects the accuracy of marginal classifiers to a variant of the receiver operating characteristic (ROC) curve, which is achieved by reversing the roles of classes and data points in the common ROC. We show that the reversed ROC (rROC) measures how well a classifier's learned representation separates the correct from the incorrect classes of given data point. We then prove that the accuracy of marginal classifiers is a function of the rROC, allowing the use of well researched ROC estimation methods (Gonçalves et al., 2014 Bhattacharya
Hughes, 2015) to predict the expected accuracy. Furthermore, the reversed area under the curve (rAUC) equals the expected accuracy of binary classifier, where the expectation taken over all randomly selected pairs of classes.
We use our results regarding the rROC to provide our second contribution (Section CleaneX (Classification Expected Accuracy Neural EXtrapolation), new neural--nttwork-baseee method for predicting the expected accuracy of given classifier differs from previous methods by using both the raw classification scores and the observed classi- fication accuracies for different class-set sizes to calibrate its predictions. In Section 3 we verify the performance of CleaneX on simulations and real data-sets. We find it achieves better overall predictions of the expected accuracy, and very few "large" errors, compared to its competitors. We discuss the implications, and how the method can be used by practitioners, in Section 6
arbitrarily large set of classe CleaneX
1.1 PRELIMINARIES AND NOTATION
In this work x are data points, y are classes, and when referred to random variables they are denoted by X, Y respectively. We denote by y(x) the correct class of x, and use y when r is implicitly understood. Similarly, we denote by y an incorrect class of
We assume that for each and the classifier l assigns a score (z), such that the predicted class of x is arg max (z). On a given dataset of l classes, [yi,..., }, the accuracy of the trained classifier h is the probability that it assigns the highest score to the correct class
A(y . vk) (S (x) Z maxt=
where Px is the distribution of the data points x in the sample of classes. Since r points are sampled from each class. Px assumes a uniform distribution over the classes within the given sample.
An important quantity for a data point æ is the probability of the correct class y" to outscore a randomly chosen incorrect class Y" ~ Prin that is C₂ = Py- (x) 2 (x)). This is the cumulative distribution function of the incorrect scores, evaluated at the value of the correct score.
We denote the expected accuracy over all possible subsets of k classes from y by [+[A] and its estimator by Éx[4], We refer to the curve of [x[A) at different values of l > 2 as the accuracy curve. Given a sample of K classes, the average accuracy over all subsets of k < K classes from the sample is denoted by AAK.
"ooee publicly available at: pettps:/ggithee com/ Yul eaneee
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Learning theory provides bounds of sample complexity in multiclass classification that depend on the number of classes (Shalev-Shwand & Ben-David 2014), and the extension to large mutliclass problems is a topic of much interest (Kuznetsov et al. 2014; Lei et al. 2015 Li et al. 2018). How- ever, these bounds cannot be used estimate the expected accuracy. Generalization lization to out-of-label accuracy includes the work of Jain & Learned- Miller (2010). The generalization of classifiers from datasets with few classes to larger class sets include those of Oquab et al. (2014) and Griffin et al. 22007), and are closely related to transfer learning (Pan et al. 2010) and extreme classification (Liu al., 2017a). More specific works include that of Abramovich & Pensky (2019), which provides lower and upper bounds for the distance between classes that is required in order achieve given
accuracy.
Kay et al. 2008 as adapted by Zheng al. (2018), propose to estimate the accuracy of marginal classifier on a given set of classes by averaging over æ the probability that its correct class outscores single random incorrect class, raised to the power of & (the number of incorrect classes in the sample), that is
E& [|] = (S, 2 =
(2)
Therefore, the expected accuracy can be predicted by estimating the values of Cz on the available data. To do so, the authors propose using kernel density estimation (KDE) choosing the bandwidth with pseudo-likellhood cross- validaaaio (Cao al., 1994).
Zheng et al. 2018) define discriminability function
D(u)= = Px (Py, (Sy-(a)) > (z)( S su),
and show that for marginal classifiers, the expected accuracy at l classes is given by
[A] = - (k- 1) D(u)ut=2da du.
(4)
The authors assume a poraametet regression model with pre-chosen basis functions by. so that D(u) Σ, Bybj- To obtain 3 the authors minimize the mean squared error (MSE) between the resulting estimation Bi [A] and the observed accuracies A
3 REVERSED ROC
In this section we show that the expected accuracy, E& [A], can be better understood by studying an ROC-like curve. To do so, we first recall the definition of the common ROC: for two classes setting where one class is considered as the positive class and the other as the negative one, the ROC defined as the graph of the true-positive rate (TPR) against the false-positive rate (FPR) (Fawcetl 2006). The common ROC curve represents the separability that classifier h achieves between data points of the positive class and those of the negative one. At a working point in which the FPR of the classifier is u, we have ROC (u) TPR (FPR-1 (w)).
In a multiclass setting, we can define ROC, for each class y by considering y as the positive class, and the union of all other classes the negative one. An adaptation of the ROC for this setting can be defined as the expectation of ROC,, over the classes, that ROC(u) = Jy ROC, (u) dP In terms of classification scores, we have (1) = (S > y(x) = y), FP),, = (S, > t y(x)+ f y) and thus FPR, (u) = sup; (S, > t [1(z()) + v) w).
Here, we single out each time one of the classes y and compare the score of the data points that belong to this class with the score of those that do not. However, when the number of classes is large, we could instead single out data point x and compare the score that it gets for the correct class with the scores for the incorrect ones. This reverse view is formalized in the following definition, where we exchange the roles of data points x and classes y, to obtain the reversed ROC:
Definition 1. Given data point x, its corresponding reversed true-positive rate is
[1
Sy (x)
rTPR, (t)
(5)
0
(y-(x) St
3

---

Published as a conference paper at ICLR 2021
The reversed false-positive rate is and uccordingly
rFPR. (t) Py (x) > >))
(6)
(FPR_'(u) sup (Py: (Sy > 1) 2 u).
(7)
Consequently, the reversed ROC is
Sy (x) supt (rr (Sy(
( rTPR, (rFPR, (u)) and the average reversed ROC
otherwise
rROC (u) rROC, (u) dP(x).
(9)
x
Since Py: (z) 1) is a decreasing function of t, it can be seen that rROC, (u) iff py (z > Sy,) = C2 (see Proposition in Appendix [A). However, even though rROC,, is a step function, the rROC resembles a common ROC curve, as illustrated Figure
Scores distribution
<TPR, rFPR
FROC,
FROC,
21
22
+ROC, dP(s)
comeet
25
S
Figure 1: The reversed ROC. The leftmost column shows example of the score distributions of four data points: the distribution of scores of incorrect classes (red), and the score of the correct class (green). The yellow shaded area the CDF of the incorrect scores distribution evaluated the correct score, that is Cr- The second column shows the corresponding rTPR (green, top) and rFPR (red, bottom). The third column depicts the resulting rROC, curves. The rightmost plot presents the average rROC over the four data points (solid grey); as the number of averaged data points grows, the rROC curve becomes smoother (dotted blue).
3.1 THE REVERSED ROC AND CLASSIFICATION ACCURACY
In what follows, we show that the classification accuracy can expressed using the average reversed ROC curve. We assume a marginal classifier which assigns scores without ties, that is for all x and all y: * Vj we have Sy, (x) o Sy, (ar) almost surely. such cases the following theorem holds:
Theorem 1. The expected classification accuracy at k classes
= (k -1 /. (1 (R-FROC - u)) k-2 2d.
(10)
Note that d(P(z) in Equation equation 9 assumes uniform distribution with respect to given sample of classes (y1.-.., Vkk and their corresponding data points.
4

---

Published as a conference paper at ICLR 2021
To prove this theorem show that
1 - rROC(1 u)= - (Pr, (5 > Sy (x)) S D(u)
(11)
and the rest follows immediately from the results of Zheng al. 2018 (see Equation the detailed proof in Appendix A
We provide
Now, using the properties of the rROC get
Ex[A] = 1 (A: 1) S'" (1 (1-rRoe -ROC -
u]) (011-])
2
= -fo (1 :ROC,(11 - w)dP(x)) (1- 1-] 2lun-200 2d Ct. -
11 1)u 1'u du
E.
(12)
Therefore, in order to predict the expected accuracy it suffices to estimate the values of C., A consequence of the result above is that the expressions that Kay et al. (2008) and ZZeng et al. (2018) estimate (Equations and respectively) are in fact the same. Nevertheless, their estimation methods differ significantly.
Finally, we note that the theoretical connection between the reversed ROC and the evaluation of classification models extends beyond this particular work. For example, by plugging = into Theorem Lit immediately follows that the area under the reversed cuCee curve (rAUC) the expected accuracy of two classes: rAUC == FROC (u) du = E2[A].
EXPECTED ACCURACY PREDICTION
In this section we present a new algorithm, CleaneX, for the prediction of the expected accuracy of a classifier. The algorithm is based on neural network that estimates the values of C₂ using the classifier's scores on data from the available h classes. These estimations are then used, based on the results of the previous section, to predict the classification accuracy at log > k classes.
The general task of estimating densities using neural networks has been widely addressed (Magdon Ismail & Atiya 1999; Papamakarios et al. 2017, Dinh et al.) 2016 Uria et al. 2014). However, our case, we need to estimate the cumulative distribution only at the value of the correct score Sy- (z). This is an easier task to perform and it allows us to design an estimation technique that learns to estimate the CDF in a supervised manner. We use neural network f(::0) R + I whose input is a vector of the correct score followed by the kn - incorrect scores, and its output is Cz. Once C₂ values are estimated for each x, the expected accuracy for each 2 I S k1 can be estimated according to Equation 12 E, [A] = 1
N E2 C%-1, where = rk1 In each iteration, the network's weights are optimized by minimizing the average over 2
k ki E, Ō Ck-1 A After the weights have converged, the estimated expected accuracy li kn classes can be calculated by ER, [A] +/2 Σ. Č Note that regardless of the target k2, the network's input size is kj. The proposed method is described in detail in Algorithm 1, below.
square distances from the observed accuracies Att: -,- Ekz
Unlike KDE and con-parametric regression, our method does not require choice of basis functions or kernels. It does require choosing the network architecture, but as will be seen in the next sections, in all of our experiments we use the same architecture and hyperparameters, indicating that our algorithm requires very little tuning when applied to new classification problems.
Although the neural network allows more flexibility compared the non- parammrmttriic regression, the key difference between the method we propose and previous ones is that CleaneX combines two sources information: the classification scores (which are used as the network's inputs) and the average accuracies of the available classes (which are used in the minimization objectíve). Unlike the KDE and regression based methods which use only one type of information, we estimate the CDF of the incorrect scores evaluated at the correct score, Cz, but calibrate the estimations to produce accuracy curves that fit the observed ones. We do so by exploiting the mechanism of neural networks to iteratively optimize an objective function over the whole dataset.
5

---

Published as a conference paper at ICLR 2021
Algorithm 1: CleaneX
Input : The classifier's 's score function S, a training set of N examples r from the set of lq available classes, target number of classes 22; feedforward neural network / (:; 0), initial network weights Og. number of training iterations learning rate 1).
Output: Estimated accuracy at k2 classes.
for k = 2 .1 do
Compute A
end
for each x in training set do
Set (x))) Sy (z)...., Syn, (x)
Sort S" S'(x)
Set S₂ +
end
for j= L....... do for each x do
| Set C₂ t I(Sz;
end
Update network parameters performing a gradient descent step:
ki
( C+ 02
Ck-
end Return [A] N En Ōkk
5 SIMULATIONS AND EXPERIMENTS
Here we compare CleaneX the KDE method (Kay al.) (008) and to the non- con-maraettric regres- sion (Zheng et al.) 2018) on simulated datasets that allow to examine under which conditions each method performs better, and on real-world datasets. Our goal is to predict the expected accuracy [A] E [0, 1] for different values of k. Therefore, for each method expected accuracies E& [A] for 2 < k S li and measure the success of the prediction using the root of the mean squared error (RMSE): ( (Bx [A] - A )²)
compute the estimated
1/2 )
For our method, we use in all the experiments an identical feed-forwar neural network with two hidden layers of sizes 512 and 128, rectified linear activation between the layers, and sigmoid applied on the output. We train the network according to Algorithm I for J = 10, 000 iterations with learning rate of 7 = 10-4 using Adam optimizer (Kingma & Ba, 2014). For the regression based method we choose radial basis and for the KDE based method normal kernel, as recommended by the authors. The technical implementation details are provided in Appendix C
A naive calculation of AK requires averaging the obtained accuracy over all possible subsets of k classes from K classes. However, Zheng et al. (2018) showed that for marginal classifiers AK
(2) PK Σn (K=)), where the sum is taken over all data points from the K classes, and R,, is the
number of incorrect classes that the correct class of x outperforms, that is R. Ey 1{5 (xr)
(x)). We use this result compute Ale values.
5.1 SIMULATIONN STUDIES
Here we provide comparison of the methods under different parametric settings. We simulate both classes and data points as d-dimensional vectors. with d 5 (and d 10 shown in Appendix B) Settings vary in the distribution of classes and data-points X|Y, and in the spread of data- data-points around the class centroids. We sample the classes yı, y,,..., ykg from multivariate normal distribution

---

Published as a conference paper at ICLR 2021
(00, I) or multivariate uniform (istriibution(((() v3 We then sample 10 data points for each class, either from a multivariaae normal distribution (y. (y. I²1) or from a multivariate uniform distribution Ul - v30² y + v3o²). The difficullyy level is determined by x² 0.1, 0.2. The classification scores Sy( are set to be the euclidean distances between z and y (in this case the correct scores are expected to be lower than the incorrect ones, requiring some straightforward modifications to our analysis). For each classification problem, we subsample 50 times k classes, for Kj 500, and predict the accuracy at 2 < k S h2 2000 classes.
N, XIV N.N of 01
VoN, X|V -M
V-d,, X. N.N
Y M. X
of 0.2
of 0.1
0.2
6.30
0.35
6.30
0.33
0:00-
0.0-
0:4-
0.01
0.0-
Cleanee
Regression
KDE
Figure 2: Simulation results. For each scenario we show boxplot representing the RMSE values obtained over 50 repetitions using CleaneX (left box, orange), regression based method (middle box, blue) and KDE (right box, purple). The boxes extend from the lower to the upper quartile values, with line at the median; whiskers show values at a distance at most 1.5 IQR (interquartile range) from the lower and the upper quartiles; outliers omitted from the figure for clarity. For h 500 the KDE based method always achieves RMSE values higher than 0.05 and therefore not shown.
The results of the simulations, summarized in Figure 2 show the distribution of RMSE values for each method and setting over 50 repetitions. The corresponding accuracy curves are shown in Figure 4 in Appendix B It can be seen that extrapolation to 20 times the original number of classes can be achieved with reasonable errors (the median RMSE is less than 5% in almost all scenarios). Our method performs better or similar to the competing methods, often with substantial gain. For example, for d = 5 the results show that our method outperforms the KDE hased method in all cases; it outperforms the regression based method in out of 8 settings for K 100, and for L1 500 the regression based method and CleaneX achieve very similar results. As can be seen in Figures 2 and 4 the accuracy curves produced by the KDE method achieves the worst performance. The results of the regression method, on the other hand, are more variable than method, especially at k 100. Additional results for d 10 (see Appendix are consistent with these results, though all methods predict better for = 10.
highly biased and therefore it consistently
5.2 EXPERIMENTS
Here we present the results of three experiments performed on datasets from the fields of computer vision and computational neuroscience, We repeat each experiment 50 times. In each repetition we sub-sample k; classes and predict the accuracy at
l classes.
Object Detection (CIFAR-100) We use the CIFAR dataset (Krizhevsky et al., 2009) that consists of 32 x 32 color images from 100 classes, each class containing 600 images. Each image is embedded into 512 312-dimensional space by VGG-16 network (Simonyan & Zisserman 2014), which was pre-trained on the ImageNet dataset (Deng et al. 2009). On the training set, the centroid of each class is calculated and the classification scores for each image in the test set are set to be the euclidean distances of the image embedding from the centroids. The classification accuracy extrapolated from h 10 to lgg 100 classes.
Experiment 1
Experiment Face Recognition (LFW) We use the "Labeled Faces the Wild" dataset (Huang al. 2007) and follow the procedure described in Zheng et al. (2018): we restrict the dataset to the 1672 individuals for whom it contains at least face photos and include our data exactly randomly chosen photos for each person. We use one of them label y, and the other data
TThe choice of / results in class covariances that equal those in the multivariate normal case.

---

Published as a conference paper at ICLR 2021
point consistent with scenario of single-shot learning. Each photo is embedded into 128- dimensional space using OpenFace embedding (Amos et al., 2016). The classification scores are set to be the euclidean distance between the embedding of each photo and the photos that are used as labels. Classification accuracy is extrapolated from L = 200 to 22 = 1672 classes.
Experiment 3 - Brain Decoding (fMRI) We analyze "mind-reading" task described by Kay al. (2008) in which a vector of v = 1250 neural responses is recorded while a subject watches n = 1750 natural images inside a functional MRI. The goal to identify the correct image from the neural response vector. Similarly to the decoder in (Naselaris et al., 201T), we use 7t 750 images and their response vectors to fit an embedding 9(-) of images into the brain response space, and to estimate the residual covariance Σ. The remaining n R2 = K2 = 1000 examples are used as an evaluation set for g g(-). For image y and brain vector x, the score is the negative Mahalanobis distance [l x1 For each response vector, the image with the highest score is selected. Classification accuracy is extrapolated from h 200 to l 1000 classes.
The experimental results, summarized in Figure DD, show that generally CleaneX outperforms the regression and KDE methods: Figure 3 shows that the median ratio between RMSE values of the competing methods and CleaneX ís higher than 1 in all cases except for KDE on the LFW dataset. As can be seen in Figure A and B. the estimated accuracy curves produced by CleaneX have lower variance than those of the regression method (which do not always decrease with k), and on average our method outperforms the regression, achieving lower prediction errors by 18% (CIFAR), 32% (LFW) and 22% (fMRI). On the other hand, the KDE method achieves lower variance but high bias in two of three experiments (Figure 3C), where it is outperformed by CleaneX by 7% (CIFAR) and 73% (IMRI). However, in contrast to the simulation results and its performance on the CIFAR and IMRI datasets, the KDE method achieves exceptionally low bias on the LFW dataset, outperforming our method by 38%. Overall, CleaneX produces more reliable results across the experiments.
(A) CleaneX
(B) Regression
(C) KDE
(D) RMSE
(E) Rel. RMSE
1..
Avaage RMSE; D1G4
B38SE
D:1
CIFAR 100
3.8-
3,00007
3.6
0.10
0101
1.0
RMSE:
Average R33SE:
B3SSE
0.70
LEW
0.00
0101
RMSE:
RMSE
0.51
DMRL
3.8-
0.00
364
de
CleaneX
Regression
KDE
Figure 3: Experimental results. A. B, C: accuracy curves of the three datasets as predicted by CleaneX, regression and KDE, respectively; dotted vertical lines denote kis grey curves correspond to Aht at each repetition, black curves correspond to E, [A] for < S l:z; average RMSE taken over all 50 repetitions. D: distribution of RMSE values over the 50 repetitions. E: distribution of the ratio between RMSE values of competing methods and CleaneX - values above (orange dotted line) indicate that CleaneX outperforms the competing methods (charts capped at 4). In D and E, boxes show lower quartile, higher quartile and median; whiskers show values at 1.5 IQR from box; outliers are omitted from the figure for clarity.
6 DISCUSSION
In this work we presented the reversed ROC and showed its connection to the accuracy of marginal classifiers. We used this result to develop a new method for accuracy prediction.
8

---

Published as a conference paper at ICLR 2021
Analysis of the two previous methods for accuracy extrapolation reveals that each of them uses only part of the relevant information for the task. The KDE method estimates C2 based only on the scores, ignoring the observed accuracies of the classifier. Even when the estimates of Cz unbiased, the exponentiation in Ex [C introduces bias. Since the estimatiin is not calibrated using the observed accuracies, the resulting estimations
often biased. As can be seen in Figure 2 the bias is higher for L1 = 500 than for Lj = 100, indicating that this is aggregated when k1 is small. In addition, we found the method to be sensitive to monotone transformations of the scores, such as taking squared-root or logarithm. In contrast, the non- parametric regression based method uses pre-chosen basis functions predict the accuracy curves and therefore has limited versatility to capture complicated functional forms. Since it ignores the values of the classification scores, the resulting predicted curves not necessarily follow the form of Ex [C introducing high variance estimations. As can be seen in Figure 2 the variance is higher for L 100 compared to k1 = 500, indicating higher variance for small values L1. This can be expected since the number of basis functions that are used to approximate the discriminability function depends on k1.
In CleaneX, we exploit the mechanism of neural networks in a manner that allows us to combine both sources of information with less restriction the shape of the curve. Since the extrapolation follows the known exponential form using the estimated C₂ values it is characterized with low variance, and since the result is calibrated by optimizing an objective that depends the observed accuracies, our method has low bias, and therefore consistently outperforms the previous methods.
Comparing the results for k 500 between different dimensions (d 5 in Figure and d = 3, 10 in Figure(o) it can be seen that the bias of the KDE based method and the variance of the regression based method are lower in higher data dimensions. Zheng & Benjamini (2016) show that if both X and Y are high dimensional vectors and their joint distribution l'actorizes, then the classification scores of Bayesian classifiers (which are marginal) are approximately Gausssian. The KDE and the regression based methods use Gaussian kernel and Gaussian basis functions respectively, and are perhaps more efficient in estimating approximately Gaussian score distributions. Apparently, scores for real data-sets behave closer to oow-dimensional data, where the flexibility of CleaneX (partly due to being a non-parametric method) is an advantage.
considerable source of noise in the estimation is the selection of the kn classes. The A curves diverge from [AA as the number of classes increases, and therefore it is hard to recover if the initial k1 classes deviated from the true accuracy. The effect of choosing different initial subsets can be seen by comparing the grey curves and the orange curves that continue them, for example in Figure 3 We leave the design of more informative sampling schemes for future work.
To conclude, found the method we present to be considerably more stable than previous methods. Therefore, it is now possible for researchers to reliably estimate how well the classification system they are developing will perform using representations learned only on subset of the target class set.
Although work focuses marginal classifiers, its importance extends beyond this class of algo- rithms. First, preliminary results show that our method yields good estimates even when applied to (shallow) non-marginal classifiers such as multi-logistic regression. Moreover, if the representation can adapt as classes are introduced, we expect accuracy to exceed that of fixed representation. We can therefore use our algorithm to measure the degree of adaptation of the representation. General- ization of our method to non marginal classifiers is prominent direction for future work.
ACKNOWLEDGMENTS
We thank Ftam Benger for many fruitful discussions, and Itamar Faran and Charles Zheng for com- menting on the manuscript. YS is supported by the Israeli Council For Higher Education Data- Science fellowship and the CIDR center at the Hebrew University of Jerusalem.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Felix Abramovich and Marianna Pensky. Classification with many classes: challenges and pluses. Journal of Multivariate Analysis, 174:104536, 2019.
Brandon Amos, Bartosz Ludwiczuk, Mahadev Satyanarayanan, et al. Openface: A general-purpor face recognition library with mobile applications. CMU School of Computer Science, 6(2), 2016.
Bhaskar Bhattacharya and Gareth Hughes. On shape properties of the receiver operating character- istic curve. Statistics Probability Letters, 103:73-79, 2015.
Ricardo Cao, Antonio Cuevas, and Wensceslao González. Manteiga. A comparative study of several smoothing methods in density estimation. Computational Statistics & Data Analysis, 17(2):153- 176, 1994.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0m 088803, 2016.
Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 2788):661-874 2006.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611. 2006.
Luzia Gonçalves, Ana Subtil, M Rosário Oliveira, and P d Bermudez. Roc curve estimation: An overview. REVSTAT-Statistiee Journal, 12(1):1-20, 2014.
Gregory Griffin, Alex Holub, and Pietro Perona. griffin200ccatech.. Technical report, California Institute of Technology, 2007.
Gary B. Huang. Manu Ramesh, Tamara Berg, and Erik Learned-Mille Labeled faces in the wild: database for studying face recognition unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007.
Vidit Jain and Erik Learned-Mir Miller. Fddb: A benchmark for face detection in unconstrained settings. Technical report, UMass Amherst Technical Report, 2010.
Kendrick N Kay, Thomas Naselaris, Ryan Prenger. and Jack Gallant. Identifying natural images from human brain activity. Nature, 452(7185):352-33 2008.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Vitaly Kuznetsov, Mehryar Mohri, and Umar Syed. Multi-class deep boosting. In Advances in Neural Information Processing Systems, pp. 2501-2509, 2014.
Yunwen Lei, Urun Dogan, Alexander Binder, and Marius Kloft. Multi-class syms: From tighter data-dependent generalization bounds to novel algorithms. In Advances in Neural Information Processing Systems, pp. 2035-2043, 2015.
Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, and Weiping Wang. Multi-class learning: From theory to algorithm. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 1586-1595. Curran Associates, Inc., 2018.
10

---

Published as a conference paper at ICLR 2021
Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. Deep learning for extreme multi- label text classification. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 115-124. ACM, 2017a.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-21222220 220, 2017b.
Malik Magdon-Isian Ismail and Amir F Atiya. Neural networks for density estimation. In Advances in Neural Information Processing Systems, pp. 522-528, 1999.
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400 410, 2011.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level im- age representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724, 2014.
Sinno Jialin Pan, Qiang Yang, et al. A survey transfer learning. IEEE Transactions knowledge and data engineering, 22(10)::345-133 2010.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 25222,2211222,2222 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering, In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015.
Katja Seeliger, Matthias Fritsche, Umut Güiçlü, Sanne Schoenmakers, J-M Schoffelen, SE Bosch, and MAJ Van Gerven. Convolutional neural network-based encoding and decoding of visual object recognition in space and time. Neurolmage, 180:253-266, 2018.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator. In Inter- national Conference on Machine Learning, pp. 467-475, 2014.
Charles Zheng, Rakesh Achanta, and Yuval Benjamini. Extrapolating expected accuracies for large multi-class problems. Journal of Machine Learning Research, 19(65):1-30, 2018.
Charles Y. Zheng and Yuval Benjamini. Estimating mutual information in high dimensions via classification error. arXiv e-prints, art. arXiv:1606.0527) Jun 2016.
11

---

Published as a conference paper at ICLR 2021
A PROOF OF THEOREM 1
We begin by proving the following simple proposition: Proposition
Given 2; let
G1
y/ (x)x) (x) sup Py (Sy (z) 2 u/
(13)
and
G2 {v Py (S (x) > (2)) u)
(14)
then G1 = G2-
Proof.
Let Gi, G5 be the complement sets of G1. G2, respectively. Then
G = E 52)2) s sup PP (Sy (x) > 2 u}}
(15)
and
= ly Pr (S y (1) Sy (x)) u).
(16)
Since py (Sy (x) > t) is not increasing in t we have
y E G. + S,(x) S sup (yy, (Sy (x) > 2 m)
+ Py (S (z) Sy (x)) 2u
(17)
- E G
and therefore G1 G2-
We now prove Theorem I which was presented in Section
Proof. We show that rROC (1 - u) = D (u), and the rest follows immediately according to
Theorem of Zheng et al. (0188 (see Equation 4
1 - --ROC, (1 u) = 1 Sy (x) sup (Py (Sy (x) 2) m]]
(Definition)
D
= 1 [Pr, (r) > (z)) < <1-w
(Proposition
[Pr, (S (x) < Sy)(x) <u 1 |Pr (S, (z) < S,-(x)) < u
(18)
where 1[-] denotes the indicator function, and the last equality follows from the assumption that (z) # Sy(r) almost surely. From here,
1 1- 1-rROC (1 u)
(19)
Ex [1 (Py, (Sy (x) < (x)(x)) s u)]
PP (Pyr (8 < (z)) S su)
= D(u)
required.
B SIMULATIONS ADDITIONAL RESULTS
Figures and show the accuracy curves for which we presented summarized results in Figure 2
In Figure 6 we provide additional simulation results for data dimensions d' = 3, 10. We simulate eight d-dimensional datasets of k 2000 classes and extrapolate the accuracy from kj 500 classes. As in Section 5.1 the datasets are combinations of two distributions of Y and two of X Y. in two different levels of classification difficulty. We sample the classes //....., yk, from a multivariate normal distribution (0, I) or multivariate uniform distribution 2d-11, 1). We sample 10 data points for each class from multivariate normal distribution N aT)) or from
12

---

Published as a conference paper at ICLR 2021
Cleaxe
R XIV N =.01
Y- V. XTY -..
0.5
=02
Y- M, NY 3Y--
You XIV -1
Y Tuu xTV -X
Y-M N1L A
Y Youk XIV -
XIV -2
102
1901
510
1000
1760
90 1000
Figure 4: Comparison of predicted accuracy curves produced by CleaneX (left, orange), regression based method (middle, blue) and KDE (right, purple), the eight simulated datasets with d and k 100 (dotted vertical line). The curves of Ali for each repetition are shown in grey. The black curves correspond to [A] for 2 < k < K2 2000.
13

---

Published as a conference paper at ICLR 2021
Cleaxe
V- R XIV - 5.0.1
Y- V. XTT -..
0.5
=02
Artmur
Y- N. NY -
Youk XIV -1
0.00
Y Y-u XTV -X
Y Y-M 31L A
Y Foud XIV -
XYY-a -2
-02
1901
1760
1200
Figure 5: Comparison of predicted accuracy curves produced by CleaneX (left, orange), regression based method (middle, blue) and KDE (right, purple), the eight simulated datasets with d and k 500 (dotted vertical line). The curves of Ali for each repetition are shown in grey. The black curves correspond to [A] for 2 < k < K2 2000.
14

---

Published as a conference paper at ICLR 2021
multivariate uniform distribution Uyy - o², 4 o²). For = set Ø² to 0.1 for the easier classification task and 0.2 for the difficult one. For 10, the values of o² are set to 0.6 and 0.9, respectively. The classification scores (x) are the euclidean distances between 2 and
can be seen that the results for = 3 are consistent with those for = 5. That is, our method outperforms the competing methods, with datasets. For 10 the regression based method achieves comparable results to ours. As in the other simulations, method produces less variance than the regression and less bias than KDE,
bigger gap, in seven of the eight simulated
N, - o2-02
N. X|V
Y-M, X.Y N
Y-M. X|V
of
0.53
0.11
e² DUB
c² 0.3
of 0.8
of
of 0.9
of 0.8
0.54
0.02
10:3
CleaneX
Regression
KDE
Figure 6: Additional simulation results. For each scenario we show a boxplot representing RMSE values obtained over 50 repetitions using CleaneX (left box, orange), regression based method (middle box. blue) and KDE (right box, purple). The boxplots for the KDE method are not shown when all obtained RMSE values are higher than 0.07. The boxes extend from the lower to the upper quartile values, with a line at the median; whiskers show values at a distance of at most 1.5 IQR (interquartile range) from the lower and the upper quartiles; outliers omitted from the figure for clarity.
C IMPLEMENTATION DETAILS
All the code this work was implemented in Python 3.6. For the CleaneX algorithm we used TensorFlow 1.14; for the regression based method used the scipy aptimiza ze package with the "Newton-CG" method; kernel density estimation was implemented using the densi tty function from the stats library in R, imported to Python through the rpy2 package.
15


Reviews:
[{"review_id": "N7fVz0OHN0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors show a relationship between classification accuracy and reverse ROC in multiclass classifiers, when there are new classes not seen in the training data. They propose a method called CleaneX that learns to estimate the accuracy of multiclass classifiers on arbitrarily large sets of classes. \n\nMajor Comments:\n\nThe concept seems rather novel; however, I am not very familiar with the literature in evaluating one-shot learning classifiers. The paper in general seems rather \"empty\" in that the authors simply show things, but do not explain the importance of them or spend time motivating the problem. For example, what practical use is there to know the expected accuracy of a classifier for predicting k classes? The use of knowing the accuracy for a specific class is obvious, but the overall accuracy is unclear.\n\nThe authors should not include the KDE in the box plots for Figure 2 for k_1=500 as they are much worse than the other two methods and stretches the axis making the other two methods very difficult to see. A note that the KDE performs much worse and isn't shown, or put in the appendix should suffice. \n\nHow is the black line representing the \"true\" accuracy curve calculated for the experiments?\n\nFor Figure 3, the authors should find a better way to display the performance that does not require eyeballing how well the colored lines follow the shape of the black one. Furthermore, it seems like for the LFW dataset, the KDE does much better overall, and for the CIFAR dataset both CleaneX and Regression seem to perform comparably. \n\nIn general, the authors provide some intuition on why the CleaneX method performs better than the others in the discussion, but this should be more closely tied to the experiments. E.g. the simulations should be pointed to, when the authors discuss how the accuracy curves converge to one-parameter families when dimensionality increases. There should also be more discussion on potentially why the CleaneX method is better than the others. E.g. why does the KDE method perform so poorly in high dimensionality; is it an artifact of being a non-parametric method in a \"parametric\" problem due to the simulations? Why does the Regression method have high variance occasionally?\n\nUPDATE: Based on the Author's response, I have updated my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "N4ZRLWLW-Ld", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors discuss how a classifier’s performance over the initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes by mean of the dual of the ROC function, swapping the roles of classes and samples. Grounded on such function, the authors develop a novel ANN approach learning to estimate the accuracy of classifiers on arbitrarily large sets of classes. Effectiveness of the approach is demonstrated on a suite of benchmark datasets, both synthetic and real-world.\n\nThe manuscript is well written and understandable also by a non-specialist audience; the reference list is up-to-date and the introduction properly details the motivations for tackling the problem. The underlying math is sound, and the proposed solution is smart, but the experimental section is not convincing, and hardly supporting the authors’ claims. Overall, I would vote for a weak accept.\n\nPros:\n- The proposed solution is grounded and interesting, and the results shown are encouraging;\n- When optimized/improved, CleaneX may have a relevant impact on the multi class classification theory,\n\nCons:\n- Classifiers are compared on the basis of RMSE, although the original problem is multi class classification; I would strongly suggest a more classification-oriented measure such as multiclass MCC.\n- CleaneX is compared only to regression and KDE - what about adding also very widespread algorithms such as RandomForest?\n- Performance gain w.r.t. regression is quite limited, especially on real-world datasets: would CleaneX benefit from adding dropout layers or more refined activation functions?", "extra_scores": {}, "extra_text": {}}, {"review_id": "JteZXHKkZaN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality: \nRigorous treatments of the ROC and reverse ROC concepts were offered. Cumulative distribution of class scores were used to estimate accuracy on a multi-class classification problem. \n\nClarity: The paper was well written and organized. \n\nOriginality/Significance:\nThe reverse ROC concept and its role in predicting accuracy of marginal classifiers for a large number of unknown classes seems to be original. Technical novelty is limited but the proposed approach could have a decent impact in the literature for a variety of problems involving large number of classes such as extreme classification, zero and one shot learning, open-world classification, multi-task learning etc. \n\nDetailed Comments:\nThis paper shows that for marginal classifiers the expected accuracy can be estimated in terms of the CDF of incorrect scores, i.e., the probability of the correct class to outscore a randomly chosen incorrect one for a data point x, which is denoted by C_x. The proposed work trains a simple neural network to learn the mapping from class scores of x onto C_x. NN is trained to minimize the difference between the expected accuracies obtained from C_x and true accuracies predicted for the first k_1 classes. Once NN is trained it can be used to obtain C_x for k_2 classes (k_2>>k_1), which are in turn used to predict accuracy for the k_2 classes. \n\nThe paper introduces the reverse ROC concept to offer an interesting interpretation to expected accuracy of k classes. Unlike earlier work that uses KDE and non-parametric regression to predict the classification accuracy for a larger number of classes the proposed work uses a neural network for the same task.\n\nExperiments performed on three different datasets (CIFAR100, Face recognition, and Brain decoding) suggest the proposed technique significantly outperforms KDE-based technique in predicting accuracy on a larger class set and also seems to be slightly more competitive than non-parametric regressio method. \n\nThe main technical novelty lie in the reverse ROC interpretation of the average accuracy. The NN part is just a substitute for the nonlinear regression Zheng et al. used. Both aims to achieve the same thing (learning a nonlinear mapping between class scores and CDF evaluation at the correct score) and results between the two techniques also seems to be very consistent. \n\nIn Algorithm 1 I am not quite sure how a neural net with a fixed number of input nodes deals with varying number of inputs? C_x is the CDF of incorrect scores evaluated at the correct score. The size of the input is k but k changes from 1 to k_1. When k is not equal to k1 there are k scores to be mapped to and therefore the input to the NN has to be k but k changes for every A_{k}.  Different subsets of the neural net weights have to be used to get \\bar{C}_x^(k-1) for different k's, which does not make much sense. Need some clarification here. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is very clear.  It provides a good overview of the problem, making it easy to follow even for researchers outside the area.  

This work provides a novel approach for extrapolating the expected accuracy on a larger set of classes from a training set with smaller number of classes with a creative, simple and elegant solution through reversed ROC.  Such an approach will be useful for extreme classification settings.  In real-world settings, classifiers are often trained on a pilot set of data, and then deployed where the classes are much larger.  It is useful to have a mechanism to estimate how the classification performance will change with larger number of classes.  

The reviewers all agree that this work provides a novel contribution to predicting classification accuracy.  The authors have satisfactorily addressed the reviewers’ comments and provided sufficient clarification to the questions.  We also appreciate the edits that the authors have made.  

Decision: Accept (Poster)