SPARDACUS SafetyCage: A new misclassification detector

Venue: NLDL_org_2025_Conference
Published: 2024-09-06 16:48:04

Abstract: Given the increasing adoption of machine learning techniques in society and industry, it is important to put procedures in place that can infer and signal whether the prediction of an ML model may be unreliable.
This is not only relevant for ML specialists, but also for laypersons who may be end-users.
In this work, we present a new method for flagging possible misclassifications from a feed-forward neural network in a general multi-class problem, called SPARDA-enabled Classification Uncertainty Scorer (SPARDACUS).
For each class and layer, the probability distribution functions of the activations for both correctly and wrongly classified samples are recorded.
Using a Sparse Difference Analysis (SPARDA) approach, an optimal projection along the direction maximizing the Wasserstein distance enables $p$-value computations to confirm or reject the class prediction.
Importantly, while most existing methods act on the output layer only, our method can in addition be applied on the hidden layers in the neural network, thus being useful in applications, such as feature extraction, that necessarily exploit the intermediate (hidden) layers.
We test our method on both a well-performing and under-performing classifier, on different datasets, and compare with other previously published approaches.
Notably, while achieving performance on par with two state-of-the-art-level methods, we significantly extend in flexibility and applicability.
We further find, for the models and datasets chosen, that the output layer is indeed the most valuable for misclassification detection, and adding information from previous layers does not necessarily improve performance in such cases.

Main paper: 
 spardacus: 
 Consider the function F i,l which corresponds to the PDF that generates the activation values at layer l for a sample correctly predicted as class i by the NN classifier. Conversely, let G i,l correspond to the PDF that generates the activation values for incorrectly classified samples. Given these two PDFs, one may infer which distribution a new test samplex ′ belongs to by designing a decision procedure to flag wrongly classified samples. In principle this is a binary classification problem to which we could apply any ML method to predict if x ′ is correctly or wrongly classified; but if F i,l and G i,l could be approximated directly, statistical tests backed by solid theory would become available. A direct application of this procedure is however challenging, since the dimensionality of the PDFs is linked to the size of the NN classifier's layers; i.e. large layers imply high-dimensional PDFs. To combat this, we project the data for each layer and class to one dimension, 151 effectively collapsing the multi-dimensional PDFs 152 into one-dimensional ones. The goal is to obtain two 153 PDFs along this 1D-projection, denoted f i,l and g i,l , 154 which are minimally overlapping. The PDFs are 155 estimated using the same training data the classifier 156 is constructed from.To this end, Mueller et al. [7] propose an 158 approach referred to as a Sparse Differences 159 Analysis (SPARDA), which given observed samples 160 from two multivariate PDFs, searches for the 161 optimal projection maximising the Wasserstein 162 distance between the projected 1D empirical 163 distribution functions (ECDFs). This is a non-164 smooth, non-concave optimization problem. We 165 apply the fastSPARDA optimization algorithm 166 available at https://bitbucket.org/jwmueller/ 167 principal-differences-analysis/src/master/. 168 The optimization problem includes a regularization 169 parameter λ to induce sparsity in the projection. We 170 denote the projection direction given by SPARDA 171 as βi,l , and ϱ i,l as the projected random variable 172 of activations along βi,l . Figure 1 summarises the 173 projection operation and notation.If a new sample x ′ is predicted to be a member 175 of class i, one can infer at any given layer l whether 176 it is more likely generated from f i,l (ρ) or g i,l (ρ) by 177 inspecting the observed value ρ along the projection. 178 However, f i,l (ρ) and g i,l (ρ) are unknown. To over-179 come this, we fit each PDF as a Gaussian mixture 180 model due to its flexibility and computational effi-181 ciency. A typical situation with overlapping can be 182 seen in Figure 2 showing histograms of f 7,-1 and 183 g 7,-1 on training data and test data for the class 184 (digit) 7 from MNIST, with l = -1 indicating the 185 output layer. With inspiration from the likelihood ratio test we 188 define the statistic, S i,l (ϱ i,l ), using the two afore-189 mentioned PDFs for a random variable ϱ i,l with 190 observed values along the projection βi,l : Moreover, consider the statistic S i,l C (ϱ i,l f ) where ). From this we can estimate the corresponding p-values as:EQUATIONϱ i,l f ∼ f i,l (ρ). The corresponding PDF of S i,l C , de- 194 noted h C (s i,l c ),EQUATION)pl S C (s ŷ′ ,l ) = 1 -F ŷ′ ,l c (s ŷ′ ,l ),and pl S W (s ŷ′ ,l ) = F ŷ′ ,l w (s ŷ′ ,l ).Whether to use S i,l C or S i,l W to get the most robust results will be based on the accuracies of the estimated PDFs f i,l and g i,l .The notation so far has been focused on a particular layer l. By our method, each layer l provides a p-value. These p-values can be combined into one, using any p-value combination test. To this end, we will apply the Cauchy combination test as it is robust for statistically correlated p-values [8] . We denote pS C and pS W the final p-values when using either statistic S C or S W , respectively. Algorithms for the method is given in B.1 and B.2 in the Appendix where we separate between the training phase of the SPARDACUS method, and the subsequent detection procedure. 
results: 
 We will evaluate our method by using the same setup as in [4] , where a feed-forward neural network is trained on MNIST, yielding a well-performing model (accuracy of 0.98), and on CIFAR-10 yielding a poor-performing model (accuracy of 0.48). The neural network consists of two hidden layers, with 256 and 128 neurons respectively, and ReLu activation functions, along with an output layer featuring ten neurons with Softmax activation. Training utilized the standard Adam optimizer. Once the model is trained, our method estimates the projections βi,l and corresponding PDFs f i,l , g i,l for all classes and layers (except the input layer, i.e. the images). On a held-out test data disjoint from the data the NN model was trained on, we evaluate our misclassification detector for different values of α C and α W . Additionally, we present tables showing the precision, recall, specificity, negative predictive value as well as the MCC for the best performing α value.We advocate for using the Matthews correlation coefficient (MCC) as the most meaningful performance metric to evaluate any binary classification model [9] . The MCC ranges from -1, meaning the classifier is always wrong, to 1 meaning a perfect classifier. A coin tossing classifier with a 50 % chance to assign a prediction to either of the two classes will give MCC = 0 [9] . Also by definition, a classifier that predicts only one class every time will give MCC = 0 (see [9] for more details). We compare our method with the MSP-detector method introduced in [2] , the DOCTOR method [3] , and the SafetyCage method introduced in [4] . The threshold in SPARDACUS and in [4] can be interpreted as the same, as they both are based on p-values. The threshold for the MSP-detector method is with respect to the probability of the prediction, while for the DOCTOR method it is with respect to the estimated odds of misclassification.For the Mahalanbois SafetyCage and the SPAR-DACUS method, we evaluate the results for different layer aggregations (combining p-values from different layers) to investigate the information that is covered in the different layers. Specifically, we investigate when only applying the output layer (out), the penultimate layer (pen) or all hidden layers plus the output layer (all).In all results to come, the SPARDA projection βi,l was computed by setting λ = 0, imposing no regularization, and using the fastSPARDA algorithm. We Notice that for all methods, only using the output layer turned out to give the largest MCC value. The same applies to the SafetyCage Mahalanobis, an aspect that was not investigated in the original paper [4] where only hidden layers up to the penultimate layer were investigated.We also show the precision, recall, specificity, and negative predictive value (NPV) for each case. While the outputs of the MSP and DOCTOR methods are deterministic, the 1D-projections calculated in SPARDACUS using the SPARDA algorithm, result in stochasticity. For this method we therefore show the result of five runs in terms of mean and standard deviation. Notice the small variation in MCC. After closer inspection, this is due to nearly equal 1Dprojection.In Figures 3 and 4 , for each of the misclassification detectors, we show how the MCC varies for different threshold values on the test data for MNIST and CIFAR-10 respectively. We see that the SPAR-DACUS method is most sensitive to the choice of threshold with respect to performance, with a sharp peak for the optimal threshold. At the same time, if we compare the MCC values at the peaks for each method, the SPARDACUS method achieves by a small margin the highest MCC values for both datasets. See Table C .1 in Appendix C where the MCC values at the peaks for the SPARDACUS method are extracted. Here, we also include results 339 when applying different sets of layers for the SPAR-340 DACUS method which confirms that only using the 341 output layer yields the highest MCC values.In practice we do not know in advance what the 343 optimal threshold is. To estimate the threshold on 344 the same data that the ML model is trained on can 345 make sense in terms of utilizing all data available, 346 particularly for data-driven methods such as the Ma-347 halanobis SafetyCage and the SPARDACUS, where 348 also PDFs are fitted to data. However, in certain 349 situations the training data is not available, and 350 instead new data must be collected during deploy-351 ment. Moreover, using the training data may lead to 352 overfitting and less generalization capabilities. For 353 comparison, we include the scenario where the pa-354 rameters needed for each misclassification detection 355 method, including the threshold value, are estimated 356 based on data never used by the ML model. We 357 do this by randomly splitting the test data (10 000 358 samples) equally in two subsets, estimate the param-359 eters on the first subset, and evaluate the detection 360 methods on the second subset. To account for dif-361 ference in performance with respect to the splitting 362 of data, we repeat the process five times each with 363 random splitting of the test data. given in Tables 3 and 4 .we see that the SPARDACUS method is superior to the SafetyCage Mahalanobis method from [4] . Moreover, compared to the MSP-detector and DOCTOR methods, the SPARDACUS method using the S C and S W is essentially on par.As expected, all methods performed much better on the well-performing model, due to there being more useful information encoded in the activation values that could be extracted and used to flag incorrect predictions. Interestingly, the output layer was by far the most useful layer, which aids to show how this is a markedly different problem compared to OOD where the output layer can be less informative [5] . An interesting observation can be made regarding the SPARDACUS method yielding the best results when evaluating only the output layer.When inspecting the fitted projection at the output layer, βi,-1 , and specifically looking at using S Cfor the MNIST model, we see that for every predicted class, the associated projection vector has its maximal-value element in the position corresponding to the predicted class itself. In fact, on average the maximum value along the class prediction dimension was 2.88 times larger than the second largest element. This shows that the projection vector for a specific class is heavily dominated by the dimension along the class itself. This is in fact in correspondence with the MSP-detector method, where the corresponding projection would have zero-elements along all dimensions except for the class dimension.This shows that, in the special case of being applied to only the output layer, SPARDACUS can be seen as an extension of the MSP-detector method.In all cases, S C performed better for the wellperforming model, while S W was better for the poorperforming model. 
discussion and conclusion 403: 
 In this work, we presented a method to infer whether 404 a particular sample is wrongly classified by an un-405 derlying NN model. SPARDACUS is based on a 406 SPARDA projection maximizing the Wasserstein 407 distance of the PDFs of the samples that were cor-408 rectly and wrongly classified, and a hypothesis test 409 inspired by the likelihood-ratio test. SPARDACUS 410 can be applied at any stage of an NN classifier, and 411 with an easy extension it could also draw information 412 from any arbitrary combination of layers. 
: 
 world setting, the reliability of its predictions need to be considered in some way prior to making actual decisions (such as stopping the production line).In most neural network (NN) classifiers, a softmax activation function is utilized on the output layer to interpret each value as the probability of belonging to a particular class. The class prediction for any sample is most often equal to the class with the largest softmax probability in the output layer.It has been shown that misclassifications may arise even when the largest softmax probability is close to one [1] . Nonetheless, a pattern was discovered where the maximum softmax probability tended to be smaller for incorrectly classified samples than for correctly classified samples. This discovery was used to make a simple threshold-based misclassification detector, see [2] , called Maximum Softmax Probability (MSP) Detector.In [3] , a method named DOCTOR for misclassification detection was proposed, based on an approximation of the misclassification probability, Pe(x), for a particular sample x by only using the softmax output layer values, P Ŷ |X (c | x), for each class c of total C classes. In particular Pe(x) ≈ 1 -C c=1 P 2 Ŷ |X (c | x).The method flags a prediction as untrustworthy whenever the odds of a misclassification event is larger than some threshold.From our literature search, it is apparent that the DOCTOR and MSP-detector methods represent the current state of the art of misclassification detection, which we use for comparison to SPARDACUS.The SafetyCage, introduced in [4] , is another misclassification detector. This statistical framework collects the pre-activation vector in each layer, and assumes the corresponding multivariate probability density function (PDF) of correctly predicted in-distribution samples to follow a Gaussian distribution. These PDFs, per class, are fitted to the training data of the NN model. To infer the uncertainty of a class prediction, the Mahalanobis distance, inspired by the approach described in [5] , is used to measure the likeliness that the pre-activation values, in each layer, are generated from the fitted Gaussian distribution of correctly predicted samples. This Mahalanobis-based SafetyCage was tested on two feed-forward neural networks trained on benchmark datasets MNIST and CIFAR-10, respectively. The classifier trained on the MNIST dataset had an accuracy of 0.93, whereas the one trained on CIFAR-10 1 We note that a task related to the detection of misclassifications is what is called out-of-distribution (OOD) detection, where the aim is to detect whenever an input sample is inherently different from the data used during training of the model, and hence the corresponding prediction should not be trusted. However, the most insidious misclassifications happen with in-distribution data, for which the model would be assumed to work correctly, and not OOD-data. Indeed, in [6] it is shown that the best OOD-detector is not always the best at detecting NN classification errors. The authors in [6] further emphasize that if the focus is on use of NNs in safety-critical applications, misclassification detection should be the paramount focus, and not OODdetection. For these reasons, this work focuses on misclassification detection using in-distribution data, and will not draw a comparison to OOD-detection methods. 
 
 Appendix: We tested SPARDACUS on two simple NN clas-414 sifiers, one well-performing trained on MNIST and 415 one poorly performing trained on CIFAR-10. The 416 results have further been compared with three pre-417 existing methods from the literature: The DOCTOR 418 method, the Mahalanobis-based method presented 419 in [4] , and the MSP-detector method [2] .Our results show that SPARDACUS significantly 421 outperforms the Mahalanobis SafetyCage [4] , where 422 in particular, detection performance on the inaccu-423 rate classifier for CIFAR-10 has been improved by 424 an order of magnitude. The MSP, DOCTOR and 425 SPARDACUS are essentially performing at com-426 parable levels with respect to the MCC values as 427 shown in Tables 1, 2 Other than that the MSP and DOCTOR method is static in terms of deterministic computations (in the output layer only), while the SPARDACUS method can be investigated further in several ways. Here, we list some aspects to investigate: 1) We have used the Wasserstein distance, but one may substitute it with other statistically-relevant distances.2)The SPARDA projection can be investigated further using other optimization algorithms.  


Decision: This paper addresses the problem of reliability and misclassification detection in machine learning. The reviewers appreciate the novelty, the writing, and the research direction. After the review and rebuttal stage, the reviewers reached a consensus to accept the paper. The reviewers have some final questions and concerns that could be addressed. For example, one novelty claim is that the proposed method can be used in all layer, but this also seems the case for SafetyCage, hence some clarification is required. There are also shared concerns about hyperparameter tuning. The AC agrees with the positive assessment and encourages the authors to update the paper based on the pointers by the reviewers.

Reviews: review: title: title: There are valuable contributions in the paper
paper_summary: paper_summary: The paper suggests a way of measuring probability of misclassification of a machine learning model. The paper suggests to use intermediate layers activations in addition to the output layer activations in this process. It is known that when the model outputs uncertain predictions (maximum softmax probability is small) the model is more likely to misclassify, which is what currently known detectors such as MSP and DOCTOR do. Other layer activations, apart from the final softmax, are used in known methods, such as SafetyCage, but there are limitations of this approach which are solved by proposed paper. 
The paper proposes to project the activation statistics for each class into 1D spaces for each layer and class to obtain 1D probability densities using SPARDA method; and then use machine learning or statistical tests to classify between two states: correctly predicted and misclassified. This is in contrast to previous work, SafetyCage, which used out of distribution detection method: the proposed direct training of the final misclassification detector on both correctly and incorrectly classified examples is expected to improve robustness and accuracy of the detector.
main_review: main_review: - The rebuttal well addressed the main concern of multiple reviewers: threshold selection is now properly done on a separate validation dataset. However, limited datasets reviewer's question was not addressed. Additionally, proper theshold selection revealed that proposed method does not outperform all competitors on the given datasets.
- My final rating is not changing, recommending acceptance. Lack of additional datasets evaluation and revealed updated results of comparison with other methods beg a question of requesting a further clarification on the part of the experiments, therefore I would highly encourage authors to additionally address the question of limited datasets evaluation (evaluate at least on another non-image dataset). This however does not invalidate the theoretical contribution, justifying the currently given rating.
strength_weakness: strength_weakness: First, there is a fundamental question about the extent to which we can use misclassification detectors to further enhance the prediction quality of the model. If the misclassification detector uses only information from the model itself, in case of misclassification that information will be incorrect and therefore may not be suitable for further judgement.  However, the existance of other misclassification detectors shows that it is indeed possible to get extra information from the model itself. Still it will be nice to touch this question in the paper and provide additional justification about why the proposed detector is able to gather that extra information from the model.

Second, there is a serious problem with tuning threshold on the test set, as described in line 308 mentioning "optimal threshold on the test data". The paper also mentions " threshold optimized on the training data" in line 319. I believe that both variants are not correct: test threshold shall never be used according to the golden rule of not using the testing set for any calculations that affect final predictions (would be better to remove such results); and training threshold is not optimal as suggested, due to possible overfitting.  Instead, validation set should be used for this purpose. On Figures 3,4 we see that proposed method is very sensitive to the threshold, which additionally confirms the fact that threshold should be re-evaluated correctly on correct data subset.

Additionally, evaluation is limited to two specific cases of image datasets: MNIST and CIFAR-10. would be nice to test the method on non-image datasets, such as the openly available datasets from the UCI datasets repository.