Overinterpretation reveals image classification model pathologies

1 Introduction . Well-founded decisions by machine learning ( ML ) systems are critical for high-stakes applications such as autonomous vehicles and medical diagnosis . Pathologies in models and their respective training datasets can result in unintended behavior during deployment if the systems are confronted with novel situations . For example , a medical image classifier for cancer detection attained high accuracy in benchmark test data , but was found to base decisions upon presence of rulers in an image ( present when dermatologists already suspected cancer ) [ 1 ] . We define model overinterpretation to occur when a classifier finds strong class-evidence in regions of an image that contain no semantically salient features . Overinterpretation is related to overfitting , but overfitting can be diagnosed via reduced test accuracy . Overinterpretation can stem from true statistical signals in the underlying dataset distribution that happen to arise from particular properties of the data source ( e.g. , dermatologists ’ rulers ) . Thus , overinterpretation can be harder to diagnose as it admits decisions that are made by statistically valid criteria , and models that use such criteria can excel at benchmarks . We demonstrate overinterpretation occurs with unmodified subsets of the original images . In contrast to adversarial examples that modify images with extra information , overinterpretation is based on real patterns already present in the training data that also generalize to the test distribution . Hidden statistical signals of benchmark datasets can result in models that overinterpret or do not generalize to new data from a different distribution . Computer vision ( CV ) research relies on datasets like CIFAR-10 [ 2 ] and ImageNet [ 3 ] to provide standardized performance benchmarks . Here , we analyze the overinterpretation of popular CNN architectures on these benchmarks to characterize pathologies . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Revealing overinterpretation requires a systematic way to identify which features are used by a model to reach its decision . Feature attribution is addressed by a large number of interpretability methods , although they propose differing explanations for the decisions of a model . One natural explanation for image classification lies in the set of pixels that is sufficient for the model to make a confident prediction , even in the absence of information about the rest of the image . In the example of the medical image classifier for cancer detection , one might identify the pathological behavior by finding pixels depicting the ruler alone suffice for the model to confidently output the same classifications . This idea of Sufficient Input Subsets ( SIS ) has been proposed to help humans interpret the decisions of black-box models [ 4 ] . An SIS subset is a minimal subset of features ( e.g. , pixels ) that suffices to yield a class probability above a certain threshold with all other features masked . We demonstrate that classifiers trained on CIFAR-10 and ImageNet can base their decisions on SIS subsets that contain few pixels and lack human understandable semantic content . Nevertheless , these SIS subsets contain statistical signals that generalize across the benchmark data distribution , and we are able to train classifiers on CIFAR-10 images missing 95 % of their pixels and ImageNet images missing 90 % of their pixels with minimal loss of test accuracy . Thus , these benchmarks contain inherent statistical shortcuts that classifiers optimized for accuracy can learn to exploit , instead of learning more complex semantic relationships between the image pixels and the assigned class label . While recent work suggests adversarially robust models base their predictions on more semantically meaningful features [ 5 ] , we find these models suffer from overinterpretation as well . As we subsequently show , overinterpretation is not only a conceptual issue , but can actually harm overall classifier performance in practice . We find model ensembling and input dropout partially mitigate overinterpretation , increasing the semantic content of the resulting SIS subsets . However , this mitigation is not a substitute for better training data , and we find that overinterpretation is a statistical property of common benchmarks . Intriguingly , the number of pixels in the SIS rationale behind a particular classification is often indicative of whether the image is correctly classified . It may seem unnatural to use an interpretability method that produces feature attributions that look uninterpretable . However , we do not want to bias extracted rationales towards human visual priors when analyzing a model ’ s pathologies , but rather faithfully report the features used by a model . To our knowledge , this is the first analysis showing one can extract nonsensical features from CIFAR-10 and ImageNet that intuitively should be insufficient or irrelevant for a confident prediction , yet are alone sufficient to train classifiers with minimal loss of performance . Our contributions include : • We discover the pathology of overinterpretation and find it is a common failure mode of ML models , which latch onto non-salient but statistically valid signals in datasets ( Section 4.1 ) . • We introduce Batched Gradient SIS , a new masking algorithm to scale SIS to highdimensional inputs and apply it to characterize overinterpretation on ImageNet ( Section 3.2 ) . • We provide a pipeline for detecting overinterpretation by masking over 90 % of each image , demonstrating minimal loss of test accuracy , and establish lack of saliency in these patterns through human accuracy evaluations ( Sections 3.3 , 4.2 , 4.3 ) . • We show misclassifications often rely on smaller and more spurious feature subsets suggesting overinterpretation is a serious practical issue ( Section 4.4 ) . • We identify two strategies for mitigating overinterpretation ( Section 4.5 ) . We demonstrate that overinterpretation is caused by spurious statistical signals in training data , and thus training data must be carefully curated to eliminate overinterpretation artifacts . Code for this paper is available at : https : //github.com/gifford-lab/overinterpretation . 2 Related Work . While existing work has demonstrated numerous distinct flaws in deep image classifiers our paper demonstrates a new distinct flaw , overinterpretation , previously undocumented in the literature . There has been substantial research on understanding dataset bias in CV [ 6 , 7 ] and the fragility of image classifiers deployed outside benchmark settings . We extend previous work on sufficient input subsets ( SIS ) [ 4 ] with the Batched Gradient SIS method , and use this method to show that ImageNet sufficient input subset pixels for training and testing often exist at image borders . Many alternative interpretability methods also aim to understand models by extracting rationales ( pixel-subsets ) that provide positive evidence for a class [ 8–11 ] , and we adopt SIS throughout this work as a particularly straightforward method for producing such rationales . This prior work ( including SIS [ 4 ] ) is limited to understanding models and does not use the enhanced understanding of models to identify the overinterpretation flaw discovered in this paper . We contrast the issue of overinterpretation against other previously known model flaws below : • Image classifiers have been shown to be fragile when objects from one image are transplanted in another image [ 12 ] , and can be biased by object context [ 13 , 14 ] . In contrast , overinterpretation differs because we demonstrate that highly sparse , unmodified subsets of pixels in images suffice for image classifiers to make the same predictions as on the full images . • Lapuschkin et al . [ 15 ] demonstrate that DNNs can learn to rely on spurious signals in datasets , including source tags and artificial padding , but which are still human-interpretable . In contrast , the patterns we identify are minimal collections of pixels in images that are semantically meaningless to humans ( they do not comprise human-interpretable parts of images ) . We demonstrate such patterns generalize to the test distribution suggesting they arise from degenerate signals in popular benchmarks , and thus models trained on these datasets may fail to generalize to real-world data . • CNNs in particular have been conjectured to pick up on localized features like texture instead of more global features like object shape [ 16 , 17 ] . Brendel and Bethge [ 18 ] show CNNs trained on natural ImageNet images may rely on local features and , unlike humans , are able to classify texturized images , suggesting ImageNet alone is insufficient to force DNNs to rely on more causal representations . Our work demonstrates another source of degeneracy of popular image datasets , where sparse , unmodified subsets of training images that are meaningless to humans can enable a model to generalize to test data . We provide one explanation for why ImageNet-trained models may struggle to generalize to out-of-distribution data . • Geirhos et al . [ 19 ] discover that DNNs trained on distorted images fail to generalize as well as human observers when trained under image distortions . In contrast , overinterpretation reveals a different failure mode of DNNs , whereby models latch onto spurious but statistically valid sets of features in undistorted images . This phenomenon can limit the ability of a DNN to generalize to real-world data even when trained on natural images . • Other work has shown deep image classifiers can make confident predictions on nonsensical patterns [ 20 ] , and the susceptibility of DNNs to adversarial examples or synthetic images has been widely studied [ 5 , 21–23 ] . However , these adversarial examples synthesize artificial images or modify real images with auxiliary information . In contrast , we demonstrate overinterpretation of unmodified subsets of actual training images , indicating the patterns are already present in the original dataset . We further demonstrate that such signals in training data actually generalize to the test distribution and that adversarially robust models also suffer from overinterpretation . • Hooker et al . [ 24 ] found sparse pixel subsets suffice to attain high classification accuracy on popular image classification datasets , but evaluate interpretability methods rather than demonstrate spurious features or discover overinterpretation . • Ghorbani et al . [ 25 ] introduce principles and methods for human-understandable concept-based explanations of ML models . In contrast , overinterpretation differs because the features we identify are semantically meaningless to humans , stem from single images , and are not aggregated into interpretable concepts . The existence of such subsets stemming from unmodified subsets of images suggests degeneracies in the underlying benchmark datasets and failures of modern CNN models to rely on more robust and interpretable signals in training datasets . • Geirhos et al . [ 26 ] discuss the general problem of “ shortcut learning ” but do not recognize that 5 % ( CIFAR-10 ) or 10 % ( ImageNet ) spurious pixel-subsets are statistically valid signals in these datasets , nor characterize pixels that provide sufficient support and lead to overinterpretation . • In natural language processing ( NLP ) , Feng et al . [ 27 ] explored model pathologies using a similar technique , but did not analyze whether the semantically spurious patterns relied on are a statistical property of the dataset . Other work has demonstrated the presence of various spurious statistical shortcuts in major NLP benchmarks , showing this problem is not unique to CV [ 28 ] . 3 Methods . 3.1 Datasets and Models . CIFAR-10 [ 2 ] and ImageNet [ 3 ] have become two of the most popular image classification benchmarks . Most image classifiers are evaluated by the CV community based on their accuracy in one of these benchmarks . We also use the CIFAR-10-C dataset [ 29 ] to evaluate the extent to which our CIFAR-10 models can generalize to out-of-distribution ( OOD ) data . CIFAR-10-C contains variants of CIFAR-10 test images altered by various corruptions ( e.g. , Gaussian noise , motion blur ) . Where computing sufficient input subsets on CIFAR-10-C images , we use a uniform random sample of 2000 images across the entire CIFAR-10-C set . Additional results on CIFAR-10.1 v6 [ 30 ] are presented in Table S4 . We use the ILSVRC2012 ImageNet dataset [ 3 ] . For CIFAR-10 , we explore three common CNN architectures : a deep residual network with depth 20 ( ResNet20 ) [ 31 ] , a v2 deep residual network with depth 18 ( ResNet18 ) [ 32 ] , and VGG16 [ 33 ] . We train these networks using cross-entropy loss optimized via SGD with Nesterov momentum [ 34 ] and employ standard data augmentation strategies [ 32 ] ( Section S2 ) . After training many CIFAR-10 networks individually , we construct four different ensemble classifiers by grouping various networks together . Each ensemble outputs the average prediction over its member networks ( specifically , the arithmetic mean of their logits ) . For each of three architectures , we create a corresponding homogeneous ensemble by individually training five networks of that architecture . Each network has a different random initialization , which suffices to produce substantially different models despite having been trained on the same data [ 35 ] . Our fourth ensemble is heterogeneous , containing all 15 networks ( 5 replicates of each of 3 distinct CNN architectures ) . For ImageNet , we use a pre-trained Inception v3 model [ 36 ] that achieves 22.55 % and 6.44 % top-1 and top-5 error [ 37 ] Additional results from an ImageNet ResNet50 are presented in Section S6 .

Summary: This paper proposes "overinterpretation" which describes the phenomenon that CNNs could achieve high test accuracy while replying on features that lack semantic meaning. To demonstrate overinterpretation on CIFAR-10 and ImageNet, the authors use Batched Gradient SIS to select a small subset of pixels for each image and trained CNNs on the modified images. While humans can not make accurate predictions on those modified images, CNNs can still achieve high test accuracy. Lastly, the authors propose to use ensembling and input dropout to address overinterpretation.