Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet

1 INTRODUCTION . A big obstacle in understanding the decision-making of DNNs is due to the complex dependencies between input and hidden activations : for one , the effect of any part of the input on a hidden activation depends on the state of many other parts of the input . Likewise , the role of a hidden unit on downstream representations depends on the activity of many other units . This dependency makes it extremely difficult to understand how DNNs reach their decisions . To circumvent this problem we here formulate a new DNN architecture that is easier to interpret by design . Our architecture is inspired by bag-of-feature ( BoF ) models which — alongside extensions such as VLAD encoding or Fisher Vectors — have been the most successful approaches to large-scale object recognition before the advent of deep learning ( up to 75 % top-5 on ImageNet ) and classify images based on the counts , but not the spatial relationships , of a set of local image features . This structure makes the decisions of BoF models particularly easy to explain . To be concise , throughout this manuscript the concept of interpretability refers to the way in which evidence from small image patches is integrated to reach an image-level decision . While basic BoF models perform just a simple and transparent spatial aggregate of the patch-wise evidences , DNNs non-linearly integrate information across the whole image . In this paper we show that it is possible to combine the performance and flexibility of DNNs with the interpretability of BoF models , and that the resulting model family ( called BagNets ) is able to reach high accuracy on ImageNet even if limited to fairly small image patches . Given the simplicity of BoF models we imagine many use cases for which it can be desirable to trade a bit of accuracy for better interpretability , just as this is common e.g . for linear function approximation . This includes diagnosing failure cases ( e.g . adversarial examples ) or non-iid . settings ( e.g . domain transfer ) , benchmarking diagnostic tools ( e.g . attribution methods ) or serving as interpretable parts of a computer vision pipeline ( e.g . with a relational network on top of the local features ) . In addition , we demonstrate similarities between the decision-making behaviour of BagNets and popular DNNs in computer vision . These similarities suggest that current network architectures base their decisions on a large number of relatively weak and local statistical regularities and are not sufficiently encouraged - either through their architecture , training procedure or task specification - to learn more holistic features that can better appreciate causal relationships between different parts of the image . 2 NETWORK ARCHITECTURE . We here recount the main elements of a classic bag-of-features model before introducing the simpler DNN-based BagNets in the next paragraph . Bag-of-feature representations can be described by analogy to bag-of-words representations . With bag-of-words , one counts the number of occurrences of words from a vocabulary in a document . This vocabulary contains important words ( but not common ones like `` and '' or `` the '' ) and word clusters ( i.e . semantically similar words like `` gigantic '' and `` enormous '' are subsumed ) . The counts of each word in the vocabulary are assembled as one long term vector . This is called the bag-of-words document representation because all ordering of the words is lost . Likewise , bag-of-feature representations are based on a vocabulary of visual words which represent clusters of local image features . The term vector for an image is then simply the number of occurrences of each visual word in the vocabulary . This term vector is used as an input to a classifier ( e.g . SVM or MLP ) . Many successful image classification models have been based on this pipeline ( Csurka et al. , 2004 ; Jurie & Triggs , 2005 ; Zhang et al. , 2007 ; Lazebnik et al. , 2006 ) , see O ’ Hara & Draper ( 2011 ) for an up-to-date overview . BoF models are easy to interpret if the classifier on top of the term vector is linear . In this case the influence of a given part of the input on the classifier is independent of the rest of the input . Based on this insight we construct a linear DNN-based BoF model as follows ( see Figure 1 ) : first , we infer a 2048 dimensional feature representation from each image patch of size q × q pixels using multiple stacked ResNet blocks and apply a linear classifier to infer the class evidence for each patch ( heatmaps ) . We average the class evidence across all patches to infer the image-level class evidence ( logits ) . This structure differs from other ResNets ( He et al. , 2015 ) only in the replacement of many 3 × 3 by 1 × 1 convolutions , thereby limiting the receptive field size of the topmost convolutional layer to q × q pixels ( see Appendix for details ) . There is no explicit assignment to visual words . This could be added through a sparse projection into a high-dimensional embedding but we did not see benefits for interpretability . We denote the resulting architecture as BagNet-q and test q ∈ [ 9 , 17 , 33 ] . Note that an important ingredient of our model is the linear classifier on top of the local feature representation . The word linear here refers to the combination of a linear spatial aggregation ( a simple average ) and a linear classifier on top of the aggregated features . The fact that the classifier and the spatial aggregation are both linear and thus interchangeable allows us to pinpoint exactly how evidence from local image patches is integrated into one image-level decision . 3 RELATED LITERATURE . BoF models and DNNs There are some model architectures that fuse elements from DNNs and BoF models . Predominantly , DNNs were used to replace the previously hand-tuned feature extraction stage in BoF models , often using intermediate or higher layer features of pretrained DNNs ( Feng et al. , 2017 ; Gong et al. , 2014 ; Ng et al. , 2015 ; Mohedano et al. , 2016 ; Cao et al. , 2017 ; Khan et al. , 2016 ) for tasks such as image retrieval or geographical scene classification . Other work has explored how well insights from DNN training ( e.g . data augmentation ) transfer to the training of BoF and Improved Fisher Vector models ( Chatfield et al. , 2014 ) and how SIFT and CNN feature descriptions perform ( Babenko & Lempitsky , 2015 ) . In contrast , our proposed BoF model architecture is simpler and closer to standard DNNs used for object recognition while still maintaining the interpretability of linear BoF models with local features . Furthermore , to our knowledge this is the first work that explores the relationship between the decision-making of BoF and DNN models . Interpretable DNNs Our work is most closely related to approaches that use DNNs in conjunction with more interpretable elements . Pinheiro & Collobert ( 2014 ) adds explicit labelling of single pixels before the aggregation to an image-level label . The label of each pixel , however , is still inferred from the whole image , making the pixel assignments difficult to interpret . Xiao et al . ( 2015 ) proposed a multi-step approach combining object- , part- and domain-detectors to reach a classification decision . In this process object-relevant patches of variable sizes are extracted . In contrast , our approach is much simpler , reaches higher accuracy and is easier to interpret . Besides pixel-level attention-based mechanisms there are several attempts to make the evidence accumulation more interpretable . Hinton et al . ( 2015 ) introduced soft decision trees that are trained on the predictions of neural networks . While this increases performance of decision trees , the gap to neural networks on data sets like ImageNet is still large . In Li et al . ( 2017 ) an autoencoder architecture is combined with a shallow classifier based on prototype representations . Chen et al . ( 2018 ) uses a similar approach but is based on a convolutional architecture to extract class-specific prototype patches . The interpretation of the prototype-based classification , however , is difficult because only the L2 norm between the prototypes and the extracted latent representations is considered1 . Finally , the class activation maps by Zhou et al . ( 2015 ) share similarities to our approach as they also use a CNN with global average pooling and a linear classifier in order to extract class-specific heatmaps . However , their latent representations are 1Just because two images have a similar latent representation does not mean that they share any similarities in terms of human-interpretable features . extracted from the whole image and it is unclear how the heatmaps in the latent space are related to the pixel space . In our approach the CNN representations are restricted to very small image patches , making it possible to trace exactly how each image patch contributes the final decision . Scattering networks Another related work by Oyallon et al . ( 2017 ) uses a scattering network with small receptive fields ( 14 x 14 pixels ) in conjunction with a two-layer Multilayer Perceptron or a ResNet-10 on top of the scattering network . This approach reduces the overall depth of the model compared to ResNets ( with matched classification accuracy ) but does not increase interpretability ( because of the non-linear classifier on top of the local scattering features ) . A set of superficially similar but unrelated approaches are region proposal models ( Wei et al. , 2016 ; Tang et al. , 2017 ; 2016 ; Arandjelovic et al. , 2015 ) . Such models typically use the whole image to infer smaller image regions with relevant objects . These regions are then used to extract a spatially aligned subset of features from the highest DNN layer ( so information is still integrated far beyond the proposed image patch ) . Our approach does not rely on region proposals and extracts features only from small local regions . 4 RESULTS . In the first two subsections we investigate the classification performance of BagNets for different patch sizes and demonstrate insights we can derive from its interpretable structure . Thereafter we compare the behaviour of BagNets with several widely used high-performance DNNs ( e.g . VGG-16 , ResNet-50 , DenseNet-169 ) and show evidence that their decision-making shares many similarities . 4.1 ACCURACY & RUNTIME OF BAGNETS ON IMAGENET . We train the BagNets directly on ImageNet ( see Appendix for details ) . Surprisingly , patch sizes as small as 17 × 17 pixels suffice to reach AlexNet ( Krizhevsky et al. , 2012 ) performance ( 80.5 % top-5 performance ) while patches sizes 33 × 33 pixels suffice to reach close to 87.6 % . We also compare the runtime of BagNet-q ( q = 33 , 17 , 9 ) in inference mode with images of size 3× 224× 224 and batch size 64 against a vanilla ResNet-50 . Across all receptive field sizes BagNets reach around 155 images/s for BagNets compared to 570 images/s for ResNet-50 . The difference in runtime can be attributed to the reduced amount of downsampling in BagNets compared to ResNet-50 . 4.2 EXPLAINING DECISIONS . For each q × q patch the model infers evidence for each ImageNet classes , thus yielding a highresolution and very precise heatmap that shows which parts of the image contributes most to certain decisions . We display these heatmaps for the predicted class for ten randomly chosen test images in Figure 2 . Clearly , most evidence lies around the shapes of objects ( e.g . the crip or the paddles ) or certain predictive image features like the glowing borders of the pumpkin . Also , for animals eyes or legs are important . It ’ s also notable that background features ( like the forest in the deer image ) are pretty much ignored by the BagNets . Next we pick a class and run the BagNets across all validation images to find patches with the most class evidence . Some of these patches are taken from images of that class ( i.e . they carry `` correct '' evidence ) while other patches are from images of another class ( i.e . these patches can lead to misclassifications ) . In Figure 3 we show the top-7 patches from both correct and incorrect images for several classes ( rows ) and different BagNets ( columns ) . This visualisation yields many insights : for example , book jackets are identified mainly by the text on the cover , leading to confusion with other text on t-shirts or websites . Similarly , keys of a typewriter are often interpreted as evidence for handheld computers . The tench class , a large fish species , is often identified by fingers on front of a greenish background . Closer inspection revealed that tench images typically feature the fish hold up like a trophy , thus making the hand and fingers holding it a very predictive image feature . Flamingos are detected by their beaks , which makes them easy to confuse with other birds like storks , while grooms are primarily identified by the transition from suit to neck , an image feature present in many other classes . In Figure 4 we analyse images misclassified by both BagNet-33 and VGG-16 . In the first example the ground-truth class `` cleaver '' was confused with `` granny smith '' because of the green cucumber at the top of the image . Looking at the three most predictive patches plotted alongside each heatmap , which show the apple-like edges of the green cucumber pieces , this choice looks comprehensible . Similarly , the local patches in the `` thimble '' image resemble a gas mask if viewed in isolation . The letters in the `` miniskirt '' image are very salient , thus leading to the `` book jacket '' prediction while in the last image the green blanket features a glucamole-like texture .

Summary: This paper suggests a novel and compact neural network architecture which uses the information within bag-of-words features. The proposed algorithm only uses the patch information independently and performs majority voting using independently classified patches. The proposed method provides the state-of-the-art prediction accuracy unexpectedly, and several additional experiments show the state-of-the-art neural networks mainly learn without association between information in different patches.