A Sample Based Method for Understanding The Decisions of Neural Networks Semantically
Authors: Ohi Dibua
Jonathan Mbuya
Mackenzie Austin
Kushal Kafle
Venue: Submitted to ICLR 2023 (2023)
Keywords: Machine Learning Interpretability
Bias
ImageNet
AlexNet
ResNet
VGG-16
Inception
CNNs
Bag of Words

Abstract: Interpretability in deep learning is one of the largest obstacles to its more widespread adoption in critical applications. A variety of methods have been introduced to understand and explain decisions made by Deep Models. A class of these methods highlights which features are most influential to model predictions. These methods have some key weaknesses. First, most of these methods are applicable only to the atomic elements that make up raw inputs to the model (e.g., pixels or words). Second, these methods generally don't distinguish between the importance of features individually versus due to interactions with other features. As a result, it is difficult to explore high level questions about how models use features. We tackle these issues by proposing Sample-Based Semantic Analysis (SBSA). We use Sobol sensitivity analysis as our sample-based method. Sobol-SBSA allows us to quantify the importance of semantic combinations of raw inputs and highlight the extent to which these features are important individually as opposed to due to interactions with other features. We demonstrate the ability of Sobol-SBSA to answer a richer class of questions about the behavior of Deep Learning models by exploring how CNN models from AlexNet to DenseNet use regions when classifying images. We present two key findings. 1) The architectural improvements from AlexNet to DenseNet manifested themselves in CNN models utilizing greater levels of region interactions for predictions. 2) Adversarially robust CNNs resist exploiting spurious correlations in ImageNet data by forcing these architectures to rely less on region-to-region interaction. Our proposed method is generalizable to a wide variety of network and input types and can help provide greater clarity about model decisions.

Under review as a conference paper at ICLR 2023
A SAMPLE-BASED METHOD FOR SEMANTIC UNDER- STANDING OF NEURAL NETWORK DECISIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Interpretability in deep learning is one of the largest obstacles to its more widespread adoption in critical applications. A variety of methods have been in- troduced to understand and explain decisions made by Deep Models. A class of these methods highlights which features are most influential to model predictions. These methods have some key weaknesses. First, most of these methods are ap- plicable only to the atomic elements that make up raw inputs to the model (c.g. pixels words). Second, these methods generally do not distinguish between the importance of features individually and their importance due to interactions with other features. As result, it is difficult to explore high-level questions about how models use features during decision-making We tackle these issues by proposing Sample-Based Semantic Analysis (SBSA). We use Sobol variance decomposition as our sample- sample-based method which allows us to quantify the importance of se- mantic combinations of raw inputs and highlight the extent to which these features are important individually as opposed to due to interactions with other features. We demonstrate the ability of Sobol-SBSA about the behavior of Deep Learning models by exploring how CNN models from AlexNet to DenseNet use regions when classifying images. We present three key findings. 1) The architectural improvements from AlexNet to DenseNet mani- fested themselves in CNN models utilizing greater levels of region interactions for predictions. 2) These same architectural improvements increased the impor- tance that CNN models placed on the background of images 3) Adversarially ro- bust CNNs reduce the reliance of modern CNNs on both interactions and image background. Our proposed method is generalizable to a wide variety of network and input types and can help provide greater clarity about model decisions.
answer richer class of questions
INTRODUCTION
Deep learning models are becoming endemic in various applications. As models are increasingly used for critical applications in medicine such as detecting lung nodules (Schultheiss et al. [022) or autonomous driving (Li et al. 202T), it is important to either create interpretable models or make opaque models human interpretable. This paper focuses on the latter. Existing methods developed over the last decade for doing this can be broken down into model agnostic vs model dependent. Model agnostic methods, such as Shapley values (Kononenko et al. 2013) and Integrated Gradients (Sundararajan al., 2017) weigh the importance of input features without relying on the structure of the model. In contrast, methods such as GradCam (Selvaraju et al. 2017) and GradCan++ (Chattopadhay et al. 2018) are heavily dependent on model architecture.
While these methods yield valuable information about models, they share common gaps. First, they do not distinguish between the features in input space that are individually important and features that are important because of their interaction with other features. Second, the above methods are generally applied to inputs at their most granular level (pixels, words, etc..). The combination of these gaps limits the conclusions that Machine Learning practitioners make about the behavior of models as a whole.
We address these limitations in two key ways. First, introduce two-part framework called Sample-Base Based Semantic Analysis (SBSA). The first part of the framework is a function that gen- erates semantic representations of inputs and associates these semantic representations with real

---

Under review as a conference paper at ICLR 2023
numbers. The second part of the framework is black-box sample-based sensitivity method. In this case, the Sobol method which reports the importance of individual features and their interactions. Second, we demonstrate the ability of Sobol-SBSA to answer richer set of questions than standard interpretabiligg methods by applying it to CNN models in the context of ImageNet.
The key results and contributions of this paper are as follows:
1. We present general-purpont purposs framework for using sample- based sensitivity methods to analyze the importance of semantic representations of inputs and test it using a variety of black-box methods.
2. We demonstrate that the Sobol method outperforms other popular black box methods, Inte- grated Gradients, Shapley (Kernel Shap), and LIME, for selecting both the most and least i important regions to CNN predictions.
3. We show, through direct measurement, that the main impacts of the evolution of CNN architectures were increasing the extent to which they used region interactions and by which they relied on background information in images. Similarly, we show that adversarially robust versions of CNNs reduce both of these effects for modern CNNs. To our knowledge, Sobol-SBSA the first pipeline to facilitate the direct measurement of such trends, and to do within single pipeline.
2 METHODOLOGY
In this section, we describe the two components of SBSA and specify how we use it to analyze the importance of image regions in ImageNet. In particular, we describe how we associate image regions to quantities that can be analyzed with sampling-hased method, and the specifics of Sobol as sampling based sensitivity method.
2.1 SAMPLE- BASED SEMANTIC ANALYSIS (SBSA)
Let us define the following variables. x E Rd is an input to a model, f x + y E RD is model that takes
argument and produces y, _(((] C Rd is sample of æt, and N E Z is prescribed integer that helps to determine the number of _ifll samples generated. Most sample-based sensitivity methods operate by generating a number of samples that is some function of N and d. The model is then evaluated on these samples and the resulting model outputs are used by Sensitivity analysis methods, such as Sobol, to determine the importance of components of to the model output, y.
One thing that immediately becomes clear is that for deep learning applications with high- dimensional inputs, such as images, videos, and long documents, applying this process naively is prohibitively expensive. This issue be greatly minimized if one turns to semantic represen- tations of inputs instead. In this paper, semantic representation of an input is defined as follows. A semantic representation of input, x, is some combination of the input which yields a human recognizable higher order feature, such as the colors in an im- age, image regions, grammatical parts of sentences. We define this semantic representation as . ) Sg E R" where m < d. Recalling that most sample-based sensitivity methods operate real numbers, we define three mapping objects.
components of that
sample
G: (SI.... S., G-1; {S Si} +E ZER S₁ E RT 1 <, m m<d
(1)
H : (S,...., Si). + m.. S E R"
rh E R.
(2)
R: (I)..))....... (r, S)) (S S7,
E
(3)
G maps the raw input, x, to semantic representations, St, H associates the semantic representation to some lower dimension vector real numbers, E R', and R creates new semantic representations based on тк and St. G is invertible. SBSA generates samples of r, [+!1),.. ,..., ,.ml]. From these samples, R is used to generate samples of the original semantic representations, R(F() Sx) = -5(0
2

---

Under review as a conference paper at ICLR 2023
and G-11 uses these samples to create samples of the raw inputs, 211.
[R(F)) R(F)") ..) (5 5/-1
(4)
G 5.") = =
(5)
The model is then evaluated on the samples of the raw input, _00), Since the sampling done in r, the sensitivity analysis reports the importance of components input, S& corresponding to the components of r, Tg. G. Hr, and I can be any functions that derive Sk from æ, associates a real number, rk. with Sj, and produces a semantic representation, St. as a func- tion of rk and Sy. However, we recommend two properties to maximize our approach. Property (Sensitivity) Any change in r should result in change x. Sample- basee sensitivity methods observe how changing different input components impact model outputs in order to determine importance. Property insures that all samples in contribute information to SBSA. Property 2 (Approximate Reconstruction): When using and produced from the original input, R and G-1 should closely approximate Sample-baset sensitivity analysis methods construct samples that uniformly dis- tributed between and I, and scales these samples to bounds determined by the original r (Herman
the semantic representations of the
Usher 2017). Property insures that produced samples æ, zii, will increase and decrease different semantic components of x with approximately equal probability.
In the following section we describe the use of SBSA via image regions.
2.2 SAMPLE- -BSSED SEMANTIC ANALYSIS (SBSA) APPLIED TO IMAGENET AND REGIONS
In this section, we discuss how we apply our pipeline to the scenario in which the semantic features of interest are regions in an image, and the campl--based sensitivity method is Sobol (Sobol-SBSA). We first describe our mapping functions for regions, then detail the Sobol method for sensitivity analysis and the relevant measures that it produces.
Given an image x Rd where = W x L, our input to semantic features function, G, extracts I regions and normalizes them by the sum of pixels in those regions. The mapping, H, associates each region, Sg, with the sum of pixels in that region, Tk- These values are used to construct the vector r E R'.
G: Rd + (Si..... Si}. S = E(rn)es Zip
F(L.)ES
(6)
H: (S (S,..... .S.,...., Si} + tri,..., rt). rj Σ
Ftp
(1,))
The function, R. multiplies each semantic feature, S₁, with the associated value, T and finally G-1 creates x by stitching together the regions to reform x. Mathematically, this is as follows.
R: (r...,). G (S) -
(S ST. S Sk
(8)
The above mapping, while simple, satisfies the properties of sensitivity and approximate reconstruc- tion. As result the samples r, -ftl, produce image samples, _il that amplify and mask regions of the image with relatively equal probability and uniformly. The Saltelli method, (Saltelli et al. 2010),
which chooses optimal points in [o. for the Sobol method, are used to construct these samples and the model outputs of these image samples are fed to the Sobol method.
We will briefly give an overview of the Sobol method. For more details about the implementation of Sobol, and the associated sampling method, see Saltelli et al. (2010) and Herman & Usher (2017).
The Sobol method is a srtiance-based sensitivity method. Given a model, f (I1,..., 2.2) E R y E R the variance based first order effect of component of the input, x, is:
V Va, (Fx
(10)
3

---

Under review as a conference paper at ICLR 2023
STew
Sctell
&
Imags
Semautic
Veesoe
Vistoe
Vecior Sample
Vecior Sample
Scmauiic Sample
Image Sample
Sumple Ousgui Schol Misleed
Figure I: Sobol-SBSA pipeline for regions and segments (ImageNet-S). 1) Image + Semantic - Vector extracts semantic features from the image and maps these onto real numbers. 2) Vector + Vector Sample produces samples from the vector. 3) Vector Sample - Semantic Sample + Image Sample produces samples of the image that mask and amplify semantic features based on the vector samples. 4) Sample Output -> Sobol Method sends outputs from the model evaluated on the image samples to the Sobol method for analysis.
where is a component the model output, 2; is a component of the input, H, X-v are samples of the input where all components of the input except for i varied, and Va; and Ex the variance and expectation over vi and X. respectively. The Sobol first-order sensitivity is then the ratio of V; the variance the model output, 3j-
S:
(11)
V
S: quantifies the individual importance of the 7th component of the input, x, to the jth component of the output, y. This equation essentially states that Sobol first order index, S, is the fraction of the variance of the output, V/ that is accounted for by Tg. What makes Sobol unique is that it simultaneously calculates S, and Sr,, the total effect index. The total effect index, Sr, is the importance of feature i due to both the feature independently and every higher level interaction of this feature with other features.
ST. = 1 Vxni (Ex, (U/ Xw))) S +
Σ A;
Sih +
Σ
Sik +...
(12)
V(y;)
kij
Dividing both sides of equation by ST: yields
S
Sik
Sike
=
Σ
(13)
ST
STI
ST:
The first term in equation 13 reports the extent to which a feature, 2;, is important to the model output due to the feature by itself. The larger the term, the greater the importance of the region by itself, contrarily, the smaller the term larger the importance other features. We will refer to this as the Primary Index Ratio (PIR) for the rest of the paper. We note that the number of samples used for the Sobol method N + 2) (Saltelli et al. 2010). For all of the experiments that follow, N 50 and d I, the number of semantic features.
the interaction of the feature with
2.2. CHOICE OF SEMANTIC FEATURES
In order to demonstrate the ability of our method to address high level questions about CNNs at semantic level, we apply our pipelines to three types of regions. 1) Equally sized image patches 2) machine annotated segmentations of the ImageNet training set from Salient ImageNet Singla &
Feizi (202T) and 3) human annotated segmentations from the ImageNet-S 919 validation set (Gao al. 2021 Salient ImageNet segments images into core regions (regions that should be important for predictions) and spurious ones. ImageNet-S a dataset that was created for use in judging image segmentation task. A key difference between ImageNet-S and the other types regions
4

---

Under review as a conference paper at ICLR 2023
that ImageNer-S strictly respects boundaries between objects and background in an image. Region patches and Salient ImageNet do not. Figure [ shows our pipeline when applied to evenly size regions of image and to ImageNet-S 919 segments.
EXPERIMENTS
3.1
VALIDATING OUR PIPELINE
We validate our pipeline in two ways. First, we demonstrate it's ability to accurately rank the impor- tance regions of different sizes. Second, we demonstrate that Sobol-SBSA accurately reproduces two trends in how CNNs use regions that were determined indirectly by two mutually exclusive pa- pers.
Brendel & Bethge 2019) showed that more modern CNNs use higher levels of interactions between equally sized region patches than older models.
Singla et al. (2022) showing that robust versions of CNN models utilize "spurious" areas of image, determined by machine annotation, less than normal versions of these CNNs.
Having established trust in Sobol-SBSA, CNNs use regions, without needing to use a different pipeline. First, we demonstrate the impact of semantic representation on Brendel & Bethge (2019)'s 's results by measuring the interaction between segmented objects and their backgrounds (lmageNet-S). We show that the trend of more modern CNNs utilizing interactions more than older models is only clear when viewed with respect to regions that do not strictly respect object boundaries. Second, we demonstrate that, regardless of whether or not the segmentation respects boundaries, result of the development CNN architectures was greater exploitation of background information when making decisions. Finally, we demonstrate that, robust models reduce the extent to which models use the interaction if regions which do not respect boundaries. Too the authors' knowledge, this is the first time that all of these results have been demonstrated through direct measurement and a single pipeline. We note that ImageNet models were pre-trained Pytorch models from ImageNet1K VV.
demonstrate how it can be used to further explore how
3.2 VALIDATING SOBOI-SBBA FOR IMPORTANCE RANKING AND SEMANTIC UNDERSTANDING
Our SBSA model can be used with a variety of black box models. Thus, we compare the following two methods against Sobol-SBSA.
Shapley (Shap-SBSA) (Kononenko et al., 2013): Used with SBSA in a similar manner Sobol-SBSA.
LIME (LIME-SBSA) (Ribeiro et al., 2016): Used with SBSA in similar manner as Sobol-SBSA.
We also compare against the following:
Integrated Gradients (Sundararajan et al. 2017): Cannot be used with SBSA since requires gradients with respect to pixels. Instead, aggregate pixel importance in regions and rank region importance based on this aggregation.
Random: A control baseline where regions are randomly selected.
Ideal: The ideal result based on our metric. This is not experimental.
We use two standard metrics for comparing interpretability methods. First, we measure the change in the ground truth label score of model when regions, as specified by the sensitivity method, are masked. We mask the top and bottom 20% of regions in an image. Second, we measure sensitivity-n correlation. Sensitivity-n correlation is quantity proposed by Ancona al. (2017) for comparing the effectiveness of attribution methods when determining which features are impor- tant a model. Quantitatively, it a measurement the Pearson correlation between Σi=1 R and (.())) (1( Ei=1 RC is the sum of the attributions associated with the input features that

---

Under review as a conference paper at ICLR 2023
Number of Regions:
Number of Reglons: 64
Number Reglons: 256
Sensitive Regions Change in Score
0.
0.
0.0
-0.1
-01
-01
(Ours)
-0.2
Sobol 51 (Durs)
-0.3
Y
Intag.
Lime
-0.4
-044
Shapley
-0.5
-03
-03
Random
-0.6
-05-
-0.6
kleal
0.0
-02
-0.4
-0.6
-0.2 -0.4
-0.5
0.0
-0.2
-0.4
-0.6
Insensitive Regions Change in Score
Figure 2: Plots the change in the ground truth score the ResNet50 when the top 20% of regions masked the bottom 20% of regions. A steep slope is indicative of since it means that the change ground truth scores are significant when the top 20% of images masked, but when the bottom 20% of images masked. Our measure, Sr, is closest the ideal.
effective method
Number Regions:
Regions:
Number Regions:
Ser Sersitivity- Correlation
(Ours)
0.2
0.5
Sobol 51 (Ours)
Integ.
0.6
0.6
0.5
Lime
Shapley
Random
kical
0.-
0.0
0.2
0.4
0.0
0.5
1.0
0.0
0.2
0.4
0.6
088
1.0
0.0
0.2
0.4
0.6
0.0
1.0
Percent of Regions Masked
Figure 3: Plots of the sensitivity-n correlation for different sensitivity methods as function of the percentage of regions masked. The sensitivity methods with values closer to one are more effective. Our measure, ST, is closest to the ideal.
masked and Sx(x) - (z)) is the difference in the score that the model produces when input features are masked versus when none are. For each value of i, random features are selected to be masked 100 times and the correlation is averaged over the examined data. A value closer to one means a more effective method.
For all methods, ageNet validation images. To explore the importance of image size, we performed our experiments for 4 4, 8 8, and 16 16 regions. The number of samples selected function of the number of regions, detailed in section 2.2 For 4 x 4, 8.8 8, and 16 x 16 regions, d = 16, d 64. and 256 respectively.
evaluate the importance of regions to ResNet50 on 1000 randomly selected Im-
Figures 2 and 3 show the results of the masking and sensitivity-n correlations respectively for all evenly sized region patches. Figure plots the change in the model score when the most important regions are masked versus when the least. A steeper curve is more ideal since it means that the sensi- tivity method significantly decreases the model score when masking the most important regions, but has little impact when masking the least important regions. The Sobol-SBSA is the most effective by both the masking and sensitivity-n measures. The key takeaways from the plot are 1) ST: performs the best overall in picking the most and least important regions in image, well more prop- erly ranking the regions in between (as measured by sensitivity-ni, and 2) Sr: is the most robust to changes in region size, followed closely by Integrated Gradients. The gap performance between Sr. and S, shows the importance accounting for region interaction when ranking importance.
Sobol-SBSA quantify how the importance of interactions between evenly sized regions to CNN model outputs evolved over time, and test whether or not this matches the trend found by Brendel & Bethge (2019). Figure 4a plots the average PIR R. for 10000 validation images
We

---

Under review as a conference paper at ICLR 2023
when 4 x 4, 8 8, and 14 x 14 regions. A lower PIR means greater region interaction importance. We see that that for all of these regions, interaction decreases from AlexNet and VGGI6 to the more modern Inception ResNet50, and DenseNet161 architectures, Brendel & Bethge| 2019)'. Because Pytorch's pre-trained InceptionV3 takes as inputs images of size 299 299 rather than 224 224, the number of regions used for Inception V3 are 5 x 5, 11 11, and 18 x 18. Finally, we apply Sobol-SBSA to 10000 Salient ImageNet images whose regions have been labeled as "core" regions that should be important to model predictions, and "spurious" regions that should not (Singla & Feizi, 2021). Figure 4 plots the average difference between the total index score of core and spurious regions for the normal and udversarially robust versions of VGG16 BN,, ResNet50, and 16nseNet161. We use pre-trained robust weights from [Salman et al. (0200) where the /2 threat radius was 3. For Salient ImageNet, the difference in core and spurious importance is greater for robust models than their normal counterparts. This compliments [Singla ct al. (2022)ss ss findings that robust CNN models relied less on spurious areas images than their normal counterparts.
identical result to
Region Interaction Trends: 10000
Segmentation Background Importance: 10000
10
10-
Salient ImageNet
3x8
Normal: Sallent ImageNet
Normal: ImageNeess
0.-
IregeNet-5
14x14 Regions
Hobust: Selien: ImageNet
Kobust: ImageWet-S
4xd Raglans
08
8
02
0.3
02-
Andiec
VIG22
VGD226 DU
Incepiion/3
Rax Uaeb
Demuelat:21
VCCI6 BN
Nes
Dansehet16t
Model
Model
(a) The plots of the average mean of PIR for 10000 im-
(b) The plots of the average difference in importance, Sr, for segmented objects ImageNet-S and Salient ImageNet when applied normal and robust models.
ages for series CNN models when applicd to im-
ages that are partitioned based on segmentations from
ImageNet-S and Salient ImageNet, as well as when ap-
10000 images from each dataset were used.
plied to the top 20% of region patches.
Figure 4: PIR trends and the difference in importance between segmented objects and background for CNN models and their robust counterrparts.
3.3 BEYOND VALIDATION: THE IMPACT OF SEGMENTATION ON REGION IMPORTANCE AND INTERACTION TRENDS
In the previous subsection we validated Sobol-SBSA's ability to 1) correctly rank the importance of regions regardless of size 2) accurately capture trends in how CNNs use regions that were determined by Singlla et al.] p2022) and Brennel & Bethge (2019). We now explore the impact of segmentation type the trends in region interaction and differences between robust and normal models. We also measure how the use background information evolved with models.
Figure 4a plots the PIR for Sobol-SBSA applied to foreground and background regions deter- mined by Salient ImageNet and ImageNet-S. For foreground and background areas of an image as determined by Salient ImageNet, interactions became more important with more modern CNNs. For TeageNet-S regions, however, these interactions only increased for DenseNet161 and Incep- V3 Recalling that ImageNet-S regions are segments that strictly respect object boundaries, we conclude that while modern CNNs use greater interactions between regions that do not respect object boundaries, this not consistently the case for those that do. Inception V3 was not used with Salient ImageNet since the masks provided were 224 x 224 which is incompatible with the expected 299 x 299 input to Inception V3. Figures 44b shows the average difference in the total index score, ST, between foreground and background areas of images determined by ImageNet-S and Salient ImageNet. This is done for normal and robust models. We see that the trend robust models reducing the extent to which CNN architectures use background information holds, regardless of whether or not the foreground strictly respects object boundaries. Examples of this are shown in Figure K Figure 5al plots the average P/ R. between foreground and background objects deter-
7

---

Under review as a conference paper at ICLR 2023
Segmentation Background
1.3
2.7
Segmentation Background Importance: 10000
Sallent
ImageNet-5 919
2.8
ST: Salient mageliet
ST: ImageNittS 919
35
8.5
6.4
34-
Normal:
ImageNet
Imagefet-SS
22-
Robust:: Salient imagelter:
Normal: dell Regions
64-
Normal:
mage
Robust: ax8 Reglons
VGG10,3N
Resbetb3
Densellet10]
Mestel
Wintb 58
Dersehettibl
Model
Model
(a) The plots of the average mean of PIft It when Sobol-
(b) The plots of the average difference in importance, ST and S; for segmented objects in ImageNet-S and Salient ImageNet as function of models. 10000 im- ages from each dataset were used.
SBSA is applied to segmented and background objects
SalientImageNet and ImageNet-S for normal bust models. 10000 images were used in their respec- tive datasets.
Figure 5: PIR trends and the difference in importance between segmented objects and background for CNN models
mined by ImageNet-S and Salient ImageNet for the robust and normal versions of CNNs. It is seen that, for areas defined by Salient ImageNet, robust models decreased the reliance of architectures on interactions, but that for objects and background determined ImageNet-S this was only clearly seen for DenseNet161. The key take away here is that robust models generally decrease model's reliance on interactions between regions that do not respect boundaries, but don't necessarily do so for those that do.
Finally, Figure 5b plots the average difference in importance between foreground and background objects when Sobol-SBSA is applied ImageNet-S and Salient ImageNet. These results plotted for AlexNet to DenseNet161. development was increase the extent to which these models use background information.
is seen that one of the effects of architectural changes in CNN
3.3. THE LIMIT OF SOBOL-SBSA
While the Sobol-SBSA is a powerful method, it has a similar weakness to other sample-based black box methods. assumes that the input features that it is analyzing are uncorrelated. When this is not the case, the ability of Sobol to decompose importance into independent subsets of the feature space weakened (Li et al. 2010). This manifests itself in the Sobol Indices, which are supposed to satisfy the property that PIR < having at least some input features for which PIR > 1.
The assumption that regions are uncorrelated breaks down as regions become smaller in size. We observe this in two ways. First, we examine figure 4a and note that PTR increases as the region sizes decrease, This is intuitively an incorrect result since interactive effects should be greater for smaller regions, demonstrated by Brendel & Bethge (2019). To confirm that this result is due to a breakdown in Sobol assumptions we calculate the average percent of regions for which PIR > across all models for given region size. For 4 x 4, 8 8, and 14 14 regions the percentages 22%, 33%, and 39% respectively. Figure shows an example of how Sobol-SBSA can breakdown with smaller region sizes. A key direction for future work is implement version of Sobol-SBSA that accounts for correlations in the data.
RELATED WORK
In recent years, interest has grown in interpretability methods that can quantify the importance of feature interactions. Janizek et al. (2000) introduced extension to Integrated Gradients, called Integrated Hessians, that quantified the importance of pairwise interactions features, while Zhang et al. 2020) built this work by presenting similar quantification through Shapley Interactions. While both methods are powerful, they require the separate calculation individual feature inter- actions, well pairwise comparisons. Fel et al. (202)) used the Sobol method, but focused
8

---

Under review as a conference paper at ICLR 2023
ST
S
ST.
S
ResNet50
Robust ResNet50
Class: Goldfish
Class: Ostritch
Figure 6: A sample result of the Sobol-SBSA when applied to segmentations determined by the ImageNet-S dataset for the normal and robust versions ResNet50, Robust ResNet50 focuses only on the segmented objects, while ResNet50 also uses the background.
importance, rather than interactions. None of the above works addressed understanding how mod- els used semantic representations, facilitated quantitatively answering high level questions about models.
5 DISCUSSION AND CONCLUSION
We proposed Sobol-SBSS, general method for understanding how Deep Learning Models use semantic features when making decisions. We demonstrated the ability of this method to answer one pipeline rich set of questions about model behavior by using it to study how CNN models use areas of images during classification. We found 1) that the primary impact of the evolution of CNN models was to make greater use of region interactions and to increase the importance image background to model predictions and 2) that adversarially robust CNN models are less susceptible to spurious correlations in the data because they force CNN architectures to rely less on region interactions and on image backgrounds.
The Sobol-SBSA method has a variety of potential applications beyond the image/region-based analysis that presented here. Many different types of input partitioning and input modalities can be analyzed using the SBSA method. For images, the partitioning into image regions can be done via external metadata, such as object detection results. For natural language processing (NLP), the use of SBSA is even more straightforward since words/tokens provide a natural way to partition the input sequence. Hence, our model can be utilized understand how parts of speech are utilized for different iterations of various NLP models, including translation, classification, and generative text models. Beyond unimodal content, SBSA can also be easily extended multimodal setups. For example. Sobol-SBSA can be used to measure the strength of bias in Image Captioning systems by quantifying the extent to which parts of an image correspond caption. In addition to these applications, there are multiple avenues for future work. These include exploring the combination of Sobol-SBSA with automatic feature detectors, implementing version of Sobol-SBSA that accounts for input correlations, and exploring whether PIR can be used proxy for the robustness of non-CNN Deep Learning models.
which parts of the generated
Through SBSA and Sobol-SBSA we have proposed strong foundation for obtaining a richer un- derstanding of how models use semantic representations inputs, regardless of whether these rep- resentations generated automatically or by end users. We hope that it can be used to both provide clarity about the mechanisms by which Deep Learning makes decisions, and to influence how we further develop these models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marco Ancona, Enea Ceolini, A. Cengiz Öztireli, and Markus H. Gross. A unified view of gradient- based attribution methods for deep neural networks. CoRR, abs/1711.06104, 2017. htth:tp:
wtxxiv.org/abb.... 1711. 11111..000 06104
Wieland Brendel and Matthias Bethge. Approximating cnns with Sag-of-local-featuue models works surprisingly well on imagenet. volume 7, 2019.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-
cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pp. 839-847. IEEE, 2018.
Thomas Fel, Rémi Cadène, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look the variance! efficient blx explanations with sobel-based sensitivity analy-
sis. CoRR, abs/211 1.04138, 2021. URL
Shanghua Gao, Zhong-Yo Yu Li, Ming-Hsuan Yang. Ming-Ming Cheng, Junwei Han, and Philip H. S.
Torr. Large-scale unsupervised semantic segmentation. CoRR, abs/2106.03149, 2021. URL
031449
Jon Herman and Will Usher. Salib: An open-source python library for sensitivity analysis. Journal of Open Source Software, 2017.
00097
Joseph D. Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. CoRR, abs/2002.04138, 2020. URL httpss://axxi...rrrr
abs/ abs/2002. 04138
Kononenko, E. Strumbelj, Z. Bosnic, Darko Pevec, M. Kukar, and M. Robnik-Sikonjy Explana- tion and reliability of individual predictions. Informatica, 37, 2013.
Genyuan Li, Herschel Rabitz, Paul E. Yelvington, Oluwayemisi O Oluwole, Fred Bacon, Charles E. Kolb, and Jacqueline Schoendorf. Global sensitivity analysis for systems with independent and/or correlated inputs. The Journal of Physical Chemisiry A, 114, 2010. URL https: / 10. 16.1021
Ying Li, Lingfei Ma. Zilong Zhong, Fei Liu, Michael A. Chapman, Dongpu Cao, and Jonathan Li. Deep learning for lidar point clouds autonomous driving: review. IEEE Transactions on Neural Networks and Learning Systems, 32, 2021.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. CoRR, abs/1602.04933 04938, 2016. URL httt::t/taxii..rrrrrrt////
abs abs/1602 160.. 04 04938
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? ArXiv preprint urXiv:2007.0888) 2020.
Andrea Saltelli, Paola Annoni, Ivano Azzini, Francesca Campolongo, Marco Ratto, and Stefano Tarantola. Variance based sensitivity analysis of model output. design and estimator for the total sensitivity index. Computer Physics Communications, 181, 2010. URL httpt::/d//:......
10.1016/3. wpw.:.p.............m
2009.000000 09. 018
Manuel Schultheiss, Sebastian Schober A., Marie Lodde, Jannis Bodden, Juliane Aichele, Christina Müller-Leisse, Bernhard Renger, Franz Pfeiffer, and Daniela Pfeiffer. A robust convolutional neural network for lung nodule detection in the presence of foreign bodies. Scientific Reports, 32, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Sahil Singla and Soheil Feizi. Causal imagenet: How to discover spurious features in deep learning? CoRR, abs/21 10.04301, 2021. htl.ttts: http://///..aaaaoooo//////////// 04301
10

---

Under review as a conference paper at ICLR 2023
Sahil Singla, Mazda Moayeri, and Soheil Feizi. Core risk minimization using salient imagenet, 2022. URL http:/awww...aaa//aaaaaaaaaaaaaa 2203.15563.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Inter- national Conference on Machine Learning, 34, 2017.
Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate interactions in dnns. CoRR, ab/2010.050455 2020. URLhttps://arat http://wwwrgg//////////o/a
org abs/ a010/010. 2010.
05045

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1
10K VS 50K EXPERIMENTS
Region Interaction Trends: 10000
Region Interaction Trends: 50000
--- Number of Regions: 16
Number of Regions: 16
--- Number of Regions: 61
Number of Regions: 64
0.0
P1R
Akoxlet
VG616
VG616 Inceptiont3 DerssNtt0o
AlcaNett
VS616
VSG16 BN Inceptionvs ResNett0 DorseNat161 Model
Modul
Figure 7: Average PIR for the top 20% regions when calculated for 10000 and 50000 ImageNet validation images. This is done for 4 x 4 and 8 x 8 regions. The results are identical.
We calculated the average PIR for the top 20% regions for 10000 and 50000 validation images in ImageNet using 4 x 4 and 8 x 8 regions respectively. We saw that the results were identical so, for the smaller regions generated when splitring an image into 14 regions, we calculated the average PIR for 10000 images save computational cost.
A.2 IMPACT OF REGION SIZE
ST.
S;
ST.
S
ST:
S,
ResNet50
Robust ResNet50
x
8 8
Figure 8: Images of S; and Sr; for Ostrich class example for different grid sizes. We see that x and 8 regions are consistent, but that this is not the case for x regions.
Figure shows an example of Sobol-SBSA when applied to Ostrich target class for different region sizes. We see that all of the examples are consistent except for the Robust ReseNet50 14 14 example. One of the weaknesses of Sobol is that the results can be corrupted when the input features are correlated. As regions get smaller, this exactly what occurs. Although, overall, Sobol-SBSA was still able correctly identify the most and least important regions the 14 14 scale, figure 2, more work must be done to address this weakness so that the Sobol results can be compared accurately across different sizes of regions or, more generally, features that have different types of correlations. Future work will involve exploring the impact of sampling size and variations of Sobol that account of input correlation on addressing this issue.
12


Reviews:
[{"review_id": "pKSkFqd2Yj6", "reviewer": "Reviewer_1XwS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed an algorithm for interpreting deep models, which is an extension of Fel et al., 2021. The main tool it relies on is called Sobol first-order sensitivity. The experiments are demonstrated on different CNN variants, including VGG, ResNet and DenseNet. ", "summary_of_the_review": "The main concern is the novelty. There are also some components which seems not right to me, and I would like to see authors' justification in the rebuttal. ", "strength_and_weaknesses": "- The main criticism is on the novelty. Most of the components have been used in Fel et al., 2021.   Until equation (11), most of the contents are covered by Fel et al 2021, and the new ingredients from this paper seems to be equation (12) only.  I would encourage authors to spend more space discussing the new components proposed by this paper instead of repeating what people have done before. Also, It would also be nice to provide literatures about Sobol methods,  as Fel et al., 2021 is not the one proposing this metric but proposing to reuse this tool. Some important literatures include\n    * I. Sobol’, “Sensitivity estimates for non linear mathematical models,” Mathematical Modelling and Computational Experiments, vol. 1, pp.\n407–414, 1993.\n    * B. Li and C. Chen, “First-order sensitivity analysis for hidden neuron selection in layer-wise training of networks,” Neural Processing Letters, vol. 48, pp. 1105–1121, 2018.\n\n- It is not clear to me how we construct the inverse function of H and G.  The inputs of H and G are with higher dimensions than their outputs, which the inverse functions \"usually\" don't exist. Also, the function includes some pooling operations (sum), it is counter-intuitive to be able to construct an exact inverse function. More descriptions would be helpful for readers to understand. \n\n- The proposed methods are based on comparing the variances of different components, such as equation (10) and equation (11). However, there are already a lot of literatures discussing the variance of features for interpretability. The oldest one might from linear regression literatures, such as the notion of R^2. There are also more recent one worths discussing in the literature. \n\n- Other minor points:\n    - The authors criticize the limitation of GradCAM is heavily depending on the architecture, which I think it's not a proper statement, as it has been widely applied to many different architectures with practical success. I would like to know authors' opinion on it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ntEQ6-2XV-i", "reviewer": "Reviewer_yYjF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors suggest a new method for interpreting decisions by deep learning models using a variance-based sensitivity analysis method. By such analysis of the input features the authors claim that they can analyse not only the contribution of individual features but their relation as well, for example, such image patches. The method is evaluated in an experimental setup using image net and some known CNNs.", "summary_of_the_review": "This paper is not ready for publication. More needs to be done to arrive at significant statements about the development of CNN architectures and their impact on the representation capability in the model. Compared to other works on interpreting CNN output, this paper lacks theoretical contribution and experimental evaluation.", "strength_and_weaknesses": "I like first, the questions asked and second the idea of exploiting known methods from feature relevance analysis to understand the output of black-box models. I am kind of disappointed about the implementation of this general, very promising idea. The details of combining Sobol with SBSA are insufficient to support their claims. For example, why shall the sum of pixels in a region be a proper semantic representation of an image(part)? Also, the generation of the low-dimensional feature is arbitrary by the selection of image patches. I am missing a detailed study and motivation for this step. Second, experimental results lack clarity and insight. When comparing with other methods to explain the output of a CNN, their metric is not always the best. Overall, the metrics used, for example, PIR, deliver interesting insight, but in my opinion not more. Figure 3 shows an observation for a very specific pipeline of SBSA and Sobol, and not even one that I find reasonable (see my comment on a feature that uses the sum of intensity values). In summary, the idea is very promising but the current result is premature and without a significant contribution to the area. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jfGavgXyFFX", "reviewer": "Reviewer_phNs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a sample-based approach for interpreting the decision-making of neural networks for an image classification task. An image is divided into multiple regions and the importance of a region is evaluated separately along with its interaction with other regions. A Sobol variance-based sensitivity metric is used to estimate the importance of regions by varying the region and noting the effect on the prediction. The approach is validated with Imagenet validation images with respect to commonly used image classification models such as AlexNet, CGG16, ResNet50, and DenseNet161. ", "summary_of_the_review": "The paper provides an interesting idea of sample-based interpretability estimation. However, there are many aspects that need to be addressed to justify the impact of the approach. Please address the issues in the weaknesses section and I will be happy to reconsider the decision. ", "strength_and_weaknesses": "Strengths:\n\n1. The proposed sample-based approach to interpreting the decision-making of neural networks is interesting. \n\n2. The Sobol variance-based sensitivity metric is a simple yet effective approach to estimating the importance of regions. \n\n3. Authors conducted extensive experiments to evaluate various aspects of the approach. \n\nWeaknesses:\n\n1. Dividing images into a few regions to interpret the decision-making is simplistic. The proposed approach sum the pixel values in a region which may squash important variance cues in a region. Additionally, a coarse grid such as 4X4 or 8X8 may not respect object boundaries. What are the authors' comments on these issues?\n\n2. Authors claim to capture interactions between regions which is missing in the existing works. However, the way the interaction is measured is effectively based on the individual importance of the regions. I am missing what aspect of Eq 11 captures non-linear region dependencies for decision-making that are missing in Eq 9?\n\n3. Authors mentioned the G^{-1} and H^{-1} functions. Where are these inverse functions used?\n\n4. Can G and H be combined in a single encoder and G^{-1} and H^{-1} combined in a single decoder? For example, one can combine G and H in a single encoder such as VAE and consider the latent representation as the lower dimensional representation $r$. Is the proposed approach applicable to such representations? \n\n5. As in figure 3 (a), the interactions between the regions increase with a lower number of regions. This is counterintuitive as discussed in section 4.2.3. This raises the question of how the approach depends on the size of the regions. It will also be useful to visually inspect the results on a set of images while changing the size of the regions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject