Democratizing Evaluation of Deep Model Interpretability through Consensus
Authors: Xuhong Li
Haoyi Xiong
Siyu Huang
Shilei Ji
Yanjie Fu
Dejing Dou
Venue:  (2021)
Keywords: interpretability evaluation
deep model interpretability

Abstract: Deep learning interpretability tools, such as (Bau et al., 2017; Ribeiro et al., 2016; Smilkov et al., 2017), have been proposed to explain and visualize the ways that deep neural networks make predictions. The success of these methods highly relies on human subjective interpretations, i.e., the ground truth of interpretations, such as feature importance ranking or locations of visual objects, when evaluating the interpretability of the deep models on a speciﬁc task. For tasks that the ground truth of interpretations is not available, we propose a novel framework Consensus incorporating an ensemble of deep models as the committee for interpretability evaluation. Given any task/dataset, Consensus ﬁrst obtains the interpretation results using existing tools, e.g., LIME (Ribeiro et al., 2016), for every model in the committee, then aggregates the results from the entire committee and approximates the “ground truth” of interpretations through voting. With such approximated ground truth, Consensus evaluates the interpretability of a model through matching its interpretation result and the approximated one, and ranks the matching scores together with committee members, so as to pursue the absolute and relative interpretability evaluation results. We carry out extensive experiments to validate Consensus on various datasets. The results show that Consensus can precisely identify the interpretability for a wide range of models on ubiquitous datasets that the ground truth is not available. Robustness analyses further demonstrate the advantage of the proposed framework to reach the consensus of interpretations through simple voting and evaluate the interpretability of deep models. Through the proposed Consensus framework, the interpretability evaluation has been democratized without the need of ground truth as criterion.

Under review as a conference paper at ICLR 2021
DEMOCRATIZING EVALUATION OF DEEP MODEL IN- TERPRETABILITY THROUGH CONSENSUS
Anonymous authors Paper under double-bling review
ABSTRACT
Deep learning interpretation tools, such as (Bau et al. 2017 Ribeiro et al. 2016 Smilkov et al. 2017). have been proposed to explain and visualize the ways that deep neural network (DNN) classifiers make predictions. However, the success of these methods highly relies on human subjective interpretations, i.e., the ground truth of interpretations, such as feature importance ranking or locations of visual objects, when evaluating the interpretability task. For tasks that the ground truth of interpretations is not available, we propose a novel framework Consensus incorporating an ensemble of deep models as the committee for interpretability evaluation. Given any task/dataset. Consensus first obtains the interpretation results using existing tools, e.g., LIME (Ribeiro et al., 2016), for every model in the committee, then aggregates the results from the entire committee and approximates the "ground truth" of interpretations through voting, With such quasi- coausi-groun--rruee Consensus evaluates the interpretability of a model through matching its interpretation result and the approximated one, and ranks the matching scores together with committee members, so as to pursue the absolute and relative interpretability evaluation results. We carry out extensive ex- periments to validate Consensus on various datasets. The results show that Con- sensus can precisely identify the interpretability for a wide range of models on ubiquitous datasets that the ground truth is not available. Robustness analyses further demonstrate the advantage of the proposed framework to reach the con- sensus of interpretations through simple voting and evaluate the interpretability of deep models. Through the proposed Consensus framework, the interpretability evaluation has been democratized without the need of ground truth as criterion.
the DNN classifiers on a specific
INTRODUCTION
Due to the over- parameterization nature (Allen-Zhu et al. 2019), deep neural networks (DNNs) (Le- Cun et al. 2013) have been widely used to handle machine learning and artificial intelligence tasks, however it is often difficult to understand the prediction results of DNNs despite the very good per- formance. To interpret the DNN classifiers' behaviors, number of interpretation tools (Bau al.)
2017
Ribeiro et al., 2016)] Smilkov et al. 2017]] Sundararaaan et al. [201]] Zhang et al. 2019 Ahern al. 2019) have been proposed to recover or visualize the ways that DNNs make decisions.
Preliminaries. For example, Network Dissection (Bau et al. 2017) uses a large computer vision
dataset with number of visual concepts identified/localized every image. Given convolutional neural network (CNN) model for interpretability evaluation, it recovers the visual features used by the model for the classification of every image via intermediate-layer feature maps, then matches the visual features with the labeled visual concepts to estimate the interpretability of the model as the intersection--ver-une objects. Related tools that interpret CNNs through locating importation subregions of visual features in the feature maps have been proposed in (Zhou et al. 2016, Selvaraju et al. 2020 Chattopadhay et al.
(IoU) between the activated feature maps and labeled locations of visual
2018 Wang et al. 2020a
Apart from investigating the inside of complex deep networks, (Ribeiro et al. 2016 van der Linden al. 2019: Ahern et al.) 2019)) proposed to use simple linear tree-based models surrogate the predictions made by the DNN model over the dataset through local or global approximations, so as to capture the variation of model outputs with the interpolation of inputs in feature spaces. Then,

---

Under review as a conference paper at ICLR 2021
Interpretations
Aggregation of interpretation
For each image in the dataset
(asing LIME example) Mob dless
results of models for this imago and achieve consensus
Step 2:
Step 1:
Consensus achievement
Prepare of trained models
Redk-ti11
MaskArta0 Alord
Step 3: Evaluating interpretability of all models through consensus
Interpretations of every model
Compute similarity berween the interpretation and the consensus.
For each model. the interpretabilliy is the average similarity to the consensus (over all data examples).
Quasi-ground-tnun
Using Consensus evaluate the interpretability of DNN classifiers with dataset. For every image in the dataset, Consensus (1) prepares set of trained models as committees, (2) aggre- gates interpretation results from every model approximate the ground truth of interpretation, and (3) compares the interpretation of every model to the aggregated one to evaluate the interpretability.
Figure I:
with the surrogate model, these methods interpret the DNN model ways the model uses features for predictions, e.g., ranking of feature importance, and compare the results with the ground truth labeled by human experts evaluate interpretability. Besides the use of linear interpolation for sur- rogates, many algorithms, like SmoothGrad (Smilkov et al. 2017). Integrated Gradients (Sundarara jan al. 2017), DeepI.IFT (Shrikumar et al.) (017)), and PatternNet (Kindermans al., 2018) have been proposed to estimate the input feature importance as the way to interpret the models, so as to interpret the model predictions by highlighting the importation subregions in the input. addition to the above methods, (Zhang et al. 20188a] 2019) proposed to learn a graphical model to clarify the process of making decision at semantic level. Note that obtaining the interpretation of model is an algorithmic procedure to explain the model (Samek et al. (017)). On the other hand, through the comparing the interpretation results with the human labeled ground truth, the interpretability evalu- ation aims at estimating the degree to which human (expert) can consistently predict the model's result (Kim et al. 2016, Doshi- Velez & Kim (017).
In summary, the ground truth of interpretation results (usually labeled by human experts) is indis- pensable to all above methods for interpretability evaluations and comparisons, no matter ways they interpret the models, e.g., visual concepts detecting (Bau et al. (017). and feature importance rank- ing for either local (Ribeiro et al., 2016) or global (Ahern et al., 2019) interpretations. While the datasets with visual objects abbeld//ocaliier and/or the importance features ranked have offered in some specific areas, the unavailability of ground truths also limits the generalization of these meth- ods to interpret brand new models on the new tasks/datasets ubiquitously. There is thus the need of a method being able to evaluate the interpretability of models on the datasets where the ground truth of interpretation results is not available.
Our contributions. In this paper, we study the problem of evaluating the interpretability of DNN classifiers the datasets without ground truth of interpretation results. The basic idea of Consensus is to leverage the interpretability of known models as reference to predict the interpretability of new models on new tasks/datasets. Especially, in terms of general purpose perception tasks, we have al- ready obtained a number of reliable models with decent interpretability, such as ResNets, DenseNets and so on. With a new dataset, one could use interpretation tools (Ribeiro al. 2016 Smilkov al.) 2017) to obtain the interpretation results of these models, then aggregate interpretation results the reference. Then for any model, one could evaluate interpretability of the model through comparing its interpretation results with the reference.
Specifically, as illustrated in Figure ! we propose a novel framework named Consensus that uses large number of known models task/dataset. Consensus first obtains the interpretation results for every model in the committee us- ing existing interpretation tools, e.g., LIME (Ribeiro al. 2016), then aggregates the results from
committee for interpretability evaluation. Given any
2

---

for every Under review as a conference paper at ICLR 2021
the entire committee and reaches the consensus of interpretations through voting. With the quasi- ground-truth, Consensus evaluates the interpretability of each model through matching its interpre- tation result and the approximated one, and ranks the matching scores of committee members, so as to pursue the absolute and relative interpretability evaluation results without the ground truth of interpretations labeled by human experts. More specifically, make contributions follows.
We study the problem of interpretability evaluation on the datasets without human labeled ground truth of interpretations. To the best of our knowledge, this work is the first to study the problem of evaluating the interpretability of DNNs while the ground truths of interpretation results are not available, by addressing the technical issues of voting and committees.
We design and implement Consensus, novel interpretability evaluation framework that incor- porates a wide range of alternating interpretation tools, such as LIME (Ribeiro et al.] (0166), SmoothGrad (Smilkov et al., 2017), interpret the model as the variation of outputs over the interpolations of inputs (in feature spaces), from massive perspectives (e.g. local or global inter- pretation, tree-based or linear surrogate and on), and carries out the interpretability evaluation through the voting based on interpretation results of models in the committee.
We carry out extensive experiments to validate Consensus on a wide range of models on new ubiquitous tasks/datasets that the ground truth is not available, and exploit the quantifiable metric of model interpretabilityy to report the overall interpretability evaluation results (Section 33. Case studies (Section 4 confirm the effectiveness of Consensus and show the closeness of Consen- sus based results to the ground truth of interpretations. Robustness analyses (Section 55) further demonstrate the advantage of the committee that the factors including the use of basic interpre- tation algorithms, the types of networks in the committee, and the size of committee would have few affects on the Consensus based interpretability evaluation results.
CONSENSUS: A FRAMEWORK OF INTERPRETABILITY EVALUATION
In this section, we introduce our proposed framework, namely Consensus, which incorporates ex- isting interpretations, such as LIME (Ribeiro et al., 2016) and SmoothGrad (Smilkov et al. 2017). to enable DNN interpretability evaluation without the use of human labeled interpretation results as reference/grount truth. Specifically, Consensus generalizes a simple electoral system, and consists of three steps: (1) Committee Formation with Deep Models, (2) Committee Voting for Consensus Achievemen T and (3) Consensus-based Interpretability Evaluation, as follow.
Committee Formation with Deep Models. Given a number of deep neural networks (which are well-known with decent performance on common perception tasks) and a target task with a dataset (potentially without ground truth of interpretation results), Consensus first trains the given neural networks (from scratch fine-tuned) using the dataset. Then, Consensus forms the post-trained networks as a committce of models, noted as .M, and considers the variety of interpretability of models in the committee that would establish the references for interpretability comparisons and evaluation. Note that while our research assumes the human labeled ground truth of interpretation results is not available in the given task/dataset for interpretability evaluation, the labels of samples are requested when handling classifications and regression tasks.
Committee Voting for Consensus Achievement. With the committee of trained models and the target task/dataset for interpretation, Consensus first leverages an existing interpretation tool A, e.g., A can be LIME (Ribeiro et al., (016) or SmoothGrad (Smilkov al.. 2017) alternatively, to obtain the interpretation results of every model in the committee on every sample in the dataset. Given some sample d,, we note the obtained interpretation results of all models as L. Then, Con- sensus proposes voting procedure that aggregates L to achieve the consensus e as the quasi- ground-truth of the interpretation for the sample d,. Specifically, e = 1 m 12, for LIME and I
maz(L.)-monn(L, for SmoothGrad. In summary, Consensus adopts a normalization- averaging procedure to obtain the quasi-ground-truie truth of interpretations for the sample. To the end, the consensus has been achieved through obtaining the collections of sample in the target dataset based on committee voting.
C₁ = - La-min(L.)
Consensus-based Interpretabilityy Evaluation. Given the the whole committee, the proposed algorithm evaluates the interpretability of every model in the
as the consensus of
We are not intending to connect our work with the multi-agent research though use the term "consensus achievement".
3

---

Under review as a conference paper at ICLR 2021
committee by considering the similarity between the interpretation result of each individual model and the consensus of the whole committee. Specifically, for the interpretations and consensus based on LIME, Consensus uses cosine similarity between the flattened vectors of interpretation of each model and the consensus. Then Consensus quantifies the interpretability of the model through the mean of similarity measures over all samples. For the results based on SmoothGrra (visual feature importance in pixel levels), Consensus follows a similar procedure, where the proposed algorithm uses Radial Basis Function (ezp(- { ))) for the similarity measurement. We rank all models in the committee using their similarities to the consensus and consider the top/bottom models with good/bad interpretability in the committee.
Consensus Framework Pseudocode. The functions interpret(), aggregate() and sim( are described in the main text and detained in Algorithm 2 in Appendix
Algorithm 1:
function Consensus(D, .A)
Input : A dataset D containing v examples (di)stt,.... and an interpretation algorithm AA. Output: E IR",, where each element sj indicates the interpretability of each model M, in M. Step 1: Committer Formation with Deep Models .M
Prepare M containing m models {My)s-,,-. ie.. the committee of deep models.
S zeros(n, m) // Initialize an empty n X m matrix for storing the interpretabllity scores of m models en data sample.
for i 1, 1,....... do
L zeros(m,
for j in 1,... m L, interpret( M,. d.)
7
end
/x Step 2: Committee Voting for Consensus Achievement at di
*/
C aggregate(L) // CE consensus as quasi-ground-trute
/. Step 3: Consensus-based Interpretability Evaluation at d. */
10
for in 1, m do
S, sim e) // the score of M, at d:
end
13 end
14 for in 1.... do
83 average (S average score for each model over n samples
16 end
17 return s
These three steps of Consensus are illustrated in Figure I and formalized in Algorithm Note that for any new models for interpretability evaluation, Consensus includes them as members of committee together with a number of known models, and performs above procedures to obtain their interpretability evaluation results (absolute evaluation results) as well as the ranking in the committee (relative evaluation results). In this way, potentials (in terms performance) of the new models on the new tasks among the known models, when the ground truth of interpretation results not available.
clearly position the interpretability and
3 OVERALL EVALUATION AND RESULTS
In this section, evaluation of deep models. We first introduce the settings of image classification tasks in details, as the setups of our experiments. Then, present the overall results of Consensus, where could observe its capacity of evaluating the interpretability of models, with connections to the model performance, while the ground truth of interpretations is not used. Through the comparisons with interpretability evaluation based on LIME and SmoothGrad algorithms using human labeled ground truth, the effectiveness of Consensus has been evaluated.
the image classification as the target task for interpretation and interpretability
3.1
EVALUATION SETUPS
Here present the setups of experiments from following perspectives.
Datasets and Models. For overall evaluation and comparisons, we use two image classification datasets ImageNet (Deng al. 2009) for ubiquitous visual objects and CUB-200-2011 (Welinder et al. 2010) for birds respectively. Note that ImageNet provides the class label for every image; CUB-2006 100-200-201 dataset includes the class label and pixel-level segmentation for the bird in every
4

---

Under review as a conference paper at ICLR 2021
image, where the pixel annotations of visual objects have been considered as the ground truth of in- terpretations (Bau et al.) 2017). In this way, we evaluate interpretability of models using Consensus, and compare the results with the existing algorithms based on the ground truth.
For the fair comparisons, we use more than 80 models publicly available from PaddlePaddla 2 which have been trained using ImageNet dataset. We also derive models based on CUB-2200-011 dataset through standard fine-tuning procedures. In our experiments, we include these models in two com- mittees based on ImageNet and CUB-200-2011 respectively for the interpretability evaluation.
Baselines. We consider two interpretation algorithms, LIME (Ribeiro et al. 2016) and Smooth- Grad (Smilkov al.) 2017). Specifically, LIME surrogates the interpretation as the assignment of visual feature importance to superpixels (Vedaldi & Soatto 2008), SmoothGrad outputs the in- terpretation results as the visual feature importance over pixels. In this way, we can evaluate the flexibility of the proposed Consensus framework over interpretation results from diverse sources (i.e., linear surrogates vs. input gradients) and in multiple gramularity (i.e., feature importance in superpixel/pixxl-leeess). Note that both algorithms use mean Average Precision (mAP) between their interpretation results and the ground truth (i.e., pixel level segmentation, if available) as the measure of interpretability evaluation.
Metrics. Given the similarity measures between the model's interpretation results and consensus of the committee, the pro- posed algorithm is used to rank the inter- pretability of every model in the commit- tee. In this way, we compare the rank- ing list of model interpretability based on Consensus evaluation results with the ranking list based on the ground truth, so as to understand how well the pro- posed Consensus framework can approx- imate the evaluation results of model in- terpretability without the use of ground truth. More specifically, visual com- parisons, Pearson correlation coefficients (which characterize the linearity between two variables, e.g., model performance vs. interpretability evaluation) and signif- icance tests have been used as the metrics for overall evaluation and comparisons.
Doubba
(a) LIME
(b) SmoothGrad
Figure 2: Correlation between the Model Performance (Testing Accuracy) and the Ground Truth based Inter- pretability Evaluation using (a) LIME and (b) Smooth- Grad with CUB-200-2011 over 85 models. Pearson correlation coefficients are 0.927 (with p-value 4c-37) for LIME and 0.916 (with p-value 9e-35) for Smooth- Grad. The points "Consensus" here refer to the testing accuracy of the ensemble of networks in the commit- tee by probabilities averaging and voting (in y-axis), as well as the mAP between the Consensus results and the ground truth (in x-axis).
Note that we do not have the ground truth of interpretations for ImageNet. However, is still possible for us to access the interpretability eval- uation through connecting Consensus outputs with the generalization performance of the models. Figure 2 illustrates an example of correlations between interpretability evaluation results based on ground truth and the testing accuracy of 85 models over CUB-200-2011 dataset, where we can see strong, significant, and consistent correlations between model performance and the interpretability.
3.2 OVERALL COMPARISONS
In Figure B. we present the interpretability evaluation results using CUB-200-200 and the overall comparisons between Consensus and the ground truth based, where we plot the scatter points (i.c., Consensus results in x-axis vs. ground truth based results in y-axis) of the two comparisons using LIME and SmoothGrad respectively. In both comparisons, Consensus performs almost-identically as the one based on ground truth with strong Spearman's 's correlation (which characterizes the con- sistency between two ranking lists) with significance tests passed.
To enable the similar comparisons using ImageNet dataset (where the ground truth of interpretations is not available), we connect the Consersus results with the model performance (testing accuracy) of models, as the model performance and the ground truth based interpretability evaluation results are usually correlated (please see also in Figure 2 and section 3.1). In Figure we present the
image_c cl assi fi caciaitionn/ooo REA JMN http://wwwww..eonooooooooorroooo hag.://eemmooooee---mmemeorrrooo
5

---

Under review as a conference paper at ICLR 2021
Medda
3091
4/5-
6:08
cid VISS "iiee" Vico
(a) LIME on ImageNet
(b) SmoothGrad on ImageNet
(c) LIME vs. SmoothGrad on ImageNet
0.80
Akollis
11:00
Disamile
0.75
0.20
Eesration
Aleeliot
AmniD Tiing
X
Reittin
Bookl
1.00
1.30
E325
1003
11.
Seailuryy the eocamana
Medaray to the mexmrra
(d) LIME on CUB
(c) SmoothGrad on CUB
(f) LIME vs. SmoothGrad on CUB
Figure 4: Model performance v.s. similarity to the consensus using LIME (a.d) and SmoothGrad (b,e) over 81 models on ImageNet (a,b) and 85 models on CUB-200-201 (d,e). The third column shows the similarity to the consensus of SmoothGrad interpretations v.s. similarity to the consensus of LIME interpretations on ImageNet committee (c) and CUB-200-200 committee (f). Pearson correlation coefficients are (a) 0.8087, (b) 0.783, (c) 0.825, (d) 0.908, (c) 0.880 and (f) 0.854.
correlations between the Consensus results (without the use of ground truth) and the model perfor- mance for both LIME and SmoothGrad using ImageNet and CUB-200-2011 datasets. Specifically, in Figure 4 (a-b) and (d-e), we present the comparisons between model performance (in y-axis) and the Consensus results (in x-axis) using LIME (a,d) and SmoothGrad (b,e) on ImageNet (a.b) and CUB-200-201 (d.e) respectively.
All correlations here are strong with sig- nificance tests passed, though in some lo-
cal areas of the correlation plots between
1.00
model performance and interpretability evaluation the trends are not always con- sistent. It has been observed that some extremely large networks work well with the datasets, while they are lack of in- terpretability (Bau al. 2017). In this way, we could conclude that, in an overall manner, interpretability evaluation results based on Consensus using both LIME and SmoothGrad over the two datasets are cor- related to model performance with signifi- cance.
AA2
420
(a) Using LIME
(b) Using SmoothGrad
Figure 3: -ruund-Tuuthbbsed v.s. Consensus-based Interpretability Evaluation using (a) LIME and (b) SmoothGrad with CUB-200-2011 Datasets over 85 models (the Committee). Spearman's 's correlation co- efficients are 0.885 (with p-value 3e-29) for LIME and 0.906 (with p-value 8e-33) for SmoothGrad.
3.3 COMPARISON RESULTS WITH NETWORK DISSECTION (BAU ET AL.
2017
Here we compare the results of Consensus with ground truth based interpretability evaluation so- lution - Network Dissection (Bau et al. 2017). With ImageNet dataset, Network Dissection gave a ranking list of models (w.rt the model interpretability)
follows: ResNet152 > DenseNet16l > VGG16 > GoogLeNet > AlexNet. We report two ranking lists based on Consensus with de- tailed numbers for every architecture above, which has been demonstrated in Figure 4 (a. LIME): DenseNet161 (0.849) x ResNet152 (0.846) > VGG16 (0.821) > GoogLeNet (0.734) > AlexNet (0.594); and (b, SmoothGrad): DenseNet161 (0.038) ResNet152 (0.037) > VGG16 (0.030) >

---

Under review as a conference paper at ICLR 2021
MocSeleett2
Reclen201
mileetteed
Resiien2102 32bid
@t. veouum
MooReNett2
ResMen101
Effile (tNed00
Replenz10), 32x0d
condenuu
Figure 5: Visual comparisons between consensus and the interpretation results of CNNs using LIME (in the upper line) and SmoothGrad (in the lower line) based on an image from ImageNet, where the ground truth of interpretation results is not available.
GoogLeNet (0.026) AlexNet (0.021). The three ranking lists almost identical, except the com- parisons between DenseNet161 and ResNet152, where in the both lists Consensus, DenseNet16 61 is similar 52 with marginally elevated interpretability, while Network Dissection consid- ers 55s is more interpretable than DenseNet16l.
We believe the results from Consensus and Network Dissection are close enough from the perspec- tives of ranking lists, and the difference may be caused by the different ways that Consensus and Network Dissection evaluate the interpretability. Consensus matches the interpretations with the visual objects on images (due to the results of LIME and SmoothGrad), while Network Dissection counts the number of neurons activated by the visual objects and patterns (color, materials, textures, scenes, and parts). Furthermore, Network Dissection evaluates the interpretability of deep models using the Broden dataset with densely labeled visual objects and patterns (Bau et al. (017). while Consensus does not need additional dataset or ground truths of interpretations. In this way, the results by Consensus and Network Dissection might be slightly different.
4
CASE STUDIES
In this section, we discuss several technical issues on Consensus for ground-truth-free evaluation of DNN interpretability, using of case studies.
Qualification and Effectiveness of Committec-based Voting In our research, Consensus pro- poses to replace the ground truth with a committee of networks, and it trusts the voting results as the interpretations of ground truth. Thus, we have to verify (1) whether the (consensus achieved by the) committee would approximate to the ground truth, and (2) whether voting is an effective way to express the interpretation results of the whole committee.
To achieve the goal, we consider the committee as an ensemble of networks, and it classifies data via committee-bsed voting through averaging the probability outputs of the member networks. In Figure we compare the committee (as ensemble of networks, entitled with "Consensus") with other architectures for the evaluation of model performance (testing accuracy) and the interpretabil- ity evaluation with the ground truth (i.e., mAP between consensus achieved by the committee and the ground truth), using both LIME and SmoothGrad over CUB 200-2001 dataset. The comparison results show that the committee is of the best testing accuracy with significant advantages compared to other models, while the committee is also with the highest mAP between its consensus interpre- tations and the ground truth compared to other models. In this way, we can confirm the qualification of the committee, as well as the effectiveness of the commiltee-based voting.
Closeness of Consensus to the Ground Truth Figure 2 between consensus and the ground truth on CUB-200-2011 dataset, we also visualize the examples to compare the consensus achieved by the committee, interpretation results of individual networks using both LIME and SmoothGrad, and (optionally) the ground truth labeled for both ImageNet and CUB-200-2011 datasets in Figures 5 and respectively. The comparison shows that the consensus can clearly segment the visual objects related to the classification from the background of images, and it would be closer to the ground truth (if available) than the individual networks. Both quantitative results in Figure 2 and the visual comparisons in Figures 5 and 6 validate the closeness of consensus to the ground truth of interpretations.
In addition to the mAp measurement (illustrated in

---

Under review as a conference paper at ICLR 2021
Wh ove Wes
desked E
PAR MANAVO 8300
Restiction
micereen 0.87
Recleonii Ride
censerAe 0.75
Ortund
Fedhecto::
Figure 6: Visual comparisons between consensus and the interpretation results of CNNs using LIME (in the upper line) and SmoothGrad (in the lower line) based on an image from CUB-200-2011 where the ground truth of interpretation results is available as pixel-wise annotations and the mean Average Precision (mAP) are measured for interpretability evaluation.
5 ROBUSTNESS ANALYSES
In this section, we discuss several factors, including the use of basic interpretation algorithms (e.g., LIME and SroothGrad), the size of committee, and the candidate pool for models in the committee, that would affect the proposed Consensus framework.
Consistency of LIME and SmoothGrad While Consensus adopts LIME and SmoothGrad as the basic interpretation algorithms, the interpretation results from these two algorithms not exactly the same. Even though the granularity of interpretation results are different, which causes mismatch- ing in mAP estimation with ground truth, the interpretability evaluation results of Consensus based on the two algorithms are generally consistent. The consistency has been confirmed by Figure (c, f), where the overall results of Consensus based LIME strongly correlated Smooth over all models. This shows that the proposed Consensus framework can work well with a wide spectrum of basic interpretation algorithms.
Consistency of Cross-Committee Interpretability Evaluation In real-word applications, the committee- based evaluation makes the results inconsistent in committee-be soammittee-b--oomiitee cycomittee manner. In this work, we are interested in whether the interpretability evaluation is consistent against the change of committee (e.g., using different sets of models). Given 16 ResNet models as the targets, we form 20 independent committees through combining the 16 models with 10-20 models randomly drawn from the networks presented Figure In each of these 20 independent committees, we use Consensus to evaluate the interpretability of T6 ResNet models and rank them accordingly. We then estimate the Pearson correlation coefficients between any of these ranking lists and the list in Fig- ure 4 (a), where the mean correlation coefficient is 0.96 with the standard deviation 0.04. Thus, we can say the interpretability evaluation based on randomly picked committees would be consistent.
Convergence over Committee Sizes To understand effects of committee sizes to interpretabil- ity evaluation, we run Consen- sus using committees of various sizes formed with networks ran- domly picked up from the pools. In Figure 7 we plot and com- pare the performance of the con- sensus with increasing commit- tee sizes, where estimate the mAP between the ground truth and the consensus based the random committees of different sizes and 20 random trials have been done for every single size independently. shows that the curve of mAP would quickly converge the complete the commit- tee, while the consensus based on a small proportion of committee (e.g., 15 networks) works good enough even compared to the complete committee with 85 networks.
0.70
0.68
mAP
0.50
Cummitoe size
Figure 7: Convergence of mAP between the ground truth and the consensus results based on committees increasing sizes, using LIME on CUB-200-2011. The green lines the mean values and the orange triangles are the median value. The red dashed line is the mAP of the consensus based on the complete committee of the original 85 models.
Applicability with Random Committees over Other Datasets To demonstrate the applicability of Consensus with varying committees over other datasets, continue our experiments using net-
8

---

Under review as a conference paper at ICLR 2021
works randomly picked up from the pool other datasets, including Stanford Cars 196 (Krause iet al. 2013). Oxford Flowers 102 (Nilsback & Zisserman (008 and Foods 101 (Bossard et al. 2014 in Figure 8 where we consider the connections between the interpretability and model per- formance (e.g., the testing accuracy, inspired by Figure 2). The results confirm that, when the ground truth of interpretations is not available, framework is still capable of identifying the interpretabil-
ity for a wide range of models on ubiquitous datasets/tasks.
Pask Bho
300
Mucaootco 2
Figure 8: Model performance versus Results of Consensus using LIME on Stanford Cars 196 Krause et al. 2013). Oxford Flowers 102 (Nilsback & Zisserman 2008) and Foods 101 (Bossard
al. 2014). Pearson correlation coefficients are 0.9522, 0.8785 and 0.9134 respectively.
6 CONCLUSION AND FUTURE WORK
We have proposed: a novel framework Consensus for evaluating the interpretability of deep models while avoiding the use of ground truth for interpretations. Specifically, Consensus forms a com- mittee of deep models and generalizes an electoral system to reach the consensus of interpretation results using some basic interpretation algorithms via committce-based voting. Then, for every model in the committee, Consensus computes the similarity between its interpretation results and the aggregated consensus, then it ranks the models in the committee accordingly, so as to pursue the absolute score (i.e., similarities the consensus) and relative results (the rank in the committee) for interpretability evaluation. To validate Consensus, we carry out extensive experimental studies using 85 deep models including ResNets, DenseNets, and so on, on top of 5 datasets, including ImageNet, CUB-200-222 and so on, in comparison with interpretation algorithms of three categories, includ- ing LIME (Ribeiro et al., 2016). SmoothGrad (Smilkov et al.) (017). and Network Dissection (Bau iet al. 2017]. The results show that (1) Consensus can evaluate the interpretability of models on the datasets even when the ground truth of interpretation results are not available, (2) the consensus of interpretation results aggregated from the committee could well approximate the ground truth of interpretations, (3) the interpretability evaluation results delivered by Consensus correlates to the model performance (testing accuracy) strongly and significantly, (4) the factors including the use of basic interpretation algorithms, the types of networks in the committee, and the size of committee would not affect the results of interpretability evaluation with Consensus.
Discussion. For the interpretability evaluation of any models on any dataset, the ground truth based evaluation approaches rely on human subjective interpretations, while Consensus can automate this process by just training a few more models and approximating the ground truth of interpretations. We thus conclude that the interpretability evaluation can be democratized through an electoral sys- tem constructed by the DNN models themselves, rather than the use of human labeled ground truth as criterion. We discuss the future work in three different directions. (1) In terms of methodologies. the proposed Consensus framework considers the segmentation of visual features as interpretations for vision tasks and adopts simple voting mechanism to aggregate results from the committee of models. We believe that the contributions made in this work are complementary with visual objects segmentation and voting. The use of advanced segmentation (Chen et al., 201 He et al., 2017) and ensemble learning methods (Dietterich) 2000; Hinton et al.) 2015) would further improve the proposed framework. (2) In terms of applications, following the steps of Consensus, on medical or financial domains where interpretations for black-box models are urged, the quasi-ground-trutt of interpretations and the interpretability evaluation of models could be easily obtained. (3) Instead of using trained models of different architectures as committee members, models of common or even the same architectures trained using various training strategies would also form an interesting committee for analyzing the interpretability of models based on different training strategies.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Isaac Ahern, Adam Noack, Luis Iizzaan-Nateral Dejing Dou, Boyang Li, and Jun Huan. Norm- lime: new feature importance metric for explaining deep neural networks. arXiv preprint arXiv:/909.0420 2019.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. In Advances in neural information processing systems, pp. 6158-6169, 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In IEEE Transactions on Pattern Anal- ysis and Machine Intelligence (TPAMI), pp. 6541-6549, 2017.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.
discriminative com- ponents with random forests. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 446-461 Springer, 2014.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian, Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 839-847. IEEE, 2018.
Liang- Chang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. pp. 834-848, 2017a.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojic Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4467-4475, 2017b.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 1251-1258, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1911-1920, 2019.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint 1rXiv:/702.08660 2017.
Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Yu Zhang, Ming-Hsur Yang, and Philip HS Torr. Res2net: A new multi-scale backbone architecture. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In IEEE Interna- tional Conference on Computer Vision (ICCV), pp. 2980-2988. IEEE, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NeurIPS Deep Learning and Representation Learning Workshop, 2015. http:
org/ abs//bb//aa/ 1503. 02531
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In JEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
10

---

Under review as a conference paper at ICLR 2021
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-ande and and- excitatiin networks. IEEE Conference on Computer Vision and Pultern Recognition (CVPR), 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely nected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700-4708, 2017.
Forrest landola, Song Han, Matthew Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv: 2016.
Been Kim, Rajiv Khanna, and Oluwasammi O Koyejo. Examples are not enough, learn to criticize! criticism for interpretability. In Advances in neural information processing systems, pp. 2280- 2288, 2016.
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Rober Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternat- tribution. In International Conference on Learning Representations (ICLR), 2018
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 554-561, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances Neural Information Processing Systems (NeurIPS), 2012.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv:1806.0900 09055, 2018.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp- 722-729. IEEE, 2008
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 1804: 2018.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intelli- gence: Understanding. visualizing and interpreting deep learning models. arXiv preprint arXiv: 1708v 08296, 2017.
11

---

Under review as a conference paper at ICLR 2021
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chigh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. International Journal of Computer Vision (IJCV). 128(2)3336-35) 2020.
Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Over- feat: Integrated recognition, localization and detection using convolutional networks. In Interna- tional Conference on Learning Representations (ICLR), 2014.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning (ICML), pp. 1145-3153. 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. In ICML. Workshop on Visualization for Deep Learning, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML), 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1-9, 2015.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), 2019.
Ilse van der Linden, Hinda Haned, and Evangelos Kanoulas. Global aggregations of local explana- tions for black box models. FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety- SIGIR 2019 Workshop, 2019.
Andrea Vedaldi and Stefano Soatto. Quick shift and kernel methods for mode seeking. In Proceed- ings of the European Conference Computer Vision (ECCV), pp. 705-718. Springer, 2008.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Work- shops, pp. 24-25, 2020a.
Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. 2020b.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD birds 200. Technical Report CNS-TR-200 2000000000, California Institute of Technology, 2010.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2017.
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Zhu. Interpreting cnn knowledge an explanatory graph. AAAI Conference on Artificial Intelligence, 2018a.
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting enns via decision trees. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6261-6270, 2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2018h.
12

---

Under review as a conference paper at ICLR 2021
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921-2929, 2016.
13

---

Under review as a conference paper at ICLR 2021
A MORE VISUALIZATION RESULTS
We present more visualization results of Consensus, where the samples are from ImageNet and CUB-200-2011.
14

---

Under review as a conference paper at ICLR 2021
Temiit crab
Nobilekee222
Resket103
FffcientNerto
Replenzz0:: 32oid
Necceet
Nobeiawety
Roohet122
CttclentHerDD
Desfacti00: 32bdd
American block boo
Alceiict
NobridhctV22
Reshett03
#ffcionNNce00
ResMeRta01 32ned
corserous
chibon
Aleslett
Nobileeet
Recket103
Reolwet110) 32yid
consencus
ge. ting cre
Ateclet
Noblalaer22
Techett33
Declwit101 37sld
CONEENOUE
s0u wchn
Alcehci
RCAvt101
COTOSTOLD
St perile Bays
Alxwlet
Mobieehet22
Recket10)
EffceenNe:00
Restes1301, 32ndd
beacon
Noblielwe22
#ffcientNer00
Restw1110: 32xid
COnRENOLE
casifionnr
Nobratwee22
COTARTRIN
Nobrichest22
Figure 9: More visual comparisons between the consensus interpretation and the interpretation sults of CNNs with LIME on samples from ImageNet, where the ground truth interpretation results is not available. Note that consensus is the Consensus aggregated interpretation.
15

---

Under review as a conference paper at ICLR 2021
ge
MicreNNttv2
cornerOLO
mitowave
Akoohes
Macichottz
HosMer101
COnSETOLS
Ameriian brock
Akooled
MocilNNut122
McsMer2011
Hos"eXL101_ 12x4)
MccioNNttz
SOTRSTOLO
stu uchh
Atodhs:
MickeNutV2
Nc2Mc1101
tiltic-UhstB:
COTOSTOIS
suaald
MocieNutt22
faseroower
MocisNatr22
CoraETOLS
vieduct
Atodiec:
Mosicitut22
ResMe:101
Effiie-thet::
Ras1e00101 12ved
Coreero15
Figure 10: More visual comparisons between the consensus interpretation and the interpretation results of CNNs with SmoothGrad samples from ImageNet, where the ground truth of interpre- tation results is not available. Note that consensus the Consensus aggregated interpretation.
16

---

Under review as a conference paper at ICLR 2021
Whi
Wexwee
gl: monood Coutine
Mcslatettc 0.23
greadd
White inseled Sprree
Modket:
Mcsiatletr2-.47 0.43
Reste:100:
1cc00035 32440 1.40
Caneeen: 2.22
de. Geccecon
Nedeec 8:27
MC2 3
Paate:200:: 8:37
derdeedd:
Sathe002 J204: 5.30
COMMERAN 2:25
grandtuth
Lowi Rycitote
Aexten
Mesleten2--222 221
Reshe1100 9:48
Recho01Es 12460 E.22
gresi
greed
Wantem.Co
Nedlec 1-34
Mcslalenn2 3.17
Remte:00::
dendeetd: 50
SanloMEEL 22w1
greed
Figure 11:
More visual comparisons between the consensus interpretation and the interpretation results of CNNs with LIME on samples from CUB-200-2011, where the ground truth of interpre- tation results is available as pixel-wise annotations and the mAPs are measured for interpretability evaluation. Note that consensus is the Consensus aggregated interpretation.
17

---

Under review as a conference paper at ICLR 2021
Whi
Wextee:
Mcstiatetr 3.54
gread
gl Conbin
AaxWett
Mcplateet2:22
Reatentl:: 8:6
Aathootti, 2244
White instled Suaree
Aecket
Mcslsteb22-244
Reoht:100:
1ea0001E 12400 1.0
caeeee: 114
de Geccacon
Nedlec 8-43
MC2 22230e66222311 2.45
Feete:t00: 8-42
dendere:: 8:44
Ba Jooe E:6
CONEE:: 1:6
gresdtuth
Lowi Rycitoee
Mesietetr 231
Reshe:100
Recho0115 12460 t.10
ires)iluh
Sparow
greed
Wantem.Co
Nedeec 6-36
Mcslatebr:2 3:42
Pewtert0:: 2:4
dendweEd 45
TanHoOSEL 32wie C.0
grendiugh
Figure 12: More visual comparisons between the consensus interpretation and the interpretation results of CNNs with SmoothGrad samples from CUB-200-2011 where the ground truth of interpretation results available pixel-wise annotations and the mAPs are measured for inter- pretability evaluation. Note that consensus is the Consensus aggregated interpretation.
18

---

Under review as a conference paper at ICLR 2021
B EXPERIMENTAL DETAILS
We present the technique details for the experiments in the main text.
B.1 COMMITTEE FORMATIONS
There are around 100 publicly available deep models trained ImageNet from PaddlePaddle We first exclude some very large models that take much more computation resources. Then for the consistency of computing superpixels, we include only the models that take images of size x as input, resulting 81 models for the committee based on ImageNet. Since there are already large number of available models, we choose to not include more models by aligning the superpixels in different sizes of images.
As for CUB-200-2011 (Welinder al. 2010), similarly we first exclude the very large models. Then we follow the standard procedures (Sermanet et al. 2014 Simonyan & Zisserman 2013) for fine- tuning pretraie-- models on LUB-200-2011
simplicity, we use the same training setup for all pre-trained models (learning rate 0.01, batch size 64, SGD optimizer with momentum 0.9, resize to 256 being the short edge, randomly cropping images to the size of 224x224), and obtain 85 models that are well trained. Different syyper-parameten may help to improve the perfor- mance of some specific networks, but for the same reason the large number of available models, choose to not search for better hyppe-paraaeete settings.
pretrained
Given the convergence over committee sizes (Figure 7. which shows that a committee of more than 15 models works good enough, thus we randomly choose around 20 models for Stanford Cars 196 (Krause et al. (003)), Oxford Flowers 102 (Nilsback & Zisserman 2008) and Foods 101 (Bossard al. 2014), following the same training procedure CUB-220-2011..
B.2 INTERPRETATION ALGORITHMS
To interpret a model, LIME Ribeiro et al. 2016) on vision tasks first performs a superpixel segmen- tation (Vedaldi & Soatto 2008) for an image, then generates samples by randomly masking some superpixels and computing the outputs through the model, and finally fits the model outputs with the set of superpixels as input by a linear regression model. The linear weights then presents the feature importance in the superpixel level as the interpretation result.
The gradients of model output w.r.t. input can partly identify influential pixels, but due to the satura- tion of activation functions in the deep networks, the vanilla gradient is usually noisy. SmoothGrad (mmikov et al. 2017) reduces the visual noise by repeatedly adding small random noises the input so as to get list of corresponding gradients, which are averaged for the final interpretation result.
C RESNET FAMILY
We show the zoomed plot of ResNet family (whose name contains "ResNet" key word) in the ImageNet-LIME committee of 81 models in Figure 13 (a). Meanwhile, we also present the results
using ResNet family as committee in Figure 13 (b). These two subfigures have no large difference, which further confirms the consistency of ranking models in different committees.
D REFERENCES OF NETWORK STRUCTURES
Many structures deep neural networks have been evaluated in this paper. including AlexNet (Krizhevsky et al. 2012), ResNet (He et al.) 2016). ResNeXt (Xie et 2017), SEResNet (Hu al. 2018), ShuffleNet (Zhang et al. 2018b; Ma et al. 2018), MobileNet (Howard et al., (andllr
et al. 2018 Howard et al. 2019 VGG (Simonyan & Zisserman 2015), GoogleNet Szzgedy al.
2013). Inception (Szegedy et al.
2015), Xception (Chollet 2017). 7). DarkNet (Redmon et al. 2016
Redmon & Farhadi 2018), DenseNet (Huang et al. 2017). DPN (Chen al. 2017b), SqueezeNet
c assi Fi BM htg.a/a......a..ooeeemmmmmmmmmmm oamp:ppppee..tooooooaoooooooooon
19

---

Under review as a conference paper at ICLR 2021
Nesige
Rodeett
002
Zacheet5
0.20
0.0
Beckett AC3S
Zaokeett Artie
0.71
Reebent..v od
livbeett and
839 Stadadd AANO0AAA
031
650
000
(a) Based on Complete Committee
(b) Based on ResNet Family
similarity to the consensus of LIME ResNet family. The consensus of (a) is voted by the complete committee on ImageNet (81 models), while the consensus of (b) is voted by ResNet family (16 models).
Figure 13: Model performance
landola et al. 2016), EfficientNet (Tan & Le 2019), Res2Net (Gao et al. 2019). HRNet (Wang al. 2020b). Darts (Luu al. 2018), AcNet (Ding et 2019) and so on.
E COMPLETE PSEUDOCODE OF Consensus
Figure T shows an illustrative pipeline of our framework Consensus of evaluating the interpretability of models without the need of the ground truth of interpretation, and Algorithm 2 gives a complete process of Consensus in pseudocode.
F NUMERICAL REPORT OF MAIN PLOTS
Due to the large number of deep models evaluated, Figure 2 3 and grouped those that are of the same architecture. Here, report all of the corresponding numerical results in Table [
G VISUALIZATION OF COCO IMAGES
For further showing the effectiveness of Consensus, we visualize several random images from MS- COCO (Lin al. 2014), shown in Figure
20

---

Under review as a conference paper at ICLR 2021
Algorithm 2: Consensus Framework Pseudocode
function sim(a, b)
Input Two vectors or tensors.
Oufput: scalar as the similarity between and b with appropriate normalization approaches. cosine simil larity for LIME interpretations and Radial Basis Function, exp(- (pp(-((le bll/o)?).
/+ This function
function aggregate(L)
Input L, collection of interpretations of m models for one given data sample.
Output: the consensus among m model for the interpretation of the given data sample.
This function returns the quasi quasi-ground-E -ground-tuth of the interpretation for the given sarple, basically it is equivalent to a nermalization-averagg
procedure in this paper.
Specifically, 1 E Lu for LIME, and C₁ 1 Σ. min-min(L man man(he)
for SmocthGrad.
function interpret(A, M, d)
Input An interpretation algorithm A, trained model M and data sample d E R", where R2 is the feature domain and the dimension may vary with the sample d.
Output: R., where the elements indicate the importances of input features.
This sunction is an implement ateentaiionnn
typical interpretation
algori like 1IIM (number of superpi xels as p) smoothGrad (number of pixels as p] or others.
function Consensus(D.
Input : dataset D containing n examples (di)se,,... and an interpretation algorithm A.
Output: s E IR",, where each element indicates the interpretability of each model M, M. /+ Step 1: Committee Formation with Deep Models .M
Prepare M containing models {M, 3-1,",, ie. the committee of deep models. matrix for storing the interpretability scores of m models on n data sample. 10 for in 1........ do
S =nros((, m) // Initialize an empty
L zeros(m,,p)
for 1,... do
L, interpret(A.. M, d.) end
14
/+ Step 2: Committee Voting for Consensus Achievement at di
*/
aggregate(L)) // Re consensus quasi-ground-tuuhh Step 3: Consensus-based Interpretability Evaluation d. */ for in 1.... m
16
17
S. sim L, c) // the score of M, at d;
end
19 end
20 for in 1.... m do
8, average 8 average score for each model over samples
end
23 return s
21

---

Under review as a conference paper at ICLR 2021
Table 1: Numerical report of model performance and similarity to the consensus using LIME and
SmoothGrad over 81 models on ImageNet in sub-table(a) corresponding to Figure 4 (a, b, c), and over 85 models on 200-2011 in sub-table(b) corresponding to Figure and 4 (d. e, f).
(n)onn Imageeher
(1) CUB-20000000
interpret-
ability
interpeet-
interpeet- ability scones of Consensus (LIME)
interpeet- ability
mAP between of segmentation and LIME inta pretation
mAP betwoon g4, of Aegmentttionn and SmoothKirad intepretation
perl.
scores of
Consensus (LIME)
ability
seares of
Consemuus (Smooth Grad)
se00es of
Consenuus (Smooth (lod))
AleeNer
0:525
0.594
0.0214
AlexNet
0.507
0.5%
0.0275
0.343
0:571
AM
0.752
AmsDLL.M
0.728
0.781
0.0271
0.594
0.693
AmaDL16
0.776
0.741
AmslDL_6M
0.754
0S81
0.0402
0160/5
0.7411
DPN107
0.798
0.828
0.031
DPN107
0.830
0.867
0.0525
0.630
0.780
DENI31
0.802
0.833
00034
DPN131
0.800
0.868
0,0498
0.643
0.785
DPNS
0.766
0.849
0.2997
DPN6
0.795
(1842)
0.0415
0.631
0.710
DPN92
0.775
0.845
0.0357
DPN92
0.806
0,872
0.0510
0.626
0.784
DPN99
0.798
0.837
00354
DPNN8
0.115
11877
0.0526
0.628
0.753
21
0.755
0.890
0.0376
0.782
0.850
0.0485
0.604
0.443
DenseNet161
0.781
0.849
0.0883
DenseNet121
0.771
0.848
0.0503
0.585
0.771
0.765
0.8876
DenseNat661
0,813
0.873
0,0542
0.641
DenseNa2201
0.779
0.3855
66
0,792
00513
DenseNec264
0.761
DamsSN:2201
0.805
0.858
0.0448
0.16
0.795
0.754
0.727
DameNNr264
0.789
0.868
0.05400
0.28
0.7%8
BllciemNNelB..5mml
0.708
0.729
0.3322
EfficienNNeDDO
0.765
0.805
0,0500
0.594
0 0.769
GoogieNe:
0.726
0.734
00260
0.737
11.05
0,0110
0.589
0.738
HRNei W18 C
0.766
0.854
0.388
0.775
0.805
07446
0.991
0.755
W30.C
0.776
0.832
0.378
FilicinNN1122
0.787
0.819
07/66
0.585
0.764
W33.C
0.781
0.845
0.0376
EfficiemNNel3333
0.791
0.812
12021
0.582
0.771
HRNet W-40.0 C
0.775
0.821
00366
EmmcicatNet54
0.792
11.29
0.612
0.766
HRNei
0.782
0.817
0.3688
FMficimNNaR5
0.774
0.808
00411
0.591
0768
HRRNel Was C
0.794
0.807
0.0399
HRRNeL W1S
0.754
0,831
19000
19000
0.992
0.336
HRNeL W64.C
0.784
0.799
0.3344
HRNeI
0.720
01832
1.832
00005
0.595
MobileNetV
0.711
HRNe,W
0.785
0.816
0.586
0.750
MobilsNecV1
0.513
0.750
0.763
MobileNaaV1.00..
0.641
0.751
HRNe, W14,C
0.788
0.80
0.522
MobileNaV/Lx0.2
0.697
0.768
0.0997
TIRNeL WAS.C
0.796
0.838
00082
0.581 1
0.757
MobileNet22
0.742
0812
0U342
HRNet W64 C
0.791
0,0085
0.766
MobleNaa22.0025
0.534
0.650
0.221
Inception44
0.745
0.797
0.4435
0.992
0.728
MobileNaV2..dd .00.5
0.642
0.768
0.0200
MobileNetV1
0,741
0.524
0,0115
0.568
0.716
MobileNaV2.s1.5
MobileNaV2.101222
0.709
0.737
0.795
0.841
0,0336
0.802
MobileNaiVL.022
MobileNaVILs..3
0.655
0.557
0.753
B.076
0,0325
0,0288
0
0.448 1
0.937
0.634
0672
MobileNetV2 x2 0
0.746
0,840
MobileNatV1 xIl
0.688
0,808
0.008
0369
0.701
Res2Ner 101 vd.26__0_44
0,0255
MobileNerV2
0.737
Res2Ne:50. H4w.
0.781
MobileNaV2xx..22
0.511
0,670
0/051
0.457
0.597
Res2Nec50 26 4a
0.780
MobileNetV2.00..
0.665
1.753
00000
0543
0,61 1
Rex2Nit S0_od_2fwdd_
0.70
0.828
0,332
MohileNarV22x0.77
0.15
0814
60009
03777
0666
ResNeXt10) 32x4d
0.784
0.841
000711
MobileNatV2.x11
0.756
0.355
00010
1990
0.719
ResNeXt101 ad. 54.4d
ResheXr1B1. ad. 32x4d
0.795
0.784
0.821
0,830
0.0366
00347
ResINet101 .d. 26 As
MobileNarV2_x2.00
0.781
0.799
0.85
11851
0/470
0693
0465
0.705
0.756
ResNeXi152. 22xdd
0.782
0.842
00377
Res2Nec10,14w..5
0.789
11.28 11826
00000
0.387
0.765
RcaNeXi152.66 64640
ReaNeXi152, vd. 32x4d
0.790
0.792
0.787
0.807
0.828
0,814
0.383
0.925
0.0325
Rec2Nr300 26w 46
Res2Nert0
RecNeX10i__a0 33x48
2d.20 44
0818
0.753
0.768
1.840
0.877
8/466
80500
0
0.000
0.639
07611
0.749
0.782
0.798
ResNcX150 31x4d
0.765
0,849
0.585
ResNcX1101 32dad wel
0.768
0.831
ResNeXr50
0.784
0.3899
ReaNcXil01 3dv44
0816
0.867 19677
06114
ResNeXi50. vd_ 32cld
0.790
0.0366
ResNeXi1D) .d.66x6d
0.844
0,871
0,42
0.77%
ResNeXt50 64s4d
0.792
0.839
00360
ResNeXi152
32x4d
0315
11872
1
0
1900
ResNett01
0.699
0.847
0.377
ResNeXi1 52.6444
0334
0875 0
ResNet10L. vd
0.788
0.810
00023
ResNeXi152 vd 32xld
0.820
0,872
0.600
0
Resher152
0.776
0.846
00374
ResNeXi152, vd. 64x4d
0822
11852
01419
0600
0.764
0.801
0.791
0.0388
1eNNoX150_12v44
0.809
0.856
0.060
0.600
0 785
ResNet18
0.715
0.816
00342
ResNeX150 64xid
0.14
0.851
0,262
0.788
ReaNer1R_yd
0.730
0.80T
0.334
ReaNeX150 vd 324d
0.806
0.874
00508
0
0766
Res
0.793
0,790
0.0300
RecNeXi50.vd..o 5ttkel
0.820
0.890
0.0544
0691
0.765
ResNei34
0.739
0.826
0.863
ResNerto1
0.784
11878
0,211 00311
0620
ReoNoi54_yd
0.757
0,0299
ReaNattl_
0.813
000000
00400
0226
0.66
ResNei50
0.763
0.858
ResNet152
0.799
0,859
0
1000
0.775
ACNNer
0.780
0.866
0.086
ResNe1122.d
0.777
0990
0.774
ResNe
0.778
0.817
0,0300
ResNet18
0.726
0.794
0946
0.735
ResNet50 vd
0.778
0.831
0U34]
ResNet18 vd
0.754
0,846
0.710
54N
08803
0.807
0.915
ResNer200,sd
0813
0.861
00002
0940
0.775
SE. R..RRNNXXI001552 .3vld
0.781
0.818
0.0325
ResNetM
0.758
0.812
09460
03569
0.756
SE ResNeXiSu 3224d
0.775
8810
00921
ResNec14 yd
0.771
11833
000035
0.570
0.231
SE RosNeXX10 vd 30vld
0.797
0.819
06942
RecNasO
0.776
0.878
80511
1
077
SC_ResNetB8.vd
0.743
0.810
0.342
ResNeE50
0.782
0,870
00181
1
0619
0.337
SE NE.RoNNar4..d
0.766
0.789
0.330
ReaNecoo.rd
0.795
1.876
00461
0.334
SF. S...RoNNe55.. od
0,0322
ReNNeX 32cld
0.795
00452
0.750
ShumaNNa12
0.325
R. X:50 32dl
0.706
0.621
0/418
0.576
0.727
ShutBeeNeV22.c
0.507
0.636
ResNeX:50 v2vld
0.799
0,861
0.4199
0617
0.729
0.547
0.651
0.277
B.002
0.0966
0.550
0.73
StummeNeV2200..
0.611
0,710
0,250
ST_ReeNet4..o
0.754
0.801
0,4500
0.544
0731
ShutthNeeV22 x1
0.689
0,778
00295
SE ResNee50 vo.
0.771
0,870
0,0446
0000
0.752
ShumaNaVVi.il.5
0712
0.807
0.386
StuMaNaaV22
0.69
0.817
0.0355
0:571
06688
ShufhheaaV2_22.00
0.738
0.816
0.0317
SbufBeNalV2..00
0.519
0.687
0.0263
0.148
0.563
S ID
0.602
0.732
0U253
StuttaNetV2 xd1 33
0.530
1.666
010294
0.465
0.22
SpuezeNet111
0.13
0.762
06242
0660
0.751
0.0007
0.500
0634
VGG11
0.694
0.801
0.0291
SauffeNNt122
0.695
0.758
0.054
0.564
0667
VGG11
0.697
0.804
StumkNerV2211.3
0.728
11815
0,3371
0.564
VOGI6
0.714
0,0305
StuffNNaV22222.00
0,402
0.574
YOG19
0.722
0.821
0.0309
Xceptoo41
0.801
0.833
0.0501
0.645
0.761
Xecprionll Jiooplab
0,775
0.753
0.0422
0.559
0.74
XceptionSS
0.801
0.837
0.0179
0.609
0.700
Xcepexut5 deeplab
0.747
11800
0,0455
0.586
0.728
Xceprisn11
0.775
0.846
0.6499
0.13
0.760
Consensus
0.859
N/A
N/A
0,704
0.818
22

---

Under review as a conference paper at ICLR 2021
Imoge
22xed
corserous
Imaga
Ramkett:01 32sdd
comnamusc
Image
AutoOu (H
DenseNert61
Shuffeekeev2
Recket10)
consensue, reel (6:3)
mmgg
Modenhwaty2.co .1.
Bedis53
HUNLLW33,
Benhi00222 vi. 3t44e
comamaui gobll (8.57)
Figure 14: Visualization of images from the MS-COCO dataset (Lin et al. 2014) for showing the effectiveness of our framework Consensus, where the predicted label with probability is noted. Note that consensus is the Consensus aggregated interpretation.
23


Reviews:
[{"review_id": "yRSwz3SSqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " The manuscript proposes \"Consensus\" an evaluation method to measure the interpretability of a given deep model without the need of a dataset with annotated ground-truth concepts.\n \n The proposed method works by leveraging a set of pre-trained deep models (committee), and a reference model explanation method. The explanation method is use to produce explanations for every image/sample on a target dataset, and the aggregated explanation from every model in the committee is considered as \"approximated/quasi ground-truth\". The interpretability of a given model is defined as the mean similarity between all the explanations produced by the model in question and the quasi ground-truth (aggregated explanation) produced by the committee.\n \nStrong points\n- Experiments cover a good amount of deep models.\n- An ablation study is in place.\n\nWeak points\n- Terminology is ambiguous.\n- limited number of model explanation methods.\n\nOn the stronger side, the evaluation of the proposed method is relatively good, it includes an ablation study considering several relevant factors of the proposed method, e.g. committee size, and the effect of the members that are part of the committee.\nMoreover, it validates the proposed method considering a large amount of deep models. \n  \nOn the weaker size, there are several aspects that I feel need to be addressed:  \n\nI had some difficulties while reading the manuscript. The use of specific terminology is quite ambiguous at times. In several parts of the manuscript there are terms like \"variety of interpretability of models\", \"interpretation result\", \"interpretability evaluation\" that are not standard and hinder the content to a good extent. I would suggest very early on the manuscript defining these terms so that their meaning is clear much later in the body of the paper. \n\nVery related to the previous point, the tasks of model interpretation (identifying the information/features encoded internally in a given model) and model explanation (justifying the predictions made by a given model on a given sample) seem to be used interchangeably. In my opinion, this is another strong source of confusion.\nThis should be addressed when discussing methods like LIME and SmoothGrad which are model explanation methods. \nMoreover, this seems to actually point to the definition of one of the ambiguous terms mentioned in the previous point. It seems that \"interpretation result\"  actually refers to the explanation produced by a given sample.\n\nThe main characteristic of the proposed method is its independence on annotated ground-truth concepts. However, as admitted by the manuscript, there is still the requirement of having the data needed to train the models in the committee. This still sounds like a expensive requirement. \nIn addition, at this point it is not clear how the proposed consensus method would apply to a task/application (e.g. medical imaging) where data is scarce and deep models are not available in abundance. Discussing this direction would strengthen the manuscript.\n\nIn the comparison against the Network Dissection method (Sec. 3.3), it is stated that the rankings produced by the proposed method are quite similar to those of Network Dissection, but with differences between the ResNet152 and DenseNet161 models. Is it possible to get an insight on the source of this difference? Is it possible to know why the proposed method prefers one over the other? \n\nWhen assessing the effect of the size of the committee (Sec. 6), what if just few specific models are responsible for the observed performance? then the number of randomly selected models might not be so relevant as the selection of these few models. Perhaps an incremental approach when models are gradually integrated in the committee should be analysed. \nThis setting, of few models determining the performance, seems to be suggested by the change in mean and standard deviation on performance in Fig.7. For committees with few models (4-9) the large standard deviation pushes performance (of some random committees) closer to the upper bound (0.70). The consideration of committees with larger sizes, and thus the likely potential inclusion of these relevant models, just makes the mean performance converge faster. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mq6Hpxy_FNZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper seems to introduce a very important model to evaluate intepretatbility of neural networks.\nHowever, the paper is not extremely clear and intepretable.\nThe main idea is to pool together interpretations coming from different systems and then selecting the best interpretation by voting. The procedure and the model is described in a single section, that is, section 2. It is extremely obscure the way Consensus operates. A running example can be very relevant in this section. Without this running example, it is difficult to attract readers into your paper.\nFor example, one between Figure 4 and Figure 5 could be this running example? Can you use it to explain your model?", "extra_scores": {}, "extra_text": {}}, {"review_id": "SGxV-UCuU-b", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with explainable machine learning in the supervised setting and especially tackles the case where no ground truth data for evaluating the generate explanations, such as bounding boxes for objects, is available. The proposed \"Concensus\" approach retrains established architectures on the target dataset and averages their generated explanations from out-of-the-box explainers, such as LIME or SmoothGrad. The approach is evaluated for image classification on ImageNet and CUB-200-2011 by comparing the averaged explanations of the committee models when using LIME and SmoothGrad with the ground truth bounding box and segmentation, respectively. Minor evaluations are included for datasets Stanford Cars 196, Oxford Flowers 102  and Foods 101. The results show that the averaged explanations strongly correlate with the mean average precision with respect to the label distances.\n\nThe approach tackles the interesting setting of evaluating explainable ML approaches - here without labelled objects for computer vison tasks. In general, I find it interesting to evaluate such kind of ensemble-based approach for generated explanations, which is a common technique for adding model robustness or approximating uncertainty, and is closely related to bootstrap aggregation. To this end, the authors provide a quite large variety of experiments for different established vision benchmark datasets, including ablation studies on varying the committee size or randomly generated committees. However, I am left with several open questions for the approach, which I summarize below. In general, however, I am missing the added value of the approach with respect to the evaluation of explainable ML, as it remains unclear to me (based on the approach description and conducted evaluation) if the proposed tool can actually replace empirical evaluations in research or, even more important (and possibly intended by the authors) could support end-users in evaluating their models via out-of-the-box explainers.  \n\nI am wondering why the networks always need to be trained from scratch for a new task/dataset? Wouldn't the approach work for standard transfer learning? Here, it would be interesting to explore smaller, challenging datasets as well. \n\nAlso, why is the use of \"known models\" for buliding the committee emphasized? I am aware of prior works which argue that activation-based explainers are bias from the underlying architecture, but the training procedure should have significant impact on the \"explanability\" (at least as measured by approaches such as SmoothGrad) as well. Wouldn't it make sense to vary the training (or additional fine-tuning) parameters/used augmentation strategies as well?\n\nAs the paper highlights the unlabelled evaluation setting, I am missing related work in the field of explainable / interpretable ML, especially on counterfactuals. Approaches such as [1,2] generate such counterfactual explanations by altering the image towards other, potentially risky classes. Could you comment on the relation of the proposed approach to these kinds of works? \n\nFor the evaluation, the authors use labelled datasets for empirically validating their claims. I am wondering - and this might be a general point for such an evaluation - why/how the explanation component can be evaluated independently of the model, as imperfect explanations might uncover model weaknesses. Is your evaluation affected by this? In addition, as mentioned before with respect to the applicability of Consensus to transfer learning, I am wondering if the evaluation should cover cases where the network might overfit or misperform. Would your approach potentially rule out possibly interesting misbehaviors of a network in such a case? Lastly, could you provide design choices for the evaluation parameters, such as committee size or number of sampled networks?\n\nReferences:\n[1] Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D. and Lee, S., 2019, May. Counterfactual Visual Explanations. In International Conference on Machine Learning (pp. 2376-2384). \n[2] Liu, S., Kailkhura, B., Loveland, D. and Han, Y., 2019, November. Generative Counterfactual Introspection for Explainable Deep Learning. In 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP) (pp. 1-5). IEEE.\n\n+++ Updates after author response +++\nI want to thank the authors for their answers as well as their attempts to improve the manuscript. I have read the other reviewers' comments and the updated version of the paper. The latter improves on the clarity of the approach in terms of the formal presentation of the approach, but a concise problem definition is still missing in my opinion. \n\nI feel that the paper still needs to additionally quantify who many labels are saved compared to classic cross-validation, as the models still need to be trained from scratch or fine-tuned. Here, it is important to establish when the method is actually (guaranteed to be) sufficiently concise, such that it can be used in practice. I therefore keep my tendency to reject the current version of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AiBQhFO3rje", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel method, called Consensus, using an ensemble of deep learning architectures, to study the interpretability of models, when ground truth of interpretations is not available.\n\nThe high-level idea of the paper is promising, i.e. evaluating interpretability without the need for human-labelled groundtruth, which is tedious and expensive to collect, and for this task, can be subjective. \nThe proposed method consists of three stages: forming a committee of deep models, aggregating the results in “quasi ground-truth”, and ranking models of the committee based on the similarity with the quasi ground-truth.\nHowever, it is not clear from the paper how the “committee voting” is performed (is it averaging, majority voting), and what does the method bring in addition to just an ensemble of multiple CNN architectures.\n\nGreat points of the paper:\n- using pretrained models to overcome the lack of ground-truth (which would be subjective and costly to collect).\n- comparison of the consensus results with several architectures, and consensus vs ground truth (via mAP).\n\nPoints on which the paper can be improved:\n1. Clarity of description of the method \n- there is no clear formal definition of how the committee voting works; adding formulas would improve readability.\n- no explanation on how RBF is used to address the dimensionality differences when comparing scores; \n- it is difficult to understand from the paper how the numbers (interpretability scores) are obtained (Sec. 3.3.); please add equations and reference them in the text - especially for correlation score and aggregating scores.\n- please clarify how the significance tests were performed\n- Since the figures are cluttered, it might be worth considering reporting the per CNN scores in a table (few models in the main paper) and the rest in the appendix. \n\n2. Experimental comparisons with other methods \n- the proposed method compares on object domain only, ImageNet and CUB200-2001 (birds). The method compares with Network Dissection (Bau et al, 2017), but no results on Broden dataset are reported. This dataset is particularly interesting, as it is unifying several datasets, spread across multiple domains (objects, scenes, objet parts, textures and materials).\n- Why is it necessary to have the models trained from scratch? (Sec. 2, committee). What would change if the models would be just finetuned?\n- What does the current method bring in addition to LIME and SmoothGrad aggregated over a large number of architectures?\nDatasets as MS-COCO / LVIS provide ground truth labels for object segmentation, which are more diverse than CUB / ImageNet.\n\n3. Style / visuals:\n- axis labels and text on the plots are too small;  pick a smaller number of models, the figures are too cluttered. The full comparison could be in tables, in appendix / supplementary material.\n- “significance tests” instead of “significant tests”.\n- are the 4 decimals in the Pearson coefficient necessary? (Fig. 1 caption) \n- please report the exact p-value.\n- Fig. 5 - please overlay the contour of ground truth over the other visualizations.\n\nThe overall idea is simple - ensemble of CNN architectures, to generate quasi-ground-truth for interpretability. Not clear what \"interpretability\" is -- and how this would be different from object segmentation masks.\nThe paper would greatly benefit from a more formal explanation of how the models are scored, and how the consensus is computed.\nIt would add value to emphasize more how the proposed method could leverage existing trained models for datasets on which such labels are non-existent;\nIt would also prove the effectiveness of the method, if at least one more datapoint would be reported, on datasets with object labels (e.g. MS-COCO / LVIS).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all found that the Consensus method introduced seemed sensible and applauded the authors on their extensive experiments.  However, clearly they struggled to understand the paper well and asked for a clearer and more formal definition of the methods introduced.  Unfortunately, the highest scoring review was also the shortest and also indicated issues with clarity.  It seems like the authors have gone a long way to improve the notation, organization and clarity of the paper, but ultimately the reviewers didn't think it was ready for acceptance.  Hopefully the feedback from the reviewers will help to improve the paper for a future submission.
Decision: Reject