Democratizing Evaluation of Deep Model Interpretability through Consensus

1 INTRODUCTION . Due to the over-parameterization nature ( Allen-Zhu et al. , 2019 ) , deep neural networks ( DNNs ) ( LeCun et al. , 2015 ) have been widely used to handle machine learning and artificial intelligence tasks , however it is often difficult to understand the prediction results of DNNs despite the very good performance . To interpret the DNN classifiers ’ behaviors , a number of interpretation tools ( Bau et al. , 2017 ; Ribeiro et al. , 2016 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Zhang et al. , 2019 ; Ahern et al. , 2019 ) have been proposed to recover or visualize the ways that DNNs make decisions . Preliminaries . For example , Network Dissection ( Bau et al. , 2017 ) uses a large computer vision dataset with a number of visual concepts identified/localized in every image . Given a convolutional neural network ( CNN ) model for interpretability evaluation , it recovers the visual features used by the model for the classification of every image via intermediate-layer feature maps , then matches the visual features with the labeled visual concepts to estimate the interpretability of the model as the intersection-over-union ( IoU ) between the activated feature maps and labeled locations of visual objects . Related tools that interpret CNNs through locating importation subregions of visual features in the feature maps have been proposed in ( Zhou et al. , 2016 ; Selvaraju et al. , 2020 ; Chattopadhay et al. , 2018 ; Wang et al. , 2020a ) . Apart from investigating the inside of complex deep networks , ( Ribeiro et al. , 2016 ; van der Linden et al. , 2019 ; Ahern et al. , 2019 ) proposed to use simple linear or tree-based models to surrogate the predictions made by the DNN model over the dataset through local or global approximations , so as to capture the variation of model outputs with the interpolation of inputs in feature spaces . Then , with the surrogate model , these methods interpret the DNN model as ways the model uses features for predictions , e.g. , ranking of feature importance , and compare the results with the ground truth labeled by human experts to evaluate interpretability . Besides the use of linear interpolation for surrogates , many algorithms , like SmoothGrad ( Smilkov et al. , 2017 ) , Integrated Gradients ( Sundararajan et al. , 2017 ) , DeepLIFT ( Shrikumar et al. , 2017 ) , and PatternNet ( Kindermans et al. , 2018 ) have been proposed to estimate the input feature importance as the way to interpret the models , so as to interpret the model predictions by highlighting the importation subregions in the input . In addition to the above methods , ( Zhang et al. , 2018a ; 2019 ) proposed to learn a graphical model to clarify the process of making decision at a semantic level . Note that obtaining the interpretation of a model is an algorithmic procedure to explain the model ( Samek et al. , 2017 ) . On the other hand , through the comparing the interpretation results with the human labeled ground truth , the interpretability evaluation aims at estimating the degree to which a human ( expert ) can consistently predict the model ’ s result ( Kim et al. , 2016 ; Doshi-Velez & Kim , 2017 ) . In summary , the ground truth of interpretation results ( usually labeled by human experts ) is indispensable to all above methods for interpretability evaluations and comparisons , no matter ways they interpret the models , e.g. , visual concepts detecting ( Bau et al. , 2017 ) , and feature importance ranking for either local ( Ribeiro et al. , 2016 ) or global ( Ahern et al. , 2019 ) interpretations . While the datasets with visual objects labeled/localized and/or the importance features ranked have offered in some specific areas , the unavailability of ground truths also limits the generalization of these methods to interpret brand new models on the new tasks/datasets ubiquitously . There is thus the need of a method being able to evaluate the interpretability of models on the datasets where the ground truth of interpretation results is not available . Our contributions . In this paper , we study the problem of evaluating the interpretability of DNN classifiers on the datasets without ground truth of interpretation results . The basic idea of Consensus is to leverage the interpretability of known models as reference to predict the interpretability of new models on new tasks/datasets . Especially , in terms of general purpose perception tasks , we have already obtained a number of reliable models with decent interpretability , such as ResNets , DenseNets and so on . With a new dataset , one could use interpretation tools ( Ribeiro et al. , 2016 ; Smilkov et al. , 2017 ) to obtain the interpretation results of these models , then aggregate interpretation results as the reference . Then for any model , one could evaluate interpretability of the model through comparing its interpretation results with the reference . Specifically , as illustrated in Figure 1 , we propose a novel framework named Consensus that uses a large number of known models as a committee for interpretability evaluation . Given any task/dataset , Consensus first obtains the interpretation results for every model in the committee using existing interpretation tools , e.g. , LIME ( Ribeiro et al. , 2016 ) , then aggregates the results from the entire committee and reaches the consensus of interpretations through voting . With the quasiground-truth , Consensus evaluates the interpretability of each model through matching its interpretation result and the approximated one , and ranks the matching scores of committee members , so as to pursue the absolute and relative interpretability evaluation results without the ground truth of interpretations labeled by human experts . More specifically , we make contributions as follows . • We study the problem of interpretability evaluation on the datasets without human labeled ground truth of interpretations . To the best of our knowledge , this work is the first to study the problem of evaluating the interpretability of DNNs while the ground truths of interpretation results are not available , by addressing the technical issues of voting and committees . • We design and implement Consensus , a novel interpretability evaluation framework that incorporates a wide range of alternating interpretation tools , such as LIME ( Ribeiro et al. , 2016 ) , SmoothGrad ( Smilkov et al. , 2017 ) , to interpret the model as the variation of outputs over the interpolations of inputs ( in feature spaces ) , from massive perspectives ( e.g. , local or global interpretation , tree-based or linear surrogate and so on ) , and carries out the interpretability evaluation through the voting based on interpretation results of models in the committee . • We carry out extensive experiments to validate Consensus on a wide range of models on new ubiquitous tasks/datasets that the ground truth is not available , and exploit the quantifiable metric of model interpretability to report the overall interpretability evaluation results ( Section 3 ) . Case studies ( Section 4 ) confirm the effectiveness of Consensus and show the closeness of Consensus based results to the ground truth of interpretations . Robustness analyses ( Section 5 ) further demonstrate the advantage of the committee that the factors including the use of basic interpretation algorithms , the types of networks in the committee , and the size of committee would have few affects on the Consensus based interpretability evaluation results . 2 CONSENSUS : A FRAMEWORK OF INTERPRETABILITY EVALUATION . In this section , we introduce our proposed framework , namely Consensus , which incorporates existing interpretations , such as LIME ( Ribeiro et al. , 2016 ) and SmoothGrad ( Smilkov et al. , 2017 ) , to enable DNN interpretability evaluation without the use of human labeled interpretation results as reference/ground truth . Specifically , Consensus generalizes a simple electoral system , and consists of three steps : ( 1 ) Committee Formation with Deep Models , ( 2 ) Committee Voting for Consensus Achievement1 , and ( 3 ) Consensus-based Interpretability Evaluation , as follow . Committee Formation with Deep Models . Given a number of deep neural networks ( which are well-known with decent performance on common perception tasks ) and a target task with a dataset ( potentially without ground truth of interpretation results ) , Consensus first trains the given neural networks ( from scratch or fine-tuned ) using the dataset . Then , Consensus forms the post-trained networks as a committee of models , noted as M , and considers the variety of interpretability of models in the committee that would establish the references for interpretability comparisons and evaluation . Note that while our research assumes the human labeled ground truth of interpretation results is not available in the given task/dataset for interpretability evaluation , the labels of samples are requested when handling classifications and regression tasks . Committee Voting for Consensus Achievement . With the committee of trained models and the target task/dataset for interpretation , Consensus first leverages an existing interpretation tool A , e.g. , A can be LIME ( Ribeiro et al. , 2016 ) or SmoothGrad ( Smilkov et al. , 2017 ) alternatively , to obtain the interpretation results of every model in the committee on every sample in the dataset . Given some sample di , we note the obtained interpretation results of all models as L. Then , Consensus proposes a voting procedure that aggregates L to achieve the consensus c as the quasiground-truth of the interpretation for the sample di . Specifically , ck = 1m ∑m i=1 L 2 ik ‖Li‖ for LIME and ck = 1 m ∑m i=1 Lik−min ( Li ) max ( Li ) −min ( Li ) for SmoothGrad . In summary , Consensus adopts a normalizationaveraging procedure to obtain the quasi-ground-truth of interpretations for the sample . To the end , the consensus has been achieved through obtaining the collections of quasi-ground-truth for every sample in the target dataset based on committee voting . Consensus-based Interpretability Evaluation . Given the quasi-ground-truths as the consensus of the whole committee , the proposed algorithm evaluates the interpretability of every model in the 1We are not intending to connect our work with the multi-agent research though we use the term “ consensus achievement ” . committee by considering the similarity between the interpretation result of each individual model and the consensus of the whole committee . Specifically , for the interpretations and consensus based on LIME , Consensus uses cosine similarity between the flattened vectors of interpretation of each model and the consensus . Then Consensus quantifies the interpretability of the model through the mean of similarity measures over all samples . For the results based on SmoothGrad ( visual feature importance in pixel levels ) , Consensus follows a similar procedure , where the proposed algorithm uses Radial Basis Function ( exp ( − 12 ( ||a− b||/σ ) 2 ) ) for the similarity measurement . We rank all models in the committee using their similarities to the consensus and consider the top/bottom models with good/bad interpretability in the committee . Algorithm 1 : Consensus Framework Pseudocode . The functions interpret ( ) , aggregate ( ) and sim ( ) are described in the main text and detained in Algorithm 2 in Appendix E. 1 function Consensus ( D , A ) Input : A dataset D containing n examples { di } i=1 , ··· , n and an interpretation algorithm A . Output : s ∈ Rm , where each element sj indicates the interpretability of each modelMj inM . / * Step 1 : Committee Formation with Deep Models M * / 2 PrepareM containing m models { Mj } j=1 , ··· , m , i.e. , the committee of deep models . 3 S = zeros ( n , m ) // Initialize an empty n×m matrix for storing the interpretability scores of m models on n data sample . 4 for i in 1 , · · · , n do 5 L = zeros ( m , pi ) 6 for j in 1 , · · · , m do 7 Lj = interpret ( A , Mj , di ) 8 end / * Step 2 : Committee Voting for Consensus Achievement at di * / 9 c = aggregate ( L ) // c ∈ Rpi , consensus as quasi-ground-truth / * Step 3 : Consensus-based Interpretability Evaluation at di * / 10 for j in 1 , · · · , m do 11 Sij = sim ( Lj , c ) // the score of Mj at di 12 end

Summary: The paper deals with explainable machine learning in the supervised setting and especially tackles the case where no ground truth data for evaluating the generate explanations, such as bounding boxes for objects, is available. The proposed "Concensus" approach retrains established architectures on the target dataset and averages their generated explanations from out-of-the-box explainers, such as LIME or SmoothGrad. The approach is evaluated for image classification on ImageNet and CUB-200-2011 by comparing the averaged explanations of the committee models when using LIME and SmoothGrad with the ground truth bounding box and segmentation, respectively. Minor evaluations are included for datasets Stanford Cars 196, Oxford Flowers 102  and Foods 101. The results show that the averaged explanations strongly correlate with the mean average precision with respect to the label distances.