On Concept-Based Explanations in Deep Neural Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) have shown great success in numerous tasks ( Goodfellow et al. , 2016 ) , from understanding images ( Zoph et al. , 2017 ) to answering questions ( Devlin et al. , 2018 ) . Yet , in many scenarios their lack of explainability serves as a bottleneck against their real-world impact , especially in high-stake decisions such as in medicine , transportation , and finance , where such explanations help identify systematic failure cases , comply with regulations , and provide feedback to model builders . This has thus led to increasing interest in human-like explanations of DNNs . The most commonly-used methods to explain DNNs explain each prediction by quantifying the importance of each input feature ( Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ) . However , such explanations typically explain the behavior locally for each case , rather than globally explaining how the model makes its decisions . Also , input features ( such as the raw pixel values ) , and weights on them , are not necessarily the most effective explanations for human understanding . Instead , “ concept-based explanations ” characterize the global behavior of a DNN in a way understandable to humans , by explaining how DNNs use concepts in arriving at particular decisions . Such conceptbased thinking , by extracting similarities from numerous examples and grouping them systematically based on their resemblance , has been shown to play an essential role in human minds for making generalizations ( Armstrong et al. , 1983 ; Tenenbaum , 1999 ) . With a similar motivation , “ concepts ” can explain the decision-making rationale of DNNs and their generalizable knowledge . A few recent studies have thus focused on bringing such concept-based explainability to DNNs . Based on the common implicit assumption that the concepts should lie in certain linear subspaces of some intermediate DNN activations , they aim to find such concepts efficiently and relate them to data . These have ranged from supervised approaches ( Kim et al. , 2018 ; Zhou et al. , 2018 ) that obtain concept representations given human-labeled data on salient concepts , to purely unsupervised approaches that provide concept explanations automatically without human labeling , ranging from k-means clustering of DNN activations ( Ghorbani et al. , 2019 ) , to a self-interpretable Bayesian generative model ( Bouchacourt & Denoyer , 2019 ) . A key motivating question we ask in this paper is whether we could build on such unsupervised approaches to extract concepts , but where in addition to ensuring that the concepts are representative of the DNN activations , we would also like to ensure the additional facet that they are sufficiently predictive of the DNN function itself . This leads naturally to a crucial unanswered question in concept-based explanation , which is how to evaluate whether a set of concepts are sufficient for prediction . Previous concept-based explanations select concepts that are salient to a particular class ( Kim et al. , 2018 ) . However , selecting a set of salient concepts does not guarantee that these concepts are sufficient for prediction . The notion of explanations that are sufficient for prediction is also called the “ completeness ” of explanations ( Gilpin et al. , 2018 ) , which is acknowledged to be valuable for evaluating explanations ( Yang et al. , 2019 ) . In this work , we propose such a completeness metric for a given set of concept explanations . The completeness measurement can be applied to a set of concept vectors that lie in the span of some intermediate DNN layer activations , which is a general assumption in previous concept-based explanation works ( Kim et al. , 2018 ) . The core idea is that , by projecting the activations onto the span of concept vectors , we keep just that information that can be explained by the concepts , and discard the information that are orthogonal to all concepts . Thus , when projecting activations onto the span of concept activation vectors result in no loss in prediction accuracy , we can learn concepts that are “ complete ” ( i.e . sufficient for prediction ) . Interestingly , we show that under a stringent degeneracy condition on the DNNs , principal component analysis ( PCA ) on the DNN activations can be shown to maximize these concept completeness metrics . Of course such degeneracy assumptions likely do not hold in general , so that maximizing these completeness metrics could be viewed as a generalization of PCA that additionally takes the DNN model into account . However the resulting “ principal components ” are not guaranteed to be interpretable to humans . We thus build on the concept-interpretability principles proposed in Ghorbani et al . ( 2019 ) , and additionally consider carefully designed objectives that favors concepts that are more semantically meaningful to humans . A key facet of our approach is that it can work without any human supervision , which reduces the human labeling cost to provide explanations . After a set of highly-complete concepts are discovered , we use game-theoretic notions to aggregate over sets to define contextualized importance of a concept , which we call ConceptSHAP . ConceptSHAP is shown to be the only scoring method that satisfies a set of axioms , which accounts for the contribution of each concept to the completeness score . We also derive a class-specific version of ConceptSHAP that decomposes the ConceptSHAP score with respect to each class in the multi-class classification setting , which can be used to find concepts that contribute the most with respect to a specific class . To verify the effectiveness of our completeness-aware concept discovery method , we create a synthetic dataset where we can obtain the ground truth concepts and test whether existing methods can retrieve them . We find that our method is able to retrieve the ground truth concepts better than all compared methods . We also demonstrate examples from real-world language and vision datasets to show that our concept discovery algorithm provides additional insights on the behavior of the model . 2 COMPLETENESS OF CONCEPTS . Problem setting : We are given a set of n training examples x1 , x2 , ... , xn P Ri , corresponding labels y1 , y2 , ... , yn P Ro , and a DNN fpxq that is learned to map the labels ( with dimension o ) from given inputs ( with dimension i ) . We choose an intermediate layer of the DNN , and define the operation for generating the intermediate features from input as Φpxq P Rd and feed forwarding from the intermediate layer to logit layer as hp¨q , yielding the decomposition fpxq “ hpΦpxqq . We define the data matrix as X P Riˆn ; the corresponding feature matrix as ΦpXq P Rdˆn , and the corresponding prediction matrix as fpXq P Roˆn . Assume that there is a set of m concepts denoted by vectors c1 , c2 , ... , cm that represented linear directions in some activation space Φp¨q P Rd given by a concept discovery algorithm . We define the concept matrix as c “ rc1 c2 . . . cms . Next , we propose two mathematical definitions that capture the completeness of a given set of given concepts . Both definitions are based on the idea that completeness should quantify how sufficient a particular set of concepts are in explaining the model ’ s behavior . A low completeness score of a set of concepts indicates that the corresponding concepts do not capture the model behavior fully , and that the model bases its decision on factors other than the given concepts . We propose two metrics of completenss based on two different assumptions , as we discuss below . Assumption 1 : If the given set of concepts is complete , then using a projection of the intermediate features from input onto the feature subspace spanned by the concepts , concept space , would not deteriorate the model performance . We define the projection of some input embedding Φpxq onto the subspace spanned by v P Rdˆr as P pΦpxq , vq “ vpvJvq´1vJΦpxq . ( 1 ) We define the completeness metric ηp1q on a set of validation data with T data points as V “ tpx1 , y1q , ... , pxT , yT qu based on the assumption that projecting input features onto the span of a complete set of concepts should not reduce the model prediction performance . Definition 2.1 . Given a prediction model fpxq “ hpΦpxqq , a set of concept vectors c1 , ... , cm , and some loss metric L , we define the completeness score ηp1q as : ηp1qpc1 , ... , cmq “ R´ ř tx , yuPV LphpP pΦpxq , cqq , yq R´ ř tx , yuPV Lpfpxq , yq , ( 2 ) where R “ ř tx , yuPV Lphp0q , yq to ensure that ηp1qp0q “ 0 . We omit the dependency of hp¨q , Φp¨q , fp¨q , and Lp¨q of ηp¨q for notation simplicity . When ηp1qpc1 , ... , cmq is high , the network maintains a high accuracy even after projection , which supports that the set of discovered concepts hold sufficient information for prediction . Assumption 2 : The second assumption is that if we remove all useful concept information for a classification task , the model should fail to discriminate different classes . Thus , when all salient information is removed from the network , predictions scores for examples in class A won ’ t be much different from other examples in class A . We define the data matrix of validation set as as Xv “ rx1 x2 . . . xT s. To quantify how much the prediction score varies across data samples , we use the sample variance of the predictions : v̂arpfpXvqq “ Trp ˆcovpfpXvqqq “ TrppfpXvq ´ ÊrfpXvqsqpfpXvq ´ ÊrfpXvqsqJq , where ÊrfpXvqs “ 1T řT i “ 1 fpxiq , and Tr stands for the trace . Then , we define the second completeness metric following this assumption . Definition 2.2 . Given a prediction model fpxq “ hpΦpxqq , and a set of concept vectors c1 , c2 , ... , cm , we define the completeness score ηp2q as : ηp2qpc1 , ... , cmq “ 1´ v̂arphpΦpXvq ´ P pΦpXvq , cqqq v̂arphpΦpXvqqq , ( 3 ) Based on our assumption 2 , the variance of the prediction gets lower after useful concept information is removed from the data , yielding a high completeness score ηp2q . We now show that under degenerate assumptions , the top k PCA vectors of Φpxq maximize the completeness score for a set of concept vectors . Top PCA vectors are designed to capture as much information in data as possible , a set of concepts with high completeness score similarly preserve the necessary information in the data for the model to reach satisfactory predictions . Proposition 2.1 . When h is an isometry function that maps from pΦp¨q , } ¨ } F q Ñ pfp¨q , ? Lq , where L is the loss metric in equation 2 and fpxiq “ yi , @ pxi , yiq P V ( i.e . the loss is minimized ) , the first m PCA vectors maximizes ηp1q . Proposition 2.2 . When h is an isometry function that maps from pΦp¨q , } ¨ } F q Ñ pfp¨q , } ¨ } F q , and each dimension of Φpxq is uncorrelated with unit variance , the first m PCA vectors maximize ηp2q . We underline the two main differences between the concept vectors that maximize the completeness score and the PCA vectors . First , the propositions depend on degeneracy assumptions such as isometry of a DNN , which may not hold in practice . Therefore , the concepts that maximize the completeness score takes the prediction of the DNN into account , which can be seen as a generalization of the original PCA . Second , since the concept score only depends on the span of the set of concept vectors , any concept vectors whose span is equal to the span of the top PCA vectors also maximize the completeness score ( i.e . the set of vectors that maximize the completeness is not unique ) . Each PCA vectors are constrained to minimize the reconstruction error and being orthogonal to other PCA directions . On the other hand , the discovered concept vectors that maximize the completeness can be designed so that each concept is interpretable and semantically-meaningful to humans , which will be further explained in the next section .

Summary: The paper proposes metrics for evaluating concept based explanations in terms of ‘completeness’ -- characterized by (1) whether the set of presented concepts if sufficient to retain the predictive performance of the original model and (2) how is performance affected when all information useful to a complete set of concepts (as per (1)) is removed from features at a specific layer. Assuming concept vectors lie in linear sub-spaces of the activations of the network at a specific layer, the underlying assumption is that if a given set of ‘concept’ vectors is complete, then using a projection of the intermediate features from input onto the sub-space spanned by concepts should not result in reduced / affected predictive performance. Based on these characterizations, the paper proposes an objective to discovering complete and interpretable set of concepts given a candidate cluster of concepts. Furthermore, the paper proposes metrics to quantify the importance of each concept (using Shapley values) and per-class importance of concepts. The authors conduct experiments on toy data, image and text classification datasets and show that their proposed approach can discover concepts that are complete and interpretable.