Explaining Latent Representations with a Corpus of Examples

1 Introduction and related work . How can we make a machine learning model convincing ? If accuracy is undoubtedly necessary , it is rarely sufficient . As these models are used in critical areas such as medicine , finance and the criminal justice system , their black-box nature appears as a major issue [ 1 , 2 , 3 ] . With the necessity to address this problem , the landscape of explainable artificial intelligence ( XAI ) developed [ 4 , 5 ] . A first approach in XAI is to focus on white-box models that are interpretable by design . However , restricting to a class of inherently interpretable models often comes at the cost of lower prediction accuracy [ 6 ] . In this work , we rather focus on post-hoc explainability techniques . These methods aim at improving the interpretability of black-box models by complementing their predictions with various kinds of explanations . In this way , it is possible to understand the prediction of a model without sacrificing its prediction accuracy . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Feature importance explanations are undoubtedly the most widespread type of post-hoc explanations . Popular feature importance methods include SHAP [ 7 , 8 , 9 ] , LIME [ 10 ] , Integrated Gradients [ 11 ] , Contrastive Examples [ 12 ] and Masks [ 13 , 14 , 15 ] . These methods complement the model prediction for an input example with a score attributed to each input feature . This score reflects the importance of each feature for the model to issue its prediction . Knowing which features are important for a model prediction certainly provides more information on the model than the prediction by itself . However , these methods do not provide a reason as to why the model pays attention to these particular features . Another approach is to contextualize each model prediction with the help of relevant examples . In fact , recent works [ 16 ] have demonstrated that human subjects often find example-based explanations more insightful than feature importance explanations . Complementing the model ’ s predictions with relevant examples previously seen by the model is commonly known as Case-Based Reasoning ( CBR ) [ 17 , 18 , 19 ] . The implementations of CBR generally involve models that create a synthetic representation of the dataset , where examples with similar patterns are summarized by prototypes [ 20 , 21 , 22 ] . At inference time , these models relate new examples to one or several prototypes to issue a prediction . In this way , the patterns that are used by the model to issue a prediction are made explicit with the help of relevant prototypes . A limitation of this approach is the restricted model architecture . The aforementioned procedure requires to opt for a family of models that rely on prototypes to issue a prediction . This family of model might not always be the most suitable for the task at hand . This motivates the development of generic post-hoc methods that make few or no assumption on the model . The most common approach to provide example-based explanations for a wide variety of models mirrors feature importance methods . The idea is to complement the model prediction by attributing a score to each training example . This score reflects the importance of each training example for the model to issue its prediction . This score will typically be computed by simulating the effect of removing each training instance from the training set on the learned model [ 23 ] . Popular examples of such methods include Influence Functions [ 24 ] and Data-Shapley [ 25 , 26 ] . These methods offer the advantage of being flexible enough to be used with a wide variety of models . They produce scores that describe what the model could have predicted if some examples were absent from the training set . This is very interesting in a data valuation perspective . However , in an explanation perspective , it is not clear how to reconstruct the model predictions with these importance scores . So far , we have only discussed works that provide explanations of a model output , which is the tip of the iceberg . Modern machine learning models involve many convoluted transformations to deduce the output from an input . These transformations are expressed in terms of intermediate variables that are often called latent variables . Some treatment of these latent variables is necessary if we want to provide explanations that take the model complexity into account . This motivates several works that push the explainability task beyond the realm of model outputs . Among the most noticeable contributions in this endeavour , we cite Concept Activation Vectors that create a dictionary between human friendly concepts ( such as the presence of stripes in an image ) and their representation in terms of latent vectors [ 27 ] . Another interesting contribution is the Deep k-Nearest Neighbors model that contextualizes the prediction for an example with its Nearest Neighbours in the space of latent variables , the latent space [ 28 ] . An alternative exploration of the latent space is offered by the representer theorem that allows , under restrictive assumptions , to use latent vectors to decompose a model ’ s prediction in terms of its training examples [ 29 ] . Contribution In this work , we introduce a novel approach called SimplEx that lies at the crossroad of the above research directions . SimplEx outputs post-hoc explanations in the form of Figure 1 , where the model ’ s prediction and latent representation for a test example is approximated as a mixture of examples extracted from a corpus of examples . In each case , SimplEx highlights the role played by each feature of each corpus example in the latent space decomposition . SimplEx centralizes many functionalities that , to the best of our knowledge , constitute a leap forward from the previous state of the art . ( 1 ) SimplEx gives the user freedom to choose the corpus of examples whom with the model ’ s predictions are decomposed . Unlike previous methods such as the representer theorem , there is no need for this corpus of examples to be equal to the model ’ s training set . This is particularly interesting for two reasons : ( a ) the training set of a model is not always accessible ( b ) the user might want explanations in terms of examples that make sense for them . For instance , a doctor might want to understand the predictions of a risk model in terms of patients they know . ( 2 ) The decompositions of SimplEx are valid , both in latent and output space . We show that , in both cases , the corpus mixtures discovered by SimplEx offer significantly more precision and robustness than previous methods such as Deep k-Nearest Neighbors and the representer theorem . ( 3 ) SimplEx details the role played by each feature in the corpus mixture . This is done by introducing Integrated Jacobians , a generalization of Integrated Gradients that makes the contribution of each corpus feature explicit in the latent space decomposition . This creates a bridge between two research directions that have mostly developed independently : feature importance and example-based explanations [ 19 , 30 ] . In Section 3 of the supplementary material , we report a user-study involving 10 clinicians . This study supports the significance of our contribution . 2 SimplEx . In this section , we formulate our method rigorously . Our purpose is to explain the black-box prediction for an unseen test example with the help of a set of known examples that we call the corpus . We start with a clear statement of the family of black-boxes for which our method applies . Then , we detail how the set of corpus examples can be used to decompose a black-box representation for the unseen example . Finally , we show that the corpus decomposition can offer explanations at the feature level . 2.1 Preliminaries . Let X ⊆ RdX be an input ( or feature ) space and Y ⊆ RdY be an output ( or label ) space , where dX and dY are respectively the dimension of the input and the output space . Our task is to explain individual predictions of a given black-box f : X → Y . In order to build our explainability method , we need to make an assumption on the family of black-boxes that we wish to interpret . Assumption 2.1 ( Black-box Restriction ) . We restrict to black-boxes f : X → Y that can be decomposed as f = l ◦ g , where g : X → H maps an input x ∈ X to a latent vector h = g ( x ) ∈ H and l : H → Y linearly maps1 a latent vector h ∈ H to an output y = l ( h ) = Ah ∈ Y . In the following , we call H ⊆ RdH the latent space . Typically , this space has higher dimension than the output space dH > dY . Remark 2.1 . In the context of deep-learning , this assumption requires that the last hidden layer maps linearly to the output . While it is often the case , it is crucial in the following since we will use the fact that linear combinations in latent space correspond to linear combinations in output space . Our purpose is to gain insights on the structure of the latent space . Remark 2.2 . This assumption is compatible with regression and classification models , we just need to clarify what we mean by output in the case of classification . If f is a classification black-box that predicts the probabilities for each class , it will typically take the form in Assumption 2.1 up to a normalizing map φ ( typically a softmax ) : f = φ ◦ l ◦ g. In this case , we ignore2 the normalizing map φ and define the output to be y = ( l ◦ g ) ( x ) . Our explanations for f rely on a set of examples that we call the corpus . These examples will typically ( but not necessarily ) be a representative subset of the black-box training set . The corpus set has to be understood as a set of reference examples that we want to use as building blocks to interpret unseen examples . In order to index these examples , it will be useful to denote by [ n1 : n2 ] the set of natural numbers between the natural numbers n1 and n2 with n1 < n2 . Further , we denote [ n ] = [ 1 : n ] the set of natural numbers between 1 and n ≥ 1 . The corpus of examples is a set C = { xc | c ∈ [ C ] } containing C ∈ N∗ examples xc ∈ X . In the following , superscripts are labels for examples and subscripts are labels for vector components . In this way , xci has to be understood as the component i of corpus example c. 1The map can in fact be affine . In the following , we omit the bias term b ∈ Y that can be reabsorbed in g. 2There is no loss of information as the output allows us to reconstruct class probabilities p via p = φ ( y ) . 2.2 A corpus of examples to explain a latent representation . Our purpose is to understand a prediction f ( x ) for an unseen test example x with the help of the corpus . How can we decompose the prediction f ( x ) in terms of corpus predictions f ( xc ) ? A naive attempt would be to express x as a mixture of inputs from the corpus C : x = ∑Cc=1 wcxc with weights wc ∈ [ 0 , 1 ] that sum to one ∑Cc=1 wc = 1 . The weakness of this approach is that the signification of the mixture weights is not conserved if the black-box f is not a linear map : f ( ∑C c=1 w cxc ) 6=∑Cc=1 wcf ( xc ) . Fortunately , Assumption 2.1 offers us a better vector space to perform a corpus decomposition of the unseen example x . We first note that the map g induces a latent representation of the corpus g ( C ) = { hc = g ( xc ) | xc ∈ C } ⊂ H. Similarly , x has a latent representation h = g ( x ) ∈ H. Following the above line of reasoning , we could therefore perform a corpus decomposition in latent space h = ∑C c=1 w chc . Now , by using the linearity of l , we can compute the black-box output of this mixture in latent space : l ( ∑C c=1 w chc ) = ∑C c=1 w cl ( hc ) . In this case , the weights that are used to decompose the latent representation h in terms of the latent representation of the corpus g ( C ) also reflect the way in which the black-box prediction f ( x ) can be decomposed in terms of the corpus outputs f ( C ) . This hints that the latent spaceH is endowed with the appropriate geometry to make corpus decompositions . More formally , we think in terms of the convex hull spanned by the corpus . Definition 2.1 ( Corpus Hull ) . The corpus convex hull spanned by a corpus C with latent representation g ( C ) = { hc = g ( xc ) | xc ∈ C } ⊂ H is the convex set CH ( C ) = { C∑ c=1 wchc ∣∣∣∣∣ wc ∈ [ 0 , 1 ] ∀c ∈ [ C ] ∧ C∑ c=1 wc = 1 } . Remark 2.3 . This is the set of latent vectors that are a mixture of the corpus latent vectors . At this stage , it is important to notice that an exact corpus decomposition is not possible if h 6∈ CH ( C ) . In such a case , the best we can do is to find the element ĥ ∈ CH ( C ) that best approximates h. IfH is endowed with a norm ‖ · ‖H , this corresponds to the convex optimization problem ĥ = argmin h̃ ∈ CH ( C ) ‖ h− h̃ ‖H . ( 1 ) By definition , the corpus representation ĥ of h can be expanded3 as a mixture of elements from g ( C ) : ĥ =∑Cc=1 wchc . The weight can naturally be interpreted as a measure of importance in the reconstruction of h with the corpus . Clearly , wc ≈ 0 for some c ∈ [ C ] indicates that hc does not play a significant role in the corpus representation ĥ of h. On the other hand , wc ≈ 1 indicates that hc generates the corpus representation ĥ by itself . At this stage , a natural question arises : how can we know if the corpus approximation ĥ is a good approximation for h ? The answer is given by the residual vector h− ĥ that measures the shift between the latent representation h = g ( x ) and the corpus hull CH ( C ) . It is natural to use this residual vector to detect examples that can not be explained with the selected corpus of examples C. Definition 2.2 ( Corpus Residual ) . The corpus residual associated to a latent vector h ∈ H and its corpus representation ĥ ∈ CH ( C ) solving ( 1 ) is the quantity rC ( h ) =‖ h− ĥ ‖H= min h̃ ∈ CH ( C ) ‖ h− h̃ ‖H . gx1 x2 x3 h1 h2 h3x h ĥ Figure 2 : Corpus convex hull and residual . In Section 1.1 of the supplementary material , we show that the corpus residual also controls the quality of the corpus approximation in output space Y . All the corpus-related quantities that we have introduced so far are summarized visually in Figure 2 . Note that this Figure is a simplification of the reality as C will typically be larger than 3 and dX , dH will typically be higher than 2 . We are now endowed with a rigorous way to decompose a test example in terms of corpus examples in latent space . In the next section , we detail how to pull-back this decomposition to input space . 3Note that this decomposition might not be unique , more details in Section 1 of the supplementary material .

Summary: In this paper, the authors propose to explain the behavior of black-box neural networks by approximating a top-level hidden layer (no non-linearities afterwards, except possibly softmax) as a linear combination of the hidden layers obtained from different examples. These examples may come from the training data, but it is not required. The contribution of each input feature is also evaluated using an extension of integrated gradient across multiple dimensions.