EDUCE: Explaining model Decision through Unsupervised Concepts Extraction

1 INTRODUCTION . While deep learning models are powerful tools to perform a large variety of tasks , their predictive process often remains obscure . Understanding their behavior becomes crucial . This is particularly true with text data , where predicting without justifications has limited applicability . There has been a recent focus on trying to make deep models more interpretable , see for example ( Ribeiro et al. , 2016 ; Bach et al. , 2015 ; Shrikumar et al. , 2017 ; Simonyan et al. , 2014 ; Sundararajan et al. , 2017 ) . Specifically , methods have been recently proposed to provide such explanations simultaneously with the prediction . For example , ( Lei et al. , 2016 ; Yu et al. , 2019 ; Bastings et al. , 2019 ) select subsets of words in the input text that can account for the model ’ s prediction ( called rationales ) . In this work , we propose a model that provides an explanation based on the absence or presence “ concepts ” that are automatically discovered in the texts . Suppose we ask a user to tell what is the category ( or class ) of the top text of Figure 1 ( text x ) . She detects that the words the government said relate to a specific concept that is present in x , a concept she also detect in the words made official what in text x′ . Let us call this concept “ politics ” in the remainder . She also notes that the words retails sales bounced back refer to a concept different from the previous one . Let us call this other concept “ economy ” . As the text is concerned with politics and economy she infers that its category is Business . Said otherwise , she detects excerpts that relate to particular concepts and decides on the text category based on the detected concepts . Similarly , our paradigm assumes that an explanation of a model ’ s prediction is understandable if it relies on a few concepts , where each concept relates to parts of text ( referred to as excerpts ) that are semantically consistent across multiple texts . Our methodology is as follows . First , our model encodes the input text into a binary low-dimensional vector , where each value ( 0 or 1 ) is computed from an excerpt and denotes the presence or absence of a specific concept in the input . Then , the prediction is made from this binary vector alone , allowing an easy interpretation of the decision in term of presence/absence of concepts . While we set a maximal value of the number of concepts ( which is the dimensionality of the binary representation ) , the concepts are unsupervised and not defined a priori . The model automatically determines them in a way that eases interpretation : each concept is encouraged to be semantically consistent and not to overlap with other concepts . Therefore , extracted excerpts for a concept must be discriminative of that concept only and share similar semantics . This is enforced through a concept consistency and separability constraint added as an auxiliary loss in the learning optimization problem . As a result , each discovered concept can be understood from the corresponding excerpts extracted that activate its appearance : in our previous example , the meaning of the first concept the user identifies is inferred from the excerpts she detected for that concept in x and x′ , i.e . the government said , made official what . Looking at these excerpts , we identify that concept as politics . Our idea relates to Latent Dirichlet Allocation ( LDA ) ( Blei et al. , 2003 ) , where text documents are described by a set of topics that are semantically consistent . However , LDA builds a probabilistic model of text generation , whereas our goal is to discover and define latent concepts that are relevant for a prediction task at hand . In comparison to rationale-based text processing models ( Lei et al. , 2016 ; Yu et al. , 2019 ; Bastings et al. , 2019 ) , we rely on a different paradigm : our model ’ s prediction is based on the absence or presence of discovered concepts , and makes no direct use of the words captured as excerpts . We do so to ease interpretation of the prediction . This makes the interpretation of a different nature than these methods , and simpler to understand for the user . Our contribution with this work is a new self-interpretable model that predicts solely from the presence/absence of concepts in the input . Concepts are learned unsupervisingly , and described by a semantically consistent set of excerpts . We experiment on three text categorization tasks and a multiaspect sentiment analysis task , compare our model ’ s performance with state-of-the-art prediction models , and demonstrate its interpretability . Note that an instance of our model for image processing is described in the supplementary Section E with illustrative experiments . 2 THE EDUCE MODEL . We present our model , called EDUCE for Explaining model Decisions through Unsupervised Concepts Extraction , using a multi-class classification task , but our method can be used to perform regression or multi-label classification . We consider a training dataset D of inputs ( x1 , .... , xN ) , xn ∈ X and corresponding labels ( y1 , ... , yN ) , yn ∈ Y . Contrarily to back-box models that use complex computations over low-level input features to predict the output class y , our objective is to enforce the model to map an x input to an easy-to-interpret representation z ( x ) ∈ { 0 , 1 } C , on which the output prediction relies . EDUCE ’ s inference process consists in the following steps . Step 1 : For each concept c ( i.e . each dimension of z ( x ) ) , compute pγ ( s|x , c ) : the probability of each excerpt s in x to be selected . Sample a unique excerpt sc ( x ) ∼ pγ ( s|x , c ) per concept ( per dimension of z ( x ) ) . Note that the same excerpt can be selected for multiple concepts . Step 2 : Decide on each value zc ( x ) of the representation z ( x ) by sampling from pα ( zc|sc ( x ) , c ) . Given sc ( x ) , zc ( sc ( x ) ) = 1 means concept c is detected as present and zc ( sc ( x ) ) = 0 means it is absent . Step 3 : Predict the output class y from z ( x ) = ( z1 ( s1 ( x ) ) , ... , zC ( sC ( x ) ) ) . We do not have concept-level annotations . To ensure semantic consistency and prevent overlap of the concepts , we jointly train a concept classifier to recognize , for every excerpt sc ( x ) such that zc ( sc ( x ) ) = 1 , the concept ( i.e . the dimension ) it was extracted for : the label for each sc ( x ) is simply c ∈ [ 1 , C ] . Figure 1 illustrates these steps , which we detail below . 2.1 GENERATING PREDICTIONS AND EXPLANATIONS . Step 1 : Extract a unique excerpt sc ( x ) per concept c. Given an input sentence x : w1 , ... , wM , where each wi is a word , a excerpt s consists of a span of consecutive words of flexible size between 3 and 10 words : s = wk , ... , wk+l , 3 ≤ l ≤ 10 . For each concept , we extract a unique excerpt , defined by its first and last word . Our extraction process is very similar to the one proposed in Question-Answering models ( e.g Devlin et al . ( 2019 ) ) . An excerpt is sampled by first sampling its start word and then its end word conditioned on the start word . We denote the probability over each excerpt s to represent concept c in x as pγ ( s|x , c ) and we write as the product of ( i ) pstart ( wk|x , c ) , the probability for s ’ s first word to be the start word of the excerpt to extract and ( ii ) pstop ( wk+l|x , c , wk ) the probability of wk+l to be the stop word given the start word wk : pγ ( s|x , c ) = pγ ( wk , ... , wk+l|x , c ) = pstop ( wk+l|x , c , wk ) pstart ( wk|x , c ) . We parametrize pstart ( wk|x , c ) and pstop ( wk+l|x , c , wk ) using recurrent neural networks . First we feed the entire input sentence x : w1 , ... , wM through a bidirectional LSTM , and we represent each word wi in the sentence as the concatenation of the forward pass and backward pass hidden states for that word : hk = [ −→ hk , ←− hk ] , ∀k ∈ [ 1 , M ] . Second , each hk is fed to a linear layer with parameters γstart which outputs a score for wk , followed by the softmax activation function over all possible words to have the probability distribution over each word to be the start word : pstart ( wk|x , c ) = exp ( γstart · hk ) ∑M k=1 exp ( γstart · hk ) , k = 1 , ... , M. ( 1 ) Using this distribution , we can sample a specific start word wstart ∼ pstart ( wk|x , c ) . Third , we feed again the vectors hk to another linear layer with parameters γstop that gives a score for each word to be the stop word . We mask out these scores before taking the softmax , such that the probabilities for wstart+l with l < 3 and l > 10 to be the stop word are 0. pstop ( wstart+l|x , c , wstart ) = exp ( γstop · hstart+l ) ∑10 l=3 exp ( γstop · hstart+l ) , l ≥ 3 or l ≤ 10 0 , l < 3 or l > 10 . We then sample a specific stop word wstop ∼ pstop ( wstart+l|x , c , wstart ) 1 . We now have the start word wstart and stop word wstop . The corresponding excerpt is wstart , ... , wstop and is represented as a fixed size vector sc ( x ) by computing the average of ( pre-trained , fixed ) words embeddings vectors of wstart , ... , wstop . Hence sampled excerpts sc ( x ) belong to Rd , where d is the dimension of embedding vectors . This process is illustrated in Figure 1 : For text x , the excerpt Retails sales bounced back is selected for concept c1 , the government said for concept c2 and a midsummer slump for concept c3 . Step 2 : For each c , from the excerpt sc ( x ) , decide on the value zc ( x ) denoting the presence/absence of concept c. Each extracted excerpt ( one per concept ) is processed to decide on the absence or presence of each concept in x . Specifically , for each concept c we take the dot product of a weight vector αc ∈ Rd with sc ( x ) , followed by a sigmoid activation function , in order to obtain the Bernoulli probability pα ( zc = 1|sc ( x ) , c ) = σ ( αc · sc ( x ) ) ( 2 ) This is the probability that sc ( x ) extracted from x for concept c triggers the presence of concept c. The binary vector z ( x ) is obtained by independently sampling each dimension : ∀c zc ∼ pα ( zc|sc ( x ) , c ) . This step can be seen as C independent binary classifiers , each of them deciding on the presence of a particular concept . We illustrate this in Step 2 in Figure 1 : For text x , the excerpt Retails sales bounced back activates the presence of concept c1 , the government said activates the presence of c2 but a midsummer slump does not activate the presence of c3 . Step 3 : Predict y from z ( x ) . As shown in Step 3 of Figure 1 , given an input x , the prediction of the output y is made solely from the intermediate binary representation z ( x ) . We use a linear classifier without bias , parametrized by a weight matrix δ ∈ R|Y|×C followed by a softmax activation function , returning pδ ( y|z ( x ) ) ∀y ∈ Y . Output classification training objective . The parameters { γ , α , δ } are learned in a end-to-end manner by minimizing cross-entropy , which writes for each x as : Loutput ( x , y , δ , α , γ ) = Es ( x ) ∼pγ [ Ez ( x ) ∼pα [ L output ( y , z ( x ) , δ ) ] ] = E∀c sc ( x ) ∼pγ ( s|x , c ) [ E∀c zc∼pα ( zc|sc ( x ) , c ) [ − log pδ ( y|z ( x ) ) ] ] . ( 3 ) We give details about the optimization in Section 2.3 . 1We also prevent stop words to be outside of the text length .

Summary: The paper introduces a new concept-based interpretability method that lies in the family of self-interpretable models (i.e. it's not a post-hoc method). Self-interpretability is achieved by a two-stage model: First, a concept-extractor finds the related pieces of consecutive words (excerpts) in a given text that are related to a concept among a set of given concepts (if any), then the model makes its predictions solely based on the presence or absence of concepts (binary).  The most useful part of the algorithm is that there is no need for concept annotations. The work then experimentally shows that their method, although does not outperform a non-self-interpretabile baseline but has better performance and interpretation compared to rival methods.