Year: 2019

Abstract:
Providing explanations along with predictions is crucial in some text processing tasks. Therefore, we propose a new self-interpretable model that performs output prediction and simultaneously provides an explanation in terms of the presence of particular concepts in the input. To do so, our model’s prediction relies solely on a low-dimensional binary representation of the input, where each feature denotes the presence or absence of concepts. The presence of a concept is decided from an excerpt i.e. a small sequence of consecutive words in the text. Relevant concepts for the prediction task at hand are automatically defined by our model, avoiding the need for concept-level annotations. To ease interpretability, we enforce that for each concept, the corresponding excerpts share similar semantics and are differentiable from each others. We experimentally demonstrate the relevance of our approach on text classification and multi-sentiment analysis tasks.

heading: 1 INTRODUCTION
text: While deep learning models are powerful tools to perform a large variety of tasks, their predictive process often remains obscure. Understanding their behavior becomes crucial. This is particularly true with text data, where predicting without justifications has limited applicability. There has been a recent focus on trying to make deep models more interpretable, see for example (Ribeiro et al., 2016; Bach et al., 2015; Shrikumar et al., 2017; Simonyan et al., 2014; Sundararajan et al., 2017). Specifically, methods have been recently proposed to provide such explanations simultaneously with the prediction. For example, (Lei et al., 2016; Yu et al., 2019; Bastings et al., 2019) select subsets of words in the input text that can account for the model’s prediction (called rationales). In this work, we propose a model that provides an explanation based on the absence or presence “concepts” that are automatically discovered in the texts.
Suppose we ask a user to tell what is the category (or class) of the top text of Figure 1 (text x). She detects that the words the government said relate to a specific concept that is present in x, a concept she also detect in the words made official what in text x′ . Let us call this concept “politics” in the remainder. She also notes that the words retails sales bounced back refer to a concept different from the previous one. Let us call this other concept “economy”. As the text is concerned with politics and economy she infers that its category is Business. Said otherwise, she detects excerpts that relate to particular concepts and decides on the text category based on the detected concepts.
Similarly, our paradigm assumes that an explanation of a model’s prediction is understandable if it relies on a few concepts, where each concept relates to parts of text (referred to as excerpts) that are semantically consistent across multiple texts. Our methodology is as follows. First, our model encodes the input text into a binary low-dimensional vector, where each value (0 or 1) is computed from an excerpt and denotes the presence or absence of a specific concept in the input. Then, the prediction is made from this binary vector alone, allowing an easy interpretation of the decision in term of presence/absence of concepts.
While we set a maximal value of the number of concepts (which is the dimensionality of the binary representation), the concepts are unsupervised and not defined a priori. The model automatically determines them in a way that eases interpretation: each concept is encouraged to be semantically consistent and not to overlap with other concepts. Therefore, extracted excerpts for a concept must be discriminative of that concept only and share similar semantics. This is enforced through a concept consistency and separability constraint added as an auxiliary loss in the learning optimization problem.
As a result, each discovered concept can be understood from the corresponding excerpts extracted that activate its appearance: in our previous example, the meaning of the first concept the user identifies is inferred from the excerpts she detected for that concept in x and x′, i.e. the government said, made official what. Looking at these excerpts, we identify that concept as politics.
Our idea relates to Latent Dirichlet Allocation (LDA) (Blei et al., 2003), where text documents are described by a set of topics that are semantically consistent. However, LDA builds a probabilistic model of text generation, whereas our goal is to discover and define latent concepts that are relevant for a prediction task at hand. In comparison to rationale-based text processing models (Lei et al., 2016; Yu et al., 2019; Bastings et al., 2019), we rely on a different paradigm: our model’s prediction is based on the absence or presence of discovered concepts, and makes no direct use of the words captured as excerpts. We do so to ease interpretation of the prediction. This makes the interpretation of a different nature than these methods, and simpler to understand for the user.
Our contribution with this work is a new self-interpretable model that predicts solely from the presence/absence of concepts in the input. Concepts are learned unsupervisingly, and described by a semantically consistent set of excerpts. We experiment on three text categorization tasks and a multiaspect sentiment analysis task, compare our model’s performance with state-of-the-art prediction models, and demonstrate its interpretability. Note that an instance of our model for image processing is described in the supplementary Section E with illustrative experiments.

heading: 2 THE EDUCE MODEL
text: We present our model, called EDUCE for Explaining model Decisions through Unsupervised Concepts Extraction, using a multi-class classification task, but our method can be used to perform regression or multi-label classification. We consider a training dataset D of inputs (x1, ...., xN ), xn ∈ X and corresponding labels (y1, ..., yN ), yn ∈ Y . Contrarily to back-box models that use complex computations over low-level input features to predict the output class y, our objective is to enforce the model to map an x input to an easy-to-interpret representation z(x) ∈ {0, 1}C , on which the output prediction relies. EDUCE’s inference process consists in the following steps. Step 1: For each concept c (i.e. each dimension of z(x)), compute pγ(s|x, c): the probability of each excerpt s in x to be selected. Sample a unique excerpt sc(x) ∼ pγ(s|x, c) per concept (per dimension of z(x)). Note that the same excerpt can be selected for multiple concepts. Step 2: Decide on each value zc(x) of the representation z(x) by sampling from pα(zc|sc(x), c). Given sc(x), zc(sc(x)) = 1 means concept c is detected as present and zc(sc(x)) = 0 means it is absent. Step 3: Predict the output class y from z(x) = (z1(s1(x)), ..., zC(sC(x))).
We do not have concept-level annotations. To ensure semantic consistency and prevent overlap of the concepts, we jointly train a concept classifier to recognize, for every excerpt sc(x) such that zc(sc(x)) = 1, the concept (i.e. the dimension) it was extracted for: the label for each sc(x) is simply c ∈ [1, C]. Figure 1 illustrates these steps, which we detail below.

heading: 2.1 GENERATING PREDICTIONS AND EXPLANATIONS
text: Step 1: Extract a unique excerpt sc(x) per concept c. Given an input sentence x : w1, ..., wM , where each wi is a word, a excerpt s consists of a span of consecutive words of flexible size between 3 and 10 words: s = wk, ..., wk+l, 3 ≤ l ≤ 10. For each concept, we extract a unique excerpt,
defined by its first and last word. Our extraction process is very similar to the one proposed in Question-Answering models (e.g Devlin et al. (2019)). An excerpt is sampled by first sampling its start word and then its end word conditioned on the start word.
We denote the probability over each excerpt s to represent concept c in x as pγ(s|x, c) and we write as the product of (i) pstart(wk|x, c), the probability for s’s first word to be the start word of the excerpt to extract and (ii) pstop(wk+l|x, c, wk) the probability of wk+l to be the stop word given the start word wk: pγ(s|x, c) = pγ(wk, ..., wk+l|x, c) = pstop(wk+l|x, c, wk)pstart(wk|x, c). We parametrize pstart(wk|x, c) and pstop(wk+l|x, c, wk) using recurrent neural networks. First we feed the entire input sentence x : w1, ..., wM through a bidirectional LSTM, and we represent each word wi in the sentence as the concatenation of the forward pass and backward pass hidden states for that word: hk = [ −→ hk, ←− hk], ∀k ∈ [1,M ]. Second, each hk is fed to a linear layer with parameters γstart which outputs a score for wk, followed by the softmax activation function over all possible words to have the probability distribution over each word to be the start word:
pstart(wk|x, c) = exp(γstart · hk)∑M k=1 exp(γstart · hk) , k = 1, ...,M. (1)
Using this distribution, we can sample a specific start word wstart ∼ pstart(wk|x, c). Third, we feed again the vectors hk to another linear layer with parameters γstop that gives a score for each word to be the stop word. We mask out these scores before taking the softmax, such that the probabilities for wstart+l with l < 3 and l > 10 to be the stop word are 0.
pstop(wstart+l|x, c, wstart) =  exp(γstop · hstart+l)∑10 l=3 exp(γstop · hstart+l) , l ≥ 3 or l ≤ 10
0, l < 3 or l > 10.
We then sample a specific stop word wstop ∼ pstop(wstart+l|x, c, wstart)1. We now have the start word wstart and stop word wstop. The corresponding excerpt is wstart, ..., wstop and is represented as a fixed size vector sc(x) by computing the average of (pre-trained, fixed) words embeddings vectors of wstart, ..., wstop. Hence sampled excerpts sc(x) belong to Rd, where d is the dimension of embedding vectors.
This process is illustrated in Figure 1: For text x, the excerpt Retails sales bounced back is selected for concept c1, the government said for concept c2 and a midsummer slump for concept c3.
Step 2: For each c, from the excerpt sc(x), decide on the value zc(x) denoting the presence/absence of concept c. Each extracted excerpt (one per concept) is processed to decide on the absence or presence of each concept in x. Specifically, for each concept c we take the dot product of a weight vector αc ∈ Rd with sc(x), followed by a sigmoid activation function, in order to obtain the Bernoulli probability pα(zc = 1|sc(x), c) = σ(αc · sc(x)) (2) This is the probability that sc(x) extracted from x for concept c triggers the presence of concept c. The binary vector z(x) is obtained by independently sampling each dimension: ∀c zc ∼ pα(zc|sc(x), c). This step can be seen as C independent binary classifiers, each of them deciding on the presence of a particular concept. We illustrate this in Step 2 in Figure 1: For text x, the excerpt Retails sales bounced back activates the presence of concept c1, the government said activates the presence of c2 but a midsummer slump does not activate the presence of c3.
Step 3: Predict y from z(x). As shown in Step 3 of Figure 1, given an input x, the prediction of the output y is made solely from the intermediate binary representation z(x). We use a linear classifier without bias, parametrized by a weight matrix δ ∈ R|Y|×C followed by a softmax activation function, returning pδ(y|z(x))∀y ∈ Y . Output classification training objective. The parameters {γ, α, δ} are learned in a end-to-end manner by minimizing cross-entropy, which writes for each x as:
Loutput(x, y, δ, α, γ) = Es(x)∼pγ [Ez(x)∼pα [L output(y,z(x), δ)]]
= E∀c sc(x)∼pγ(s|x,c)[E∀c zc∼pα(zc|sc(x),c)[− log pδ(y|z(x))]]. (3) We give details about the optimization in Section 2.3.
1We also prevent stop words to be outside of the text length.

heading: 2.2 UNSUPERVISED DISCOVERY OF LATENT CONCEPTS
text: To ease interpretation of the concepts’ meaning, we want each dimension z to correspond to semantically consistent concepts. In other words, for a given concept c (i.e. a given dimension zc), for all inputs where that concept is present (i.e. ∀x s.t. zc(sc(x)) = 1), the corresponding excerpts sc(x) should share common semantics. While the C binary classifiers deciding on the presence of each concept c (Step 2) enforce consistency within the set of extracted excerpts sc, there is no guarantee that the concepts are not overlapping (i.e. are separable). We want concepts to be separable so that each concept captures a particular notion. Excerpts extracted for concept c must be distinguishable from the excerpts extracted for another concept c′. If we take the example of Figure 1: In Step 2 the excerpt Retails sales bounced back is identifiable as relating to economy/finance, contrarily to the government said, made official what which trigger the concept politics.
We rephrase our desiderata for consistency and separability as classification of the excerpts, with as many classes as the number of concepts. An external classifier should recognize that elements in the set {sc(x)|∀x zc(sc(x)) = 1} belong to concept c, and {sc′(x)|∀x zc′(sc′(x)) = 1} belong to concept c′ and not c. Recall that we do not have labels for the concepts. The ground-truth label of each extracted excerpt sc(x) is the index c of the dimension of z(x) for which it was extracted. We learn a linear concept classifier without bias, parametrized by a weight matrix θ ∈ R|C|×d followed by a softmax activation function, returning pθ(c′|sc(x)),∀c′ ∈ [1, C]. The concept classifier is trained by minimizing cross-entropy, where the label of each excerpt sc(x) the index of the concept for which it was extracted (i.e. c):
Lconcept(x, θ, α, γ) = Es(x)∼pγ [Ez(x)∼pα [L concept(z(x), s(x), θ)]] (4) = Es(x)∼pγ [Ez(x)∼pα [ ∑ c −zc(sc(x)) log pθ(c|sc(x))]],
where s(x) refers to the set of excerpts extracted for each concept: s(x) = {sc(x), c = 1, ..., C}. The loss considers concepts that are present in x (i.e. zc(sc(x)) = 1). The role of the concept classifier is to further enforce concepts consistency and to prevent overlap of concepts, and is jointly train with the rest of the model.
Note that adding a sparsity constraint on the number of concepts present in the inputs enforces semantic consistency, as sparse coding has been shown to induce useful, interpretable representations (Bengio et al., 2013a; Mairal et al., 2010). We experimentally demonstrate that a sparsity constraint is not sufficient, and can harm output prediction performance. On the opposite, with our concept classifier, sparsity is encouraged as Lconcept depends on the concepts that are present, but if concepts are consistent and separable Lconcept can be low without harming task performance.

heading: 2.3 EDUCE OBJECTIVE FUNCTION AND OPTIMIZATION
text: We jointly learn the concept classifier and output classifier and our objective function is the sum Equations 3 and 4:
L(x) = Es(x)∼pγ [Ez(x)∼pα [L output(y,z(x), δ) + λLconcept(z(x), s(x), θ)]] (5)
where λ controls the strength of the concept consistency constraint wrt the output prediction. The loss L(x) is differentiable, therefore the gradients with respect to the output classifier’s weights (δ) and concept classifier’s weights (θ) can be computed and back-propagated. However, the gradients with respect to the parameters of the excerpts extraction (γ) and the Bernoulli distribution over presence of concepts (α) pose an issue due to the sampling of these discrete random variables in Steps 1 and 2. As the explicit computation of the expectation involves expensive summations over all possible values of s(x) and z(x), we resort to Monte-Carlo approximations of the gradient (Sutton & Barto, 1998), with the loss L(x) used as reinforcement signal. The gradient derivation is provided in Supplementary Material. Our code will be released upon acceptance.
Note that the concept classifier encourages concept consistency and concept separability. The discovered concepts are not guaranteed to align with human defined concepts. This is a general challenge in learning explainable/interpretable models in an unsupervised setting. We believe it is a reasonable assumption to employ a linear concept classifier over pre-trained word embeddings such that it linear separates the word embedding space.

heading: 3 RELATED WORK
text: Our work relates to topic models, especially to Latent Dirichlet Allocation (LDA) (Blei et al., 2003) where each document is modeled as a mixture of topics and each word relates to one or more topics. While LDA has been recently combined with recurrent neural networks (Zaheer et al., 2017), the methodology remains different. LDA learns the parameters of a probabilistic graphical model of text generation while our goal is to build an interpretable prediction model that relies on binary presence/absence of concepts.
Some existing works interpret an already trained model, typically using perturbation and gradientbased approach, (Ribeiro et al., 2016; Bach et al., 2015; Shrikumar et al., 2017; Simonyan et al., 2014; Sundararajan et al., 2017) Alvarez-Melis & Jaakkola (2017) design a model that detects inputoutput pairs that are causally related. Contrarily to these methods, our work falls in the domain of self-interpretable models, which produce an explanation simultaneously with their prediction. Lei et al. (2016); Yu et al. (2019); Bastings et al. (2019) develop interpretable models for NLP tasks by selecting rationales, i.e. parts of text, on which a consequent model bases its prediction. A rationale acts as a justification supporting the prediction and is different from our definition of excerpt. In EDUCE, excerpts support the attribution of presence of concepts, and this attribution supports the prediction. Goyal et al. (2019) propose visual explanations of a classifier’s decision and Alaniz & Akata (2019) use an observer-classifier pair model, whose prediction can be exposed as a binary tree. Contrarily to ours, their model does not provide a local explanation based on parts of the input. Quint et al. (2018) extend a classic variational auto-encoder architecture with a differentiable decision tree classifier. However, their methodology is different and they only experiment on image data.
Most related to our work are methods providing concept-based explanations. Alvarez Melis & Jaakkola (2018) learn a self-explainable classifier that takes as input a set of concepts extracted from the original input. They define a set of desiderata for what is an interpretable concept, but simply represent extracted concepts as an encoding of the input learned with an auto-encoding loss. Kim et al. (2018) learn concept activation vectors, and Zhou et al. (2018) build on Bau et al. (2017) to propose a method that generates visual explanations of a classifier, in the form of semantically interpretable components with a corresponding label and heatmap. However, in these models, the final classifier is already trained and fixed while EDUCE is learned end-to-end. More importantly, these works need concepts to be predefined from human annotations, while EDUCE is unsupervised. Ghorbani et al. (2019) propose a method based on unsupervised concept discovery to explain a model a posteriori. We differ from their work as EDUCE is trained end-to-end to classify solely based on the discovered concepts, which presence is discriminant, resulting in a fully interpretable model. Finally, we focus on text data while the aforementioned works experiment on image data.

heading: 4 TEXT CLASSIFICATION EXPERIMENTS
text: We experiment on three text classification datasets. The DBpedia ontology classification dataset (Zhang et al., 2015) was constructed by picking 14 non-overlapping categories from DBpedia 2014 (Lehmann et al., 2015). There are 14 classes. We use 56, 000 examples of the train dataset for training, and 56, 000 for validation. For testing, we use 7, 000 examples of the test dataset (using stratified sampling). Also in Zhang et al. (2015), the AGNews topic classification dataset was constructed from the AG dataset’s 4 largest categories. There are 4 classes. We separate the training set into 84, 000 training samples and 24, 000 validation samples and report results on the full test dataset. We also experiment on the Stanford Sentiment Treebank (SST) (Socher et al., 2013) that includes 5 sentiment classes. For DBPedia and AGNews we use fixed word vectors trained on Common Crawl (Grave et al., 2018). For SST, we use fixed Glove word vectors (Pennington et al., 2014).
Comparative models. Since we are concerned with building self-interpretable models, we do not compare with the methods presented in Section 3 that explain a model a posteriori. Rather, for comparison, we report the performance of (i) using a Bidirectional LSTM on the full text and predict directly from its hidden state (Baseline) (ii) the No Concept Loss model that uses a binary discrete intermediate representation but no concept classifier, it corresponds to λ = 0 and its training objective is simply Loutput and (iii) adding a L1-norm sparsity constraint to the No Concept Loss model (referred to as No Concept Loss + L1), i.e. Equation 5 changes to:
L(x) = Es(x)∼pγ [Ez(x)∼pα [L output(y,z(x), δ) + λL1 |z|]]. (6)
Therefore both No Concept Loss and No Concept Loss + L1 models extract excerpts and use a
binary intermediate representation, but are trained without concept loss.
Hyperparameters selection. We tried different values of hyperparameters (details in supplementary Section B.1) and for each set of hyperparameters, we run 5 different random seeds. We choose the best hyperparameters set using its average validation performance over the 5 random seeds.
Metrics. We report the test output accuracy (Output Acc.) over the task. Evaluating the relevance of discovered concepts is a challenge in the unsupervised setting. To measure how well the discovered concepts align with human defined concepts one would need human experiments, which we consider as future work. However, we evaluate how consistent and separable the discovered concepts are, as a proxy to measure how easy it would be for a user to interpret them. For EDUCE, we can compute the accuracy of the concept classifier on the test data. However, this should be low for models No Concept Loss and No Concept Loss + L1 that do not train with a concept loss. Therefore, to evaluate interpretability, we report an a posteriori concept accuracy: after training, for each model (including EDUCE), we gather the excerpts extracted (zc = 1) in the test data. We separate these excerpts into two sets (training and testing, note that these are both generated from the test data). We train a new, separate linear concept classifier a posteriori on the extracted excerpts to evaluate and we report its performance as a posteriori concept accuracy (A Posteriori Concept Acc.).
Quantitative results on DBPedia and AGNews. Table 1 reports test performance on the DBPedia and AGNews datasets for C = 10 concepts. We report results with values λ = 0.1 and λL1 = 0.1 that give best trade-off performance between output and concept accuracies. Complete results are available in supplementary Table 4. For all metrics we report the mean and standard error of the mean (SEM) over the training random seeds. Table 1 shows that the Baseline model outperforms the other models that all employ a binary discrete encoding of the input. This is expected, and shows the trade-off between interpretability and output accuracy. In terms of output accuracy, EDUCE is comparable with its counterpart that does not encourage concept consistency (No Concept Loss), loosing only 0.4% on DBPedia and 0.7% on AGNews, and as expected outperforms it in terms of concept accuracy. Adding an L1 constraint to the No Concept Loss model increases the consistency of the concepts (as per A posteriori Concept Accuracy values) yet is largely outperformed by EDUCE on that metric, and output accuracy is also lower than EDUCE’s.
Interpreting EDUCE on AGNews. We turn to show how EDUCE’s output prediction is easily interpretable. The following results were generated with λ = 0.1 and C = 10 concepts. Figure 2a shows examples of the AGNews test set that were correctly classified by EDUCE. The underlined words correspond to the excerpts extracted for different concepts. Separately, in Figure 2b we show, for each concept detected in the examples of Figure 2a, examplar excerpts extracted from others test documents (excerpts are separated by “/"). We interpret the concepts’ meaning as follows: concept 0 maps to informatics notions, concept 2 to corporations/petrol, concept 4 to the notions of investments and finance, concept 6 to sports events and concept 8 to governmental/state affaires. Concept 1 is less clearly defined. Note that in Figure 2b excerpts are consistent yet are extracted in texts from multiple output classes. The excerpts extracted in the examples of Figure 2a are consistent with these interpretations. These results show how easily the classification of any text can be explained by the detection of multiple, relevant, and intelligible concepts. More qualitative examples are in supplementary Section C. Note that supplementary Table 10 shows the concepts extracted by the No Concept Loss model, we see that they form less semantically consistent units and are hard to interpret. Supplementary table 8 shows test examples where the final classifier’s prediction was incorrect. In some of these, the predicted class could be considered as an appropriate label, and importantly the concepts that are identified as present in the text are relevant with the set of discovered concepts.
Class Business: sports retailer jjb yesterday reported a near 25 drop in profits and continuing poor sales , and ended shareholders #39 hopes of a takeover by announcing that a potential bidder had walked away . Class World: bangkok , thailand sept . 30 , 2004 - millions of volunteers led by emergency teams fanned out across thailand on thursday in a new drive to fight bird flu after the prime minister gave officials 30 days to eradicate the epidemic . Class Sports: the spanish government responded to diplomatic pressure from britain yesterday by starting a search for fans who racially abused england players during a quot friendly quot football match with spain . Class Sci/Tech: los angeles ( reuters ) - a group of technology companies including texas instruments inc . &lt txn . n&gt , stmicroelectronics &lt stm . pa&gt and broadcom corp . &lt brcm . o&gt , on thursday said they will propose a new wireless networking standard up to 10 times the speed of the current generation .
(a) AGNews test examples correctly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.
Concept 0 software services giant / moonwalk to home / video display chip / downloading music . Concept 1 upcoming my prerogative video / his ever-growing swimming / launch of a video display chip / illegality of downloading Concept 2 oil market . / oil giant sibneft / oil prices and / corp . &lt brcm Concept 4 cash settlement of up to #36 50 million / six-year deal worth about $40 million / the dollar dipped to a four-week low against the euro / five shares , Concept 6 olympic 100-meter freestyle / sox ’ family / olympics should help / athletes were already Concept 8 frail pope john paul / indian army major shot / unions representing workers / goverment
representatives .
(b) Examples of excerpts that are extracted accross the test set, corresponding to the concepts detected in Figure 2a. Colors match the colors used in Figure 2a.
Figure 2: Interpretation of EDUCE prediction through concept analysis.
SST dataset Figure 3d shows output accuracy versus a posteriori concept accuracy on the SST dataset, for different different number of concepts C and different values of λ: Each marker is a different value of λ, and values are (from left to right markers) {0, 0.01, 0.1, 0.5}. We do the same for the No Concept Loss + L1 model. Note that the left most markers in all curves correspond to λ = 0 and λL1 = 0, therefore to the model referred to as No Concept Loss. While our goal and model is very different than Lei et al. (2016); Bastings et al. (2019), we report their test scores (obtained using 40% of the text on the SST dataset, we report directly from Bastings et al. (2019)). The a posteriori concept accuracy for Baseline, Lei et al. (2016) and Bastings et al. (2019) is not applicable. Figure 3d confirms our results: EDUCE (in blue, purple and red) overperforms the model No Concept Loss + L1 (in green) on both output and a posteriori concept accuracies, and achieve output accuracy comparable to Lei et al. (2016) (orange line) while having 60% a posteriori concept accuracy. Regarding the effect of the number of concept C, for a fixed C, choosing the adequate value of λ allows the user to balance between final accuracy and concepts’ accuracy. Similar performances can be achieved by different number of concepts: C = 20, λ = 0.1 (dotted purple line, squared marker) achieves ∼ 41.5% final accuracy and ∼ 70% concept accuracy, while C = 5, λ = 0.01 (plain blue line, diamond marker) achieves ∼ 42.0% final accuracy and ∼ 60% concept accuracy. The smaller the number of concepts, the smaller the value of λ which is expected as with less concepts the task of the concept classifier is easier.
Figure 3a shows the empirical frequency of presence of each concept, per output class in the SST dataset, using λ = 0.1 and C = 10 concepts. We see that concepts 3, 5 and 7 are often triggered when a sample is positive or very positive, and on the opposite concept 8 is present in negative/very negative texts. The values of weight matrix δ of the output classifier in Figure 3b proves that the presence of these concepts is responsible for the output classifier’s prediction. We can further analyze the concepts: Each word in SST sentences is annotated with a label for the sentiment it expresses, referred to as word label. We do not use them during training or validation, but at test-time this allows us to analyze the repartition of words labels in the set of excerpts extracted for each concept. Figure 3c shows that indeed the excerpts selected for concepts 3, 5 and 7 are mostly composed of positive or very positive words, and the excerpts extracted when concept 8 is triggered are mostly composed of negative or very negative words2.
2We do not show the amount of neutral words selected as it squeezes the histograms.
(a) Per class concept frequency. (b) Output classifier weight matrix δ.
(c) Repartition of words labels for concepts 3, 5, 7, 8. (d) SST test performance, output accuracy vs a posteriori concept accuracy. A posteriori concept accuracy is not applicable for Baseline, Lei et al. (2016) and Bastings et al. (2019).
Figure 3: Results on the SST dataset.

heading: 5 MULTI ASPECT SENTIMENT ANALYSIS EXPERIMENT
text: BeerAdvocate dataset While our paradigm is different than Lei et al. (2016); Bastings et al. (2019), we also perform a multi-aspect sentiment regression experiment on the pre-processed subset of the BeerAdvocate3 dataset (McAuley et al., 2012). It consists of 260, 000 beer reviews where ratings for four aspects (look, smell, palate and taste) are given, as well as an overall rating. The reviews are separated into training/validation sets specific to each aspect, and ratings are mapped to scalars in [0, 1]. This is a regression task, so we modify the Baseline model and EDUCE in the last layer of the prediction with a sigmoid activation function, and use Mean Squared Error (MSE) instead of cross-entropy for Loutput(y,z(x), δ). McAuley et al. (2012) provided a test set (994 reviews) with sentence-level rationale annotations: different parts of each review are annotated with one (or multiple) aspect label, indicating what aspect it covers (referred to as gold rationales). Importantly, we do not use these gold rationales during training or validation, but use to evaluate our model. Hyperparameters selection is similar to our classification experiments.
We train EDUCE on the prediction of the 5-values vector at once (4 aspects and the overall score, there are therefore 5 scalar ratings to predict) using the 260, 000 reviews. We compare EDUCE’s performance with the Baseline that accesses the full text. The test MSE is 0.0089± 0.0 (mean and SEM accross training seeds) for the Baseline model and 0.0119± 0.0 for EDUCE using λ = 0.01 and C = 10 concepts, with a corresponding concept accuracy of 92.8± 0.7. Of specific interest to us is the per-concept precision of gold rationales and the total percentage of selected words. Indeed, Table 3 shows that when predicting the 4 aspects and the overall score, some concepts are capturing a specific aspect: for example, the precision of Concept 7 on the Appearance aspect is 97.09%. This is confirmed by Table 2 that shows that excerpts extracted for Concept 7 are mostly related to the color and head (the frothy foam on top of beer) of the beer4. Supplementary Table 5 reports examplar excerpts for all concepts.
As in Lei et al. (2016); Bastings et al. (2019) we also train EDUCE on the prediction of each of the first three aspects separately. Supplementary Table 6 reports the precision of gold rationales for each aspect. While the results are less convincing than the models specifically designed for rationales extraction, extracted excerpts match gold rationales to a certain extent.
3https://www.beeradvocate.com/ 4Tables 3 and 2 reports results of the best seed for the cross-validated hyperparameters.
Concept 0 fruity esters , and/ fruits , caramelized pecans , and/ toffee and caramel accents ,/ coffee and chocolate flavors/ earthy hop resin . Concept 2 creamy and a/ chewy and rich and drinkability/ creamy mouthfeel that/ smooth and just velvety on the/ thick , and Concept 3 rich malt scents/ aroma is quite hoppy with big citrus/ tons of different sweet malts , toffe/ smells extremely roasty/ boom of grapefruity Concept 7 good head and lacing/ beautiful golden-amber color/ creamy tan head/ nice deep brown color/ proud head has settled . nothing Concept 9 carbonation is graceful/ drinkability is excellent/ mouthfeel is wonderful/ mouthfeel is exemplary/ drinkability : excellent
Table 2: Examples of excerpts extracted on the Beer test set.
Concept Appearance Smell Palate Taste Concept 0 0.85 55.75 1.79 40.60 Concept 1 0.54 5.48 11.70 30.58 Concept 2 1.22 0.90 78.18 15.03 Concept 3 1.46 92.39 0.30 5.34 Concept 4 45.33 0.23 0.26 0.16 Concept 5 7.27 22.50 2.27 28.86 Concept 6 0.36 5.86 6.40 28.73 Concept 7 97.09 0.98 0.45 0.53 Concept 8 0.18 0.09 18.07 5.93 Concept 9 0.00 0.00 68.42 10.53
Table 3: Per-concept precision of gold rationales (in % for each aspect), trained to predict all 4 aspects and the overall score. In bold we emphasize the concepts which precision is above 50% for an aspect’s gold rationales.

heading: 6 DISCUSSION AND PERSPECTIVES
text: We propose a new self-interpretable model, EDUCE, that maps inputs to an easy-to-interpret representation on which the output prediction relies. We experimentally demonstrate its ability (i) to extract semantically meaningful concepts described by consistent excerpts, and (ii) to ground its output prediction on the presence/absence of these concepts in each input. EDUCE generates relevant explanations while retaining high predictive performance. As future direction for research, we contemplate using hierarchical representations, as human notions can often be represented with hierarchical structures, for example as in WordNet (Miller, 1995). Furthemore, our principles can be applied on different types of inputs like images, and we provide initial experiment in Supplementary Section E.

heading: A DETAILS ON THE LEARNING ALGORITHM
text: As mentioned in our main paper, when computing Equation 5 the cross-entropy losses of the output classifier and concept classifier are differentiable, therefore the gradients with respect to the weights δ and θ can be computed. However, the gradients with respect to the parameters of the LSTM (γ) and the Bernouilli distribution over presence of concepts (α) pose a difficulty because we sample ∀c sc(x) ∼ pγ(s|x, c) and ∀c zc ∼ pα(zc|sc(x), c). We use straight-through estimator (Bengio et al., 2013b) in the backward pass for the parameters α, and resort to Monte-Carlo approximations of the gradient employed in Reinforcement Learning (Sutton & Barto, 1998) for the parameters γ. The loss L(x) serves as reinforcement signal. We weight the two terms of the gradient using a parameter r as follows:
∇δ,θ,γ,αL(x)=̂(1− r)∇δ,θ,αL(x) + rEs(x)∼pγ [(L(x)− b)∇γ log pγ(s(x)|x)] (7)
where r controls the strength of the Reinforcement Learning term and b is the average of the loss, used as control variate.

heading: B EXPERIMENTAL DETAILS AND DETAILED RESULTS.
text: We monitor validation performance and use early stopping. For the comparative models we monitor the output prediction accuracy. For EDUCE we monitor the sum of the output prediction accuracy and the concept classifier accuracy (weighted by the parameter λ).
B.1 HYPERPARAMETERS CONSIDERED
In all experiment, we increase the value of λ by 10% every epoch. In order to choose the best set of hyperparameters, we take the average value on the 5 random seeds.
DBPedia and AGNews We try the following ranges of hyperparameters λ ∈ {0.0, 0.001, 0.01, 0.1}, λL1 = {0.0, 0.001, 0.01, 0.1, 1}, r ∈ {0.01, 0.1}, learning rate lr = {0.0005, 0.001}. The hidden state of the LSTM is of size 200. We use Adam optimizer (Kingma & Ba, 2014), batches of size 64. For the Baseline model, we try adding weight decay in {0.0, 1e− 6} and dropout in {0.0, 0.1, 0.2}. SST We try the following ranges of hyperparameters λ ∈ {0.0, 0.001, 0.01, 0.1}, λL1 = {0.0, 0.001, 0.01, 0.1, 1}, r ∈ {0.01, 0.1}, learning rate lr = {0.0002, 0.001} and weight decay in {0.0, 1e − 6}. The hidden state of the LSTM is of size 150. We use Adam optimizer (Kingma & Ba, 2014), batches of size 25. For the Baseline model, we follow Bastings et al. (2019) and use weight decay 1e− 6 and dropout {0.5}. BEER We try the following ranges of hyperparameters λ ∈ {0.001, 0.01, 0.1}, r ∈ {0.001, 0.01, 0.1}, learning rate lr = {0.0005, 0.001}. The hidden state of the LSTM is of size 200. We use Adam optimizer (Kingma & Ba, 2014), batches of size 256. We also clip the gradient to the value of 1 and employ an entropy maximization regularization on mean of the Bernoulli distributions ∀c, pα(zc|sc(x), c), weighted by 0.001. We train all models for 250 epochs and use early stopping on the validation set. For the Baseline model on the BEER dataset, we use the same hyperparameters as in Bastings et al. (2019) (lr = 0.0004, hidden dimension 200, dropout in {0.1, 0.2} and weight decay 1e− 6) and train for 100 epochs.
B.1.1 DETAILED RESULTS ON AGNEWS AND DBPEDIA
Table 4 details output accuracy, a posteriori concept accuracy, and sparsity (number of concepts presents per input) for different values of λ and λL1 considered.
B.2 ADDITIONAL RESULTS ON BEER
As in Lei et al. (2016); Bastings et al. (2019), we also train EDUCE on the prediction of each of the first three aspects (the 4th aspect they do not experiment on)5. Table 6 reports the precision of gold rationales for that aspect and the total percentage of selected words, values for Lei et al. (2016); Bastings et al. (2019) are taken from Bastings et al. (2019). We use C = 10 and λ = 0.1.
5Bastings et al. (2019) call the third aspect Taste but they actually refer to Palate.
We consider a word as selected if any of the concept selects it, and count it as one selected even if it is selected by multiple concepts. While the results are less convincing than the aforementioned rationales models that are specifically designed for this task, we see that extracted excerpts match gold rationales to a certain extent, especially for the aspects Appearance and Smell. For the Palate aspect, manual inspection shows us that for some seeds, the model defines a concept dedicated to the notion of numbers (e.g. corresponding to excerpts “12 ounce bottle”) that is consistent, yet not part of the gold rationale.
C ADDITIONAL QUALITATIVE EXAMPLES ON DBPEDIA AND AGNEWS
D VARYING THE NUMBER OF CONCEPTS ON DBPEDIA
Figures 4a and 4b (the second is a zoom to better see the differences between 10, 20 and 50 concepts) show the trade-off between concept accuracy and final accuracy on the DBPedia dataset, for multiple values of C and different value of λ. Each line is a value of C. Each marker is a different value of λ, and values are (from left to right markers) {0, 0.01, 0.1, 0.5}. Separately, we show examplar excerpts of concepts learned with C = 50 and λ = 0.5 (purple dot in Figure 4b) in Table 13 (only 36 concepts are identified as present and we report those) to compare with the ones learned with C = 10 and λ = 0.1 in Table 11 (blue square in Figure 4b). With C = 50, the learned concepts qualitatively seem more consistent, even if the a posteriori concept accuracy is 81.37% for the model with C = 50 which excerpts we report in Table 13, which is slightly less than the value (83.90%) for the model with C = 10. Hence, learned concepts with C = 50 might be more similar to each others (e.g. less separable). Furthermore, an explanation with more concepts to look at and interpret is also harder to parse, which is why we consider that the binary intermediate representation should be low-dimensional. In future work, we would like to conduct a large scale experimentation involving human judgments to conclude on the interpretability benefits of different values of C. Note that with a too small value of C (here, C = 5) final accuracy is harmed as the intermediate representation is too coarse.
Concept 0 member of sire records act / primate native to / an indictment of policy changes / annexation of mrauk-u kingdom / regents of the university of / litterateur historian and activist / baseman first baseman and manager / democrat to the thirtieth congress to fill the vacancy caused / lieutenant commander henry crommelin / governor ’ s residence and / prime minister of canada arthur meighen who held office / committee in ilam district Concept 1 is portuguese wine grape Concept 2
william james young ( 1800–1870 ) probably born in / april 1983 – 30 december 1992 ) was a british-bred / ramos ’ crepidium ) is a member / brian wilson presents smile / film starring chad everett sarah / robert service then a professor / st . george ’ s r . c . primary / vicky hamilton ( born april / james jim arthur bacon ( birth registered october–december / jonathan david morris ( october / clarence l . evans ( de-113 ) was a cannon-class / arthur goodson house
Concept 4 a tribe of / a neighborhood in / a village in Concept 6
correspondents located in most of the district headquarters / forest on the western slopes on / river range in / album from stone temple / forest home of / seaside town of southwold and centers / whisky distillery . the distillery / favela of capão redondo in / bar and moved / hospital ships was built / house at 1747 mooresfield road / island west of the dingle peninsula . at / village in pathanamthitta district located
Concept 7 shoes and other dance shoes and apparel . / racehorse who won at / his section macrophylli / players in the / man who flees the city and works in / biography of jesse lauriston livermore . the / painter who lived in / artist . hamilton is noted for managing the / footballer of the 1910s and ’ 20s and coach of / singer from finland . elomaa was born in / exhibition at the / resident paid for / fan gyhirych in Concept 9
top 50 company / of the five operating divisions of the thomson corporation / rankings 1978/1979 the professional world rankings / collection of educational dvds tv series music / independent alternative newspaper / leading public university in the field of agriculture / network manager at the national / hong kong cross-media creator / electrical multiple units / artist-run centre located in vancouver bc with a vision / supply drinking water / centre of Ørskog municipality
Concept 10 an international producer / last studio album / a 1971 road movie directed by monte hellman starring singer-songwriter / a damned poet / fryderyk chopin university of music / a british musician . he was a bassist / a french passport / a canadian record producer music publisher Concept 11
is a top 50 company on the / is portuguese wine grape that is used in / is a demo tape by / was first broadcast in syndication on / was a regional fashion entertainment and / is established under the / is an american record executive personal manager promoter and / was an american racecar driver . primarily / was a merchant and public official from / was a 4337-ton steamship built by / is a house at 1747 mooresfield road in / is located in / is a police station at
Concept 12 an institute of the romanian academy / cultivar raised at the university / a guide to british schools / a secondary school / college oxford university he reviewed poetry / an american college / a degree in chemistry / a mars-orbiting aeronomy probe / is the volleyball academy / taluka of amravati Concept 14
university . the first of the / school and it was released on / school during the early 1990s specifically between / school for grades 9-12 located in / school later returning to / school . she graduated from dartmouth college in 1998 with / schools . he studied law . he was admitted to / university also known as evolutionsmuseet / high located in / skole which covers løken momoen
Concept 15 bird of the arabian nights . / bird of prey in the falcon family . this bird / subfamily within the family euphorbiaceae . / hyena label in 2006 / faunistic studies and biogeography . / dolphins . he / mammal of africa / fossil collection in scandinavia / cat about to pounce . bruce
Concept 16 first listed in the / considered by the / identified by melville / listed on the / delivered to nearly 8200 homes monday through / built in 1959 and has undergone some modifications from the / a 1965 us number 1 hit single for herman / powered by a miller four-cylinder . it / the former first lady of new / launched on 25 february 1937 and commissioned on / built in 1932-1933 . house design was by john / an uninhabited steep rocky island west of the Concept 17 provides public bus transportation Concept 21
31 march 2014 pfzw / april 1983 – 30 december 1992 / april 27 1993 / july 6 1975 / 1933 to july 28 1934 / september 29 1962 / born april 1 1958 / birth registered october–december 1896 / born 5 august 1946 / november 18 1985 / december 29 1962
Concept 22 business since 1996 . / heliconiinae commonly called / name has been misapplied for / label in 1992 . / production created by / based in egremont / founded in 2005 / producer based in toronto who has / united states ) . / company thornaby-on-tees for burdick and cook london in / sweden established in / name could derive from Concept 23
newspaper comic strip reprints but later on original material / first described by / published his first description of it in / pornographic parodies of / was published in / newspaper reported in / romance novels from / published author himself as / newspaper reporter and editor and ( briefly ) a / motorcyclist magazine stated / a copy of / his novel on / first written notice about
Concept 24 a church established / an historical text / a building of the trinity church / an american beaux-arts architect / 145th state house / a victorian era admiral-class battleship / a house at / a tourist attraction Concept 27
single-engined jet . the company / light golden color / list of motorcycles scooters and mopeds produced / hits released only / bollywood film released / whaling ship called / trains indian army personnel / american film and television screenwriter best / american racecar driver / united states navy commander and / type viia u-boat of nazi germany ’ s kriegsmarine / submarine museum managed
Concept 28 gecko as they claim their cars stick / bird of prey in the falcon family . this bird / subfamily within the family / japanese phantom thief / eagle . the avon eagles / brazilian butterfly swimmer / monoplane with wire braced wings / subsidiary of sparrow health / bird species including ducks great crested grebes swans and geese Concept 29
largest icelandic commercial banks / is the only mare / is the pacific ocean / water by the spoonful / agua fria high school / karnik ( devanagari / represents the creuse ’ s 2nd constituency / a mars-orbiting aeronomy probe / is a large boulder / is an uninhabited steep rocky island west of the dingle / is situated on the lap of vindhya ranges
Concept 30 a top 50 company / is portuguese wine grape / the original soundtrack on the sony bmg label / an american disney comedy franchise / a free independent alternative newspaper / the statutory and regulatory organization / an american record executive personal manager promoter / an american stock car racing driver / a grain buyer / high-speed electrical multiple units / an artist-run centre / to supply drinking water Concept 31
was only newspaper comic / is a selection / was released in 1986 . this particular release / are 1993 pornographic parodies / is the third book / was the original dj / is the eleventh / is a survey launch / is the world ’ s tenth longest river
Concept 34 british pottery manufacturing firm / a collection of american singer laura / italian film . it stars actor gabriele / damned poet and somewhat heartbreaking images / american comic strip creator / poker player pioneer poker theorist / female bodybuilding champion and pop singer / scientist astronomer and botanist Concept 35
ponderosa steakhouse and / is a medium-sized long-winged species with / is an endemic australian genus of around / is a venomous tree-dwelling snake . / is a brazilian butterfly swimmer . rooted in / is a family of / is a saltwater pond in
Concept 36 a naive bayes / a bird of prey / an endemic australian genus / a venomous tree-dwelling snake / a young adult / an american sociobiologist / a brazilian butterfly swimmer / a large ruminant mammal / the largest and most important cave

heading: E APPLYING EDUCE PRINCIPLES FOR IMAGE CLASSIFICATION
text: Having assessed the relevance of our model on text data, we now turn to image data and explore if EDUCE is also able to extract meaningful concepts. In that case, the excerpts extracted in textual data are replaced by regions (or patches) in an image, and the recurrent neural architecture processing text is replaced by a classical convolutional neural network.
Principles: Each image x is first encoded through a convolutional neural network that output a representation in h(x) = RW×H×D where D is the number of output filters. Each vector hi,j,.(x) ∈ RD corresponds to a particular region s of the input image (i.e the receptive field of the convolutional neural network). Then each hi,j,. is used to compute pγ(s|x, c) through a linear model and a softmax activation as in Section 2.1 (step 1), and to compute pα(zc = 1|sc(x), c) (Section 2.1, step 2) here again with a linear model and a Bernoulli distribution. The semantics homogeneity is ensured by using an image classification network over the region captured by the receptive field in the previous convolutional neural networks that ensure that all regions captured from a particular concept ”look the same” and look differently from the regions captured for other concepts.
Experiments: To ensure the validity of our adaptation of EDUCE to images, we conduct very simple experiments on a dataset composed of 224 × 224 RGB images split in 3 categories: dogs, cats, and birds6 in equal proportion. We train on 3, 000 images and test on 3, 000 images. We build our model on top of a pretrained VGG-11 model Simonyan & Zisserman (2014) which is used for computing the intermediate representations h(x). Said otherwise, the output of the VGG-11 model is used for both detecting relevant regions and thus building the binary representation z, but also to ensure the homogeneity of extracted regions during training. Figure 5 shows extracted patterns and associated categories. Final classification performance is 91.6% with 10 concepts. In Figure 5 we plot extracted patterns for the 10 concepts and report in Figure 6a the weights of the final classifier. From these two figures, we can interpret the model’s behavior: concept 8 and concept 9 show what differentiates a dog from a cat or a bird, and support the classifier’s prediction of the dog category. Moreover, by looking at which regions are extracted by our model (Figure 6b ), we can see that EDUCE can focus its attention to discriminative regions of the images.
6We construct this dataset by combining random images from the Caltech Bird 200-2011 dataset (Wah et al., 2011) with images of the cats-and-dogs Kaggle dataset (Kaggle, 2013)
Bird
Cat
Dog
(a) Final Classifier weights, each column is a concept (0 to 9, from left to right) each row a final category. The lighter, the higher the weight is. Concepts 6 and 7 are associated to category ’Bird’, Concept 1 and 4 to ’Cat’ and concepts 8 and 9 to ’Dog’.
(b) Examples of classified images (column 2 and 4) and corresponding extracted patterns (column 1 and 3) for concept detected as present.
Figure 6: Interpretation of EDUCE on images.

Review summaries:
The paper introduces a new concept-based interpretability method that lies in the family of self-interpretable models (i.e. it's not a post-hoc method). Self-interpretability is achieved by a two-stage model: First, a concept-extractor finds the related pieces of consecutive words (excerpts) in a given text that are related to a concept among a set of given concepts (if any), then the model makes its predictions solely based on the presence or absence of concepts (binary).  The most useful part of the algorithm is that there is no need for concept annotations. The work then experimentally shows that their method, although does not outperform a non-self-interpretabile baseline but has better performance and interpretation compared to rival methods.

The authors proposed a self-explainable deep net architecture that could be used for text categorization. The main idea is to force the network to extract "excerpts", from the input text, each corresponds to a concept, which are also learned for interpretation. The classification is finally made based off of the learned concept, which is a binary vector. All three steps are learned in an end-to-end manner. The learning of concepts is regularized to make sure the concepts are consistent and non-overlapping. The idea sounds interesting and the experimental results support the usefulness of the proposed method on a variety of datasets. My sole concern is about the sensitivity analysis of the explanation, i.e. how robust is the explanation with respect to the perturbations that do not change the classifier prediction. It has been discussed in the literature that many explanation methods suffer from this sensitivity issue.