EDUCE: Explaining model Decision through Unsupervised Concepts Extraction
Authors: Diane Bouchacourt
Ludovic Denoyer
Venue:  (2020)
Keywords: Interpretability
explainability
text processing

Abstract: Providing explanations along with predictions is crucial in some text processing tasks. Therefore, we propose a new self-interpretable model that performs output prediction and simultaneously provides an explanation in terms of the presence of particular concepts in the input. To do so, our model's prediction relies solely on a low-dimensional binary representation of the input, where each feature denotes the presence or absence of concepts. The presence of a concept is decided from an excerpt i.e. a small sequence of consecutive words in the text. Relevant concepts for the prediction task at hand are automatically defined by our model, avoiding the need for concept-level annotations. To ease interpretability, we enforce that for each concept, the corresponding excerpts share similar semantics and are differentiable from each others. We experimentally demonstrate the relevance of our approach on text classification and multi-sentiment analysis tasks.

Under review as a conference paper at ICLR 2020
EDUCE: EXPLAINING MODEL DECISIONS THROUGH UNSUPERVISED CONCEPTS EXTRACTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Providing explanations along with predictions is crucial in some text processing tasks. Therefore, we propose a new self-interpretable model that performs output prediction and simultaneously provides an explanation in terms of the presence of particular concepts in the input. To do so, our model's prediction relies solely on low-dimensional binary representation of the input, where each feature denotes the presence or absence of concepts. The presence of a concept is decided from an excerpt i.e. a small sequence of consecutive words in the text. Relevant concepts for the prediction task at hand are automatically defined by our model, avoiding the need for concept-level annotations. To ease interpretability, we enforce that for each concept, the corresponding excerpts share similar semantics and are differentiable from each others. We experimentally demonstrate the relevance our approach text classification and multi--eentmmet analysis tasks.
INTRODUCTION
While deep learning models are powerful tools to perform a large variety of tasks, their predictive process often remains obscure. Understanding their behavior becomes crucial. This particularly true with text data, where predicting without justifications has limited applicability. There has been a recent focus on trying to make deep models more interpretable, see for example (Ribeiro al. Bach
Bach et al. 2015 SShrikumar et al. 2017: Simonyan et al.) 2014] |Sundararajan et al.) 2017) Specifically, methods have been recently proposed to provide such explanations sinultaneousty with the prediction. For example, (Lei et al. 2016 Yu et al. 2019 Bastings et al. (201)) select subsets of words in the input text that can account for the model's prediction (called rationales). this work, propose model that provides explanation based the absence or presence "concepts" that are automatically discovered in the texts.
Suppose we ask user to tell what is the category (or class) of the top text of Figure T (text x). She detects that the words the government said relate to a specific concept that is present in zr a concept she also detect in the words made official what in text x' Let call this concept "politics" in the remainder. She also notes that the words retails sales bounced back refer to a concept different from the previous one. Let call this other concept "economy". As the text is concerned with politics and economy she infers that its category is Business. Said otherwise, she detects excerpts that relate to particular concepts and decides on the text category based on the detected concepts.
Similarly, our paradigm assumes that an explanation of a model's prediction is understandable if it relies on few concepts, where each concept relates to parts of text (referred as excerpts) that are semantically consistent across multiple texts. Our methodology is as follows. First, our model encodes the input text into a binary -in-niieniional vector, where each value or computed from an excerpt and denotes the presence or absence of specific concept in the input. Then, the prediction made from this binary vector alone, allowing an easy interpretation of the decision in term of presence/absence of concepts.
While we set a maximal value the number of concepts (which is the dimensionality of the binary representation), the concepts are unsupervised and not defined a priori. The model automatically determines them in a way that eases interpretation: each concept is encouraged to be semantically consistent and not to overlap with other concepts. Therefore, extracted excerpts for concept must be discriminative of that concept only and share similar semantics. This is enforced through a concept consistency and separability constraint added as auxiliary loss in the learning optimization problem.

---

Under review as a conference paper at ICLR 2020
Pestix
hebalt sntes taurced sack di Los
[Sj Ratall sotes beunsed bads m 6,000
waes, the covrrneet cald
Tadte The gtatrminnl 1010
dassy
corneyy Augroving From
siccunner clump (M550
Vost
Wiite sings made ctticcaa war
A.0 the verts
Consopt
aothing arong sass Huure sade africcal 10
Cbea:: Spor
government 1u aass afficial Whar"
Conceptie
ai viite
ralango Mo wen's galr
tasts
por
Comorpt
pooneest
Step 1. Payse tott and sample one excerpt per concegt.
Fredictyy
Concept 6s
Superiiedd putput classifier
Unsupervised concept clsssitier
Figure 1: Illustration of EDUCE's prediction process on AGNews topic classification dataset (Zhang & LeCun 2015) examples. The different steps are explained in the text.
As a result, each discovered concept can be understood from the corresponding excerpts extracted that activate its appearance: in our previous example, the meaning of the first concept the user identifies is inferred from the excerpts she detected for that concept in z and x', i.e. the government said, made official what. Looking at these excerpts, we identify that concept as politics.
Our idea relates to Latent Dirichlet Allocation (LDA) (Blei et al., 2003). where text documents are described by a set of topics that are semantically consistent. However, LDA builds probabilistic model of text generation, whereas our goal is to discover and define latent concepts that are relevant for a prediction task at hand. In comparison to rationale- 2016: Yu et al.) 22199; Bastings et al. 2019), we rely different paradigm: model's prediction is based on the absence or presence of discovered concepts, and makes no direct use of the words captured excerpts. We do to ease interpretation of the prediction. This makes the interpretation of a different nature than these methods, and simpler to understand for the user.
text processing models (Lei et al.,
Our contribution with this work is new self- selfrrnterpeetable model that predicts solely from the presence/absence of concepts in the input. Concepts are learned unsupervisingly. and described by a semantically consistent set of excerpts. We experiment
three text categorization tasks and multi- aspect sentiment analysis task, compare our model's performance with state-of-the- prediction models, and demonstrate its interpretability. Note that an instance of our model for image processing is described in the supplementary Section E with illustrative experiments.
2 THE EDUCE MODEL
We present our model, called EDUCE for Explaining model Decisions through Unsupervised Con- cepts Extraction, using a multi-class classification task, but our method can be used perform regression multi-label classification. We consider a training dataset D of inputs (I1, &N), In E X and corresponding labels (y1..... MN). Un E V.
Contrarily to back-box models that use complex computations over low-level input features to predict the output class y. our objective is to enforce the model to map an æ input to an casy-to-interpret representation x(x) E 1) on which the output prediction relies. EDUCE's inference process consists in the following steps. Step 1: For each concept e (i.e. each dimension of (x:)), compute (s]a, c): the probability of each excerpt s in x to be selected. Sample a unique excerpt s=(x) Py (3| c) per concept (per dimension of z xxx)). Note that the same excerpt can be selected for multiple concepts. Step 2: Decide on each value ze (ar) of the representation =(x) by sampling from Pa (= c). Given sz (z)), ze (x) = means concept is detected as present and z (x)
0 means it absent. Step 3: Predict the output class y from =xx) (z1 .... (c(sc()))
We do not have concept-level annotations. To ensure semantic consistency and prevent overlap of the concepts, we jointly train a concept classifier to recognize, for every excerpt sx(x) such that =e (s{(x)) )=]] = 1, the concept (i.e. the dimension) it was extracted for: the label for each sa(r) is simply
e EE [1, CJ. Figure Iillustrates these steps, which we detail below.
2.1 GENERATING PREDICTIONS AND EXPLANATIONS
Step 1: Extract unique excerpt (z) per concept c. Given an input sentence Wi...., WMs where each W; is a word, a excerpt s consists of a span of consecutive words of flexible size between 3 and 10 words: Wk:-.. WKk:..,.444,,3 3 < 1 < 10. For each concept, we extract unique excerpt,
2

---

Under review as a conference paper at ICLR 2020
defined by its first and last word. Our extraction process is very similar to the one proposed in Question-Anse Antiering models (e.g Devlin et al. (2019). An excerpt is sampled by first sampling its start word and then its end word conditioned on the start word.
We denote the probability over each excerpt s to represent concept c in x as (s|z,, c) and we write as the product of (i) Pstart (Wrk [z; c), the probability for s's first word to be the start word of the excerpt to extract and (ii) Pstop! [144122 c, W&) the probability of w/+t to be the stop word given the start word wa: Py py(s|2, (s|2r, c) = p., (u(Wu,
Wk++t [x, c) Pstop Pstop(wu+.]le (Ww++//;; (E, c, WF (ottrr (wrx [2, c). We parametrize Pstort (2, e) and Pstop (ottop[xx=:/0 22, C, wb.) using recurrent neural networks. First we feed the entire input sentence W1, .... WM through a bidirectional LSTM, and we represent each word we in the sentence as the concatenation of the forward pass and backward pass hidden states for that word: hg [h,, hg]. Vk E [1, M). Second, each h& fed linear layer with parameters Ystart which outputs a score for wk, followed by the softmax activation function over all possible words to have the probability distribution over each word to be the start word:
cxp(Tsstrrt hx)
Petart (uxar( c)
k== exp(Tstart h2) k = 1, M.
Using this distribution, can sample specific start word Wstart Pstart (Wk/2z c). Third, feed again the vectors h2 to another linear layer with parameters Ystop that gives a score for each word to be the stop word. We mask out these scores before taking the softmax, such that the probabilities for Wstart+t with < 3 and > 10 to be the stop word are O.
exp(7stop" hatartt()
12 or 1 S 10
E1 (Tsto" Nstart++1) Pstop [Wop[[Wstaae Stttrt C, Wstart)
1 10.
We then sample a specific stop word Wstop Pstop (Wstert+1 /2, c, Watort [ We now have the start word Wstart and stop word Wstop- The corresponding excerpt is Wstarts ... Wstop and is represented as a fixed size vector (xx) by computing the average of (pre-trained, fixed) words embeddings vectors of Wstart; Wstop- Hence sampled excerpts sc(x)) belong to Rd, where is the dimension of embedding vectors.
This process is illustrated in Figure For text æ, the excerpt Retails sales bounced back is selected for concept C1, the government said for concept C2 and midsummer slump for concept C3-
Step 2: For each c, from the excerpt sc(x), decide on the value ze(xx) denoting the pres- ence/absence of concept Each extracted excerpt (one per concept) is processed to decide on the absence or presence of each concept in x. Specifically, for each concept we take the dot product of weight vector Or E Rd with (x)) followed by sigmoid activation function, in order obtain the Bernoulli probability
(z [)) ala. - ss(())
This is the probability that sz(x) extracted from æ for concept e triggers the presence of concept c. The binary vector (z) is obtained by independently sampling each dimension: Ve ze Pa (e (z).
This step can be seen as C independent binary classifiers, each of them deciding on the presence of particular concept. We illustrate this in Step 2 in Figure For text x, the excerpt Retails sales bounced back activates the presence of concept C1, the government said activates the presence c2 but a midsummer slump does not activate the presence of C3.
Step 3: Predict from z(x). As shown in Step of Figure I given an input x, the prediction of the output y is made solely from the intermediate binary representation =(x). We use linear classifier without bias, parametrized by weight matrix 5 E RDI followed by softmax activation function, returning po(vi Vy E y
Output classification training objective. The parameters a,,8, are learned in end-to-end manner by minimizing cross-entropy, which writes for each x as:
cutput (I, y. a, 7) = E Ex(z)~p,
[C Contput (y. (x))
Eve se (=))) Eye zo (t (2.1).
log pa (s)) (3)
We give details about the optimization in Section 2.3
We also prevent stop words be outside of the text length.
3

---

Under review as a conference paper at ICLR 2020
2.2 UNSUPERVISED DISCOVERY OF LATENT CONCEPTS
To ease interpretation of the concepts' meaning, we want each dimension 2 to correspond to semanti- cally consistent concepts. In other words, for given concept (i.e. a given dimension z.), for all inputs where that concept is present (i.e. Vr s.1. =x)) 1), the corresponding excerpts sx()) should share common semantics. While the C binary classifiers deciding on the presence of each concept с (Step 2) enforce consistency within the set of extracted excerpts Se, there is no guarantee that the concepts are not overlapping (i.e. are separable). We want concepts to be separable so that each concept captures particular notion. Excerpts extracted for concept must be distinguishable from the excerpts extracted for another concept c'. If we take the example of Figure In Step 2 the excerpt Retails sales bounced back is identifiable as relating to economy/finance, contrarily to the government said, made official what which trigger the concept politics.
We rephrase our desiderata for consistency and separability as classification of the excerpts, with as many classes as the number of concepts. An external classifier should recognize that elements in the set {s (sx(x)\V zz()x(z)) }} belong to concept and {s (x) (x) 1} belong to concept e' and not c. Recall that we do not have labels for the concepts. The ground-truth label of each extracted excerpt (x) is the index of the dimension of z(z) for which it extracted. We learn linear concept classifier without bias, parametrized by weight matrix 0 R igliixd followed by a softmax activation function, returning po (c' (c/().())))))))))))) (x)),, ve E C].
The concept classifier is trained by minimizing cross-entropy, where the label of each excerpt sa(x)) the index of the concept for which it was extracted (i.e. c):
Cooncept (2,8,a,7) 7)
Ea(a)~n, (z) Commenyt =Ea(s)>pp (x
(4)
2e 2<(ss(x)) (ogpor)))ogpo((sss())))))))
where s(x) refers to the set of excerpts extracted for each concept: s(x)
C).
The loss considers concepts that are present in x (i.e. ze(sc(r)) = 1). The role of the concept classifier is to further enforce concepts consistency and to prevent overlap of concepts, and is jointly train with the rest of the model.
Note that adding sparsity constraint the number of concepts present in the inputs enforces semantic consistency, as sparse coding has been shown to induce useful, interpretable representations (Bengio al.] 2013a Mairal al., 2010). We experimentally demonstrate that sparsity constraint is not sufficient, and can harm output prediction performance. On the opposite, with our concept classifier, sparsity is encouraged Leoncept depends on the concepts that are present, but if concepts are consistent and separable conneppt can be low without harming task performance.
2.3 EDUCE OBJECTIVE FUNCTION AND OPTIMIZATION
We jointly learn the concept classifier and output classifier and our objective function is the sum Equations 3 andd
[(x) = E.(s)<p, ( (C (y, x(x),)) + X (x(x), s(z),
[Coutpat
(5)
where A controls the strength of the concept consistency constraint wrt the output prediction. The loss L(x) is differentiable, therefore the gradients with respect to the output classifier's weights (6) and concept classifier's weights (!) can be computed and back-propagated. However, the gradients with respect to the parameters of the excerpts extraction (7) and the Bernoulli distribution over presence of concepts (a) pose an issue due to the sampling of these discrete random variables in Steps and As the explicit computation of the expectation involves expensive summations over all possible values of sax: and 2ar), we resort to CooneeCCarlo approximations of the gradient (Sutton Barto 1998), with the loss L(x) used as reinforeement signal. The gradient derivation is provided in Supplementary Material. Our code will be released upon acceptance.
Note that the concept classifier encourages concept consistency and concept separability. The discovered concepts are not guaranteed to align with human defined concepts. This is a general challenge in learning explainable//nterrrtabbe models in an unsupervised setting. We believe it is reasonable assumption to employ a linear concept classifier over pre-trained word embeddings such that it linear separates the word embedding space.
4

---

Under review as a conference paper at ICLR 2020
3 RELATED WORK
Our work relates to topic models, especially to Latent Dirichlet Allocation (LDA) (Blei et al. 2003 where each document is modeled as a mixture of topies and each word relates to one or more topics. While LDA has been recently combined with recurrent neural networks (Zaheer et al. 2017), the methodology remains different. LDA learns the parameters of probabílistic graphical model of text generation while our goal is to build an interpretable prediction model that relies on binary presence/absence of concepts.
Some existing works interpret an already trained model, typically using perturbation and gradient- based approach, (Ribeiro et al. 2016; |Bach et al.) 2015: Shrikumar et al., 2017; (Sioonyan et al.) 2014: Sundararajan et al. (017]) Alvarez-Melis & Jaakkola (2017) design model that detects input- output pairs that are causally related. Contrarily to these methods, our work falls in the domain of self-interpretable models, which produce explanation simultaneously with their prediction, Lei jet al. (2016); Yu et al.] (Bstings et al. (2019) develop interpretable models for NLP tasks by selecting rationales, i.e. parts of text, on which a consequent model bases its prediction. A rationale acts as a justification supporting the prediction and is different from our definition of excerpt. In EDUCE, excerpts support the attribution of presence of concepts, and this attribution supports the prediction. Goyal et al. (2019) propose visual explanations of a classifier's ' decision and Alaniz & Akata 2019) use an obssrver-llassffee pair model, whose prediction can be exposed as a binary tree. Contrarily to ours, their model does not provide local explanation based on parts of the input. Quint ct al. 2018 extend a classic variational outo-encoder architecture with a differentiable decision tree classifier. However, their methodology is different and they only experiment on image data.
Most related to our work are methods providing concept-based explanations. Alvarez Melis & Jaakkola (2018) learn self-explainable classifier that takes as input set of concepts extracted from the original input. They define a set of desiderata for what is an interpretable concept, but simply represent extracted concepts as encoding of the input learned with an auto-encoding loss. Kim et al. (2018) learn concept activation vectors, and Zhou et al. (2018) build on Bau et al. (2017) to propose a method that generates visual explanations of classifter, in the form of semantically interpretable components with a corresponding label and heatmap. However, in these models, the final classifier is already trained and fixed while EDUCE is learned end-to-end. More importantly, these works need concepts to be predefined from human annotations, while EDUCE is unsupervised. Ghorbani et al. (2019) propose a method based on unsupervised concept discovery to explain a model a posterioni We differ from their work as EDUCE is trained end-to-end to classify solely based on the discovered concepts, which presence is discriminant, resulting in a fully interpretable model. Finally, we focus on text data while the aforementioned works experiment on image data.
4 TEXT CLASSIFICATION EXPERIMENTS
We experiment on three text classification datasets. The DBpedia ontology classification dataset (Zhang et al. (015)) was constructed by picking 14 non-overlapping categories from DBpedia 2014 4 (Lehmann et | 2015). There are 14 classes. We use 56, 000 examples of the train dataset for training, and 56, 000 for validation. For testing, we use 7, 000 examples of the test dataset (using stratified sampling). Also in Zhang et al. (2015). the AGNews topic classification dataset was constructed from the AG dataset's s largest categories. There are 4 classes. We separate the training set into 84, 000 training samples and 24, 000 validation samples and report results on the full test dataset. We also experiment on the Stanford Sentiment Treebank (SST) (Socher et al. (203)) that includes 5 sentiment classes. For DBPedia and AGNews we use fixed word vectors trained on Common Crawl (Grave et al. 2018). For SST, we use fixed Glove word vectors (Pennington et al. 2014).
Comparative models. Since we are concerned with building self-interpretable models, we do not compare with the methods presented in Section 3 that explain a model a posteriori. Rather, for comparison, we report the performance of (i) using a Bidirectional LSTM on the full text and predict directly from its hidden state (Baseline) (ii) the No Concept Loss model that uses binary discrete intermediate representation but no concept classifier, it corresponds to À = 0 and its training objective is simply Coutput and (iii) adding T. Jormorm sparsity constraint to the No Concept Loss model (referred to as No Concept Loss + L1), i.c. FJchange to:
C(x) = Ea(n)~<p EZ (=)xppp [Cotpput (y, x(x),8) + XL, |=]].
(6)
Therefore both No Concept Loss and No Concept Loss + T. models extract excerpts and use
5

---

Under review as a conference paper at ICLR 2020
A Posterioni
Data
Model
Output Acc. (%)
Concept Acc. (%)
DBPedia
EDUCE
97.0 0.1
82.4 + 0.8
No Concept Loss
97.4 0.1
25.9 + 0.6
No Concept Loss+1.1
96.5 0.2
44 2.6
Baseline
98.75 + 0.0
n/a
AGNews
EDUCE
97.50.02
78 + 6.5
No Concept Loss
88.2 0.1
31.0 + 0.7
No Concept L1
86.3 + 0.7
56 + 3.2
Haseline
92.08 + 0.1
n/a
Table 1: Test performance on DBPedia and AGNews (mean + SEM).
binary intermediate representation, but are trained without concept loss.
Hyperparameters selection. We tried different values of hyperparameters (details in supplementary Section B.I and for each set of hyperparameters. we run 5 different random seeds. We choose the best hyperparameters set using its average validation performance over the 5 random seeds.
Metrics. We report the test output accuracy (Output Acc.) over the task. Evaluating the relevance of discovered concepts is challenge in the unsupervised setting. To measure how well the discovered concepts align with human defined concepts one would need human experiments, which we consider as future work. However, we evaluate how consistent and separable the discovered concepts are, as a proxy to measure how easy it would be for a user to interpret them. For EDUCE, we can compute the accuracy of the concept classifier on the test data. However, this should be low for models No Concept Loss and No Concept Loss L1 that do not train with a concept loss. Therefore, to evaluate interpretability, we report an a posteriori concept accuracy: after training, for each model (including EDUCE), we gather the excerpts extracted (ze- 1) in the test data. We separate these excerpts into two sets (training and testing, note that these are both generated from the test data). We train a new, separate linear concept classifier a posteriori on the extracted excerpts to evaluate and we report its performance as a posteriori concept accuracy (A Posteriori Concept Acc.).
Quantitative results on DBPedia and AGNews. Table reports test performance the DBPedia and AGNews datasets for C' = 10 concepts. We report results with values À = 0.1 and Xr, = 0.1 that give best trade-off performance between output and concept accuracies. Complete results are available in supplementary Table E For all metrics we report the mean and standard error of the mean (SEM) over the training random seeds. Table Sable[[shrr shows that the Bascline model outperforms the other models that all employ a binary discrete encoding of the input. This is expected, and shows the trade-off between interpretability and output accuracy. terms of output accuracy, EDUCE is comparable with its counterpart that does not encourage concept consistency (No Concept Loss), loosing only 0.4% on DBPedia and 0.7% on AGNews, and as expected outperforms in terms of concept accuracy. Adding an L constraint to the No Concept Loss model increases the consistency of the concepts (as per A posteriori Concept Accuracy values) yet is largely outperformed by EDUCE on that metric, and output accuracy is also lower than EDUCE's.
Interpreting EDUCE on AGNews. We turn to show how EDUCE's ' output prediction is easily interpretable. The following results were generated with X = 0.1 and C' 10 concepts. Figure 2a shows examples of the AGNews test set that were correctly classified by EDUCE. The underlined words correspond to the excerpts extracted for different concepts. Separately, in Figure 2b we show, for each concept detected in the examples of Figure 2a examplar excerpts extracted from others test documents (excerpts separated by "/"). We interpret the concepts' meaning follows: concept 0 maps to informatics notions, concept 2 to corporations/petrol, concept 4 to the notions of investments and finance, concept 6 to sports events and concept to governmental/statt affaires. Concept is less clearly defined. Note that in Figure 2b excerpts are consistent yet are extracted in texts from multiple output classes. The excerpts extracted the examples of Figure 2a are consistent with these interpretations. These results show how easily the classification of any text can be explained by the detection of multiple, relevant, and intelligible concepts. More qualitative examples are in supplementary Section C Note that supplementary Table 10 shows the concepts extracted by the No Concept Loss model, we see that they form less semantically consistent units and are hard to interpret. Supplementary table shows test examples where the final classifier's prediction was incorrect. In some of these, the predicted class could be considered as an appropriate label, and importantly the concepts that are identified as present in the text are relevant with the set of discovered concepts.

---

Under review as a conference paper at ICLR 2020
Class Business: sports retailer ijb yesterday reported near 25 drop in profits and continuing poor sales. and ended shareholders #39 hopes of a takeover by announcing that a potential bidder had walked away- Class World: bangkok, thailand sept. 30, 2004 millions of volunteers led by emergency teams fanned out across thailand on thursday in a new drive to fight bird flu after the prime minister gave officials 30 days to eradicate the epidemic.
Class Sports: the spanish government responded to diplomatic pressure from britain yesterday by starting search for fans who racially abused england players during a quot friendly quot football match with spain.. Class Sci/Tech: los angeles reuters )- group of technology companies including texas instruments inc &lt ixn. n&gt, stmicroclectronics &it stm. pa&gt and broadcom corp &lt brem. o&gt, , thursday said they will propose new wireless networking standard up to 10 times the speed of the current generation.
(a) AGNews test examples correctly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.
Concept
software services giant / moonwalk to home video display chip / downloading music. upcoming my prerogative video / his ever-growing swimming / launch of a video display chip/ illegality of downloading
Concept
Concept
oil market. / oil giant sibneft/ oil prices and / corp &lt brem
Concept 4
cash settlement of up to #36 50 million/ six-year deal worth about $40 million / the dollar dipped to four-week low against the euro / five shares
Concept
olympic 100-meter freestyle/ sox family/ olympics should help athletes were already
Concept 8
frail pope john paul / indian army major shot unions representing workers / goverment
representatives.
(b) Examples excerpts that are extracted accross the test set, corresponding to the concepts detected in Figure
2a Colors match the colors used in Figure Za
Figure 2: Interpretation of EDUCE prediction through concept analysis.
SST dataset Figure 3d shows output accuracy versus a posteriori concept accuracy on the SST dataset, for different different number of concepts C' and different values of X: Each marker is a different value of X, and values are (from left to right markers) {0, 0.01, 0.1, 0.5}. We do the same for the No Concept Loss + L1 model. Note that the left most markers in all curves correspond to A = 0 and Åτ. = 0, therefore to the model referred to as No Concept Loss. While our goal and model is very different than Lei et al. (2at6)/Bastings [Bastings et al. (2019), we report their test scores (obtained using 40% of the text on the SST dataset, we report directly from Bastings et al. (019)). The a posteriori concept accuracy for Baseline, Lei et al. (2016) and Basttings et al. (2019) is not applicable. Figure 3d confirms our results: EDUCE (in blue, purple and red) overperlorms the model No Concept Loss L (in green) on both output and posteriori concept accuracies, and achieve output accuracy comparable to Lei et al. (2016) (orange line) while having 60% a posteriori concept accuracy. Regarding the effect of the number of concept C, for a fixed C, choosing the adequate value of X allows the user to balance between final accuracy and concepts' accuracy. Similar performances can be achieved by different number of concepts: C' = 20, À 0.1 (dotted purple line, squared marker) achieves 41.5% final accuracy and ~ 70% concept accuracy, while C' = 5, A = 0.01 (plain blue line, diamond marker) achieves 42.0% final accuracy and 60% concept accuracy. The smaller the number of concepts, the smaller the value of À which is expected as with less concepts the task of the concept classifier easier.
Figure 3a shows the empirical frequency of presence of each concept, per output class in the SST dataset, using À 0.1 and C = 10 concepts. We see that concepts 3, 5 and 7 are often triggered when a sample is positive or very positive, and on the opposite concept 8 is present in negative/very negative texts. The values of weight matrix of the output classifier in Figure (3b proves that the presence of these concepts is responsible for the output classifier's 's prediction. We can further analyze the concepts: Each word in SST sentences is annotated with a label for the sentiment it expresses, referred to word label. We do not use them during training validation, but at test-time this allows us to analyze the repartition of words labels in the set of excerpts extracted for each concept. Figure 3c shows that indeed the excerpts selected for concepts 3, 5 and are mostly composed of positive or very positive words, and the excerpts extracted when concept 8 is triggered are mostly composed of negative or very negative word
We do not show the amount of neutral words selected squeezes the histograms.
7

---

Under review as a conference paper at ICLR 2020
Treq. per category
Wrights
very negative
negetive
regative
negalive
neutral-
noutral
post lve
very pasitive
very positive
Concepts
Concapts
(a) Per class concept frequency.
(h) Output classifier weight matrix 5.
Per cencept word izoels repartition
CDUKE, CHS
Goncept
Culpet Acc (6)
Numberr of worde
70c
Concept
Concept
REUKA,
eoc
Linceept
soc
acc-
Dulling dt (2823)
al (Aun))
30c
200
10c-
very regabive negative
positive
very positive ve
70
Word andts
Pootertori
(c) Repartition of words labels for concepts 7,
(d) SST test performance, output accuracy vs poste- riori concept accuracy. A posteriori concept accuracy is not applicable for Baseline, [Lei al. (2016) and Bastings el al. (2019).
Figure 3: Results on the SST dataset. MULTI ASPECT SENTIMENT ANALYSIS EXPERIMENT
BeerAdvocate dataset While our paradigm is different than Lei et al. (2016); Bastings et al. (2019), also perform a multi-aspect sentiment regression experiment on the pre-processed subset of the BeerAdvocate dataset (McAuley et al. 2012). consists of 260, 000 beer reviews where ratings for four aspects (look, smell, palate and taste) are given, well an overall rating. The reviews are separated into training/validation validation sets specific to each aspect, and ratings are mapped to scalars in j0, 1]. This is a regression task, so we modify the Baseline model and EDUCE in the last layer of the prediction with a sigmoid activation function, and use Mean Squared Error (MSE) instead of cross-entropy for Coutpid (y. (xx), 5). McAuley al. (012)) provided a test set (994 reviews) with sentence-level rationale annotations: different parts of each review are annotated with one (or multiple) aspect label, indicating what aspect it covers (referred to as gold rationales). Importantly, we do not use these gold rationales during training or validation, but use to evaluate our model. Hyperparameters selection is similar to our classification experiments.
We train EDUCE on the prediction of the 5-values vector at once (4 aspects and the overall score, there are therefore scalar ratings to predict) using the 260, 000 reviews. We compare EDUCE's performance with the Baseline that accesses the full text. The test MSE is 0.0089 = 0.0 (mean and SEM accross training seeds) for the Baseline model and 0.01 19 + 0.0 for EDUCE using À = 0.01 and C 10 concepts, with corresponding concept accuracy of 92.8 + 0.7. Of specific interest us is the per-concept precision of gold rationales and the total percentage of selected words. Indeed, Table shows that when predicting the 4 aspects and the overall score, some concepts are capturing a specific aspect: for example, the precision of Concept 7 on the Appearance aspect is 97. 97.09% This confirmed by Table 2 that shows that excerpts extracted for Concept 7 are mostly related to the color and head (the frothy foam on top of beer) of the beer" Supplementary Table 5 reports examplar excerpts for all concepts.
As in Lei et al. 2016);; [Bastings et al. (2019) we also train EDUCE on the prediction of each of the first three aspects separately. Supplementary Taalle/repon the precision of gold rationales for each aspect. While the results are less convincing than the models specifically designed for rationales extraction, extracted excerpts match gold rationales to a certain extent.
hhtps://www. www.heeaddyoatt.cooom commattcccccm
"aalles 3 and reports results of the best seed for the cross-validated hyperparameters.
8

---

Under review as a conference paper at ICLR 2020
Concept 0) fruity esters, and/ fruits, caramelized pecans, , and/ tof- fee and caramel accents coffee and chocolate flavors/ earthy hop resin..
Concept Appearance
Smell
Palate
Taste
Concept
0.85
55.75
1.79
40.60
Concept
0.54
5.48
11.70
30.58
Concept 2
creamy and a/ chewy and rich and drinkability/ creamy mouthfeel that/ smooth and just velvety on the/ thick
Concept
1.22
0.90
78.18
15.03
Concept
1.46
92.39
0.30
5.34
and
Concept
45.33
0.23
0.26
0.16
Concept 3
rich malt scents/ aroma is quite hoppy with big citrus/ tons of different sweet malts, toffe/ smells extremely roasty/ boom of grapefruity
Concept
7.27
22.50
2.27
28.86
Concept
0.36
5.86
6.40
28.73
Concept
97.09
0.98
0.45
0.53
Concept 7 good head and lacing/ beautiful golden-amber color/
Concept 0.18
0.09
18.00 88.07
5.93
creamy tan hcad/ nice doep brown color/ proud head
Concept 9
0.00
0.00
68.42
10.53
has settled.. nothing
Concept 9
carbonation graceful/ drinkability is excellent/ mouth- feel is wonderful/ mouthfeel is exemplary/ drinkability excellent
Table 3: Per-concepl precision of gold rationales (in % for each aspect), trained to predict all 4 aspects and the overall score. In bold cepts which precision is above 50% for aspect's 's gold rationales.
emphasize the con-
Table 2: Examples of excerpts extracted on the Beer test set.
DISCUSSION AND PERSPECTIVES
We propose new self-interpretabbl model, EDUCE, that maps inputs to easy-to-interpret rep- resentation on which the output prediction relies. We experimentally demonstrate its ability (i) to extract semantically meaningful concepts described by consistent excerpts, and (ii) to ground its output prediction on the presence/absence of these concepts in each input. EDUCE generates relevant explanations while retaining high predictive performance. As future direction for research, we contemplate using hierarchical representations, as human notions can often be represented with hierarchical structures, for example in WordNet (Miller [995). Furthemore, our principles can be applied on different types of inputs like images, and we provide initial experiment in Supplementary Section +
REFERENCES
Stephan Alaniz, and Zeynep Akata. XOC: explainable observer-classifier for explainable binary decisions. CoRR, ab2/1900.,0 2019. htt.:::pp://aaxii.orr///a/////// 01780
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7775-7784. 2018.
David Alvarez-Melis and Tommi S. Jaakkola. A causal framework for explaining the predictions of black-bor box sequence- eequence-to-sequene te-seqeence models. In EMNLP, pp. 412-421. Association for Computational Linguisties, 2017.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Claus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7):e0130140, 07 2015. doi: 10.1371/joumall pone. 01 40.
Joost Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables, 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 Honolulu, HI, USA, July 21-26, 2017, pp. 3319-3327, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8), August 2013a.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. abs/1308.3432, 2013b. URL http:
/arxiiv. 3

---

Under review as a conference paper at ICLR 2020
David M. Blei, Andrew Y. Ng, Michael I. Jordan, and John Lafferty. Latent dirichlet allocation. Journal of Machine Learning Research, 3:2003, 2003.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Fre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Compulational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), 2019.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations, 2019.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations, 2019.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Kaggle. Kaggle dogs vs cats dataset, 2013.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Lear pp. 2673-2682, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. DBpedia a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web Journal, 6(2):167-195, 2015.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107-117, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 20.18653/1/D116. 1ttp//////110000 1.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online learning for matrix factoriza- tion and sparse coding. JMLR, 11, March 2010.
Julian McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi-aspect reviews. In Proceedings of the 2012 IEEE 12th International Conference on Data Mining, ICDM 122 pp. 1020-1025, Washington, DC, USA, 2012. IEEE Computer Society. ISBN 978-0-7695- 4905-7. doi: 10.1100//CDM.22022.11..
George A. Miller. Wordnet: A lexical database for english. Commun. ACM, (1:31)8119944444 41, 1995.
ISSN 0001-0782.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. 2014.
Eleanor Quint, Garrett Wirka, Jacob Williams, Stephen Scott, and N.V. Vimodchandran. Interpretable classification via supervised variational autoencoders and differentiable decision trees, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the predictions of any classifier. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16. ACM, 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. CoRR, abs/ 70s//170.000 02665, 2017.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409, 1009.1550 1556, 2014.
10

---

Under review as a conference paper at ICLR 2020
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. ICLR Workshop Track, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
Richard S. Sutton and Andrew Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSI) Dataset. Technical Report
California Institute of Technology, 2011.
Mo Yu, Shiyu Chang, and Tommi $ Jaakkola. Learning corresponded rationales text matching, 2019. URL http:ptttt/:::peeeeeeet
wtt.:noummtfdrrrooormommmm 99t
Manzil Zaheer, Amr Ahmed, and Alexander Smola. Latent Istm allocation: Joint clustering and non-linear dynamic modeling of sequence data. 2017.
Xiang Zhang and Yann LeCun. Text understanding from scratch. CoRR, abs//500.001700 2015. URL http::/arxiv. wttgga/b/////////
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classifi- cation. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume I, NIPS' 15, pp. 649-657, Cambridge, MA, USA, 2015. MIT Press.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In ECCV, 2018.
A DETAILS ON THE LEARNING ALGORITHM
As mentioned in main paper, when computing Equation the the cross-entropy losses of the output classifier and concept classifier are differentiable, therefore the gradients with respect to the weights 5 and / can be computed. However, the gradients with respect the parameters of the LSTM (7) and the Bermouilli distribution over presence concepts (ax) pose a difficulty because we sample Ve (xx) py (s|z, c) and Vc ze ~ (z (x),
We use straight-through estimator (Bengio et al. 2013b in the backward pass for the parameters o, and resort to Monte-Carlo approximations of the gradient employed in Reinforcement Learning (Sutton & Barto 1998) for the parameters ?. The loss C(x) serves as reinforcement signal. We weight the two terms of the gradient using parameter as follows:
V Vo
a. + rB., [[((z) - b, (og
(7)
where r controls the strength of the Reinforcement Learning term and b the average of the loss, used as control variate.
B EXPERIMENTAL DETAILS AND DETAILED RESULTS.
We monitor validation performance and early stopping. For the comparative models monitor the output prediction accuracy. For EDUCE we monitor the sum of the output prediction accuracy and the concept classifier accuracy (weighted by the parameter X).

---

Under review as a conference paper at ICLR 2020
B.1 HYPERPARAMETERS CONSIDERED
In all experiment, we increase the value of À by 10% every epoch. In order to choose the best set of hyperparameters, we take the average value on the 5 random seeds.
DBPedia and AGNews We try the following ranges of hyperparameters À E {0.0, 0.001, 0.01, 0.1}. {0.01, 0.1}. learning rate (0.0005, 0.001 The hidden state of the LSTM is of size 200. We use Adam optimizer (Kingma & Ba 2014), batches of size 64. For the Baseline model, we try adding weight decay {0.0, 6] and dropout {0.0, 0.1,0.1,.00 0.2).
AL1 {0.0, 0.001, 0.01, 0.1, 1].
SST We try the following ranges of hyperparameters À E {0.0. 0.001, 0.010 0.13. AL1 {0.0, 0.001, 0.01, 0.1, 1). {0.01, 0.1}, learning rate
{0.0002, 0.001} and weight decay in (0.0, e - 6}. The hidden state of the LSTM is of size 150. We use Adam optimizer (Kingma Ba 2014p, batches of size 25. For the Baseline model, we follow Bastings et (2019) and use weight decay 1e- 6 and dropout {0.5}.
BEER We try the following ranges of hyperparameters À {0.001, 0.01, 0.1), r {0.001, 0.01, 0.1J, learning rate I = {0.000 f0.0005, 0.001 The hidden state of the LSTM is of size 200. We use Adam optimizer (Kingma & Ba 2014). batches of size 256. We also clip the gradient to the value of 1 and employ entropy maximization regularization
mean of the Bernoulli distributions Ve. Po.Poa( (1))8)())) c). weighted by 0.001. We train all models for 250 epochs and use early stopping on the validation as in BBastings et al. (2019) (Ir = 0.0004, hidden dimension 200, dropout in (0.1, 0.2) and weight decay le 6) and train for 100 epochs.
For the Baseline model the BEER dataset, we the same hyperparameters
B.1.1 DETAILED RESULTS ON AGNEWS AND DBPEDIA
Table details output accuracy, a posteriori concept accuracy, and sparsity (number of concepts presents per input) for different values of X and Åτ. considered.
A Posteriori Concept Acc. (%)
Dataset
Model
Output Acc. (%)
Sparsity
Baseline
98.75 + 0.0
n/a
n/a
No Concept Loss
97.4 0.1
25.9 + 0.6
5.51 + 0.T
DBPEDIA
EDUCE À 0.001
97.3 1 0.1
28 1 1.7
5.60 1 0.1
EDUCE 0.01
97.310.
39 3.5
5.39 1 0.I
EDUCE 0.1
97.0
82.4 0.88
3.5 1 0.2
No Concept Loss + L1, AL, = 0.001
9.14
27 1.9
5.5 - 0.1
No Concept Loss + L1. dL, = 0.01
97.46 + 0.1
28 2.1
5.12 + 0.0
No Concept Loss L1, dL, 0.1
0.2 96.5
44 2.6
3.1 0.1
No Concept Loss L1, XL, T.00
7.
1.31 E 0.0
Baseline
92,08 + 0.1
n/a
No Concept Loss
88.2 0.120011
31.0 = 0.7
5.2 5.200.2
AGNeWs
EDUCE À = 0.001
88.4
33 + 1.6
5.2 0.2
EDUCE À 0.01
88.6 3.6=0011
40
4.6 16600 0.2
EDUCE
0.1
87.5 0.2
78 6.5
2.4
No Concept Loss + L,, dr.
= 0.001
88.3100
31.6
0.2
No Concept Loss + L₁, Xt, = 0.01
88.7 - 0.2
3111. 1.4
1.6 1 0.2
No Concept Loss L1. AL; 0.I
86.3 0.7
56 5613.2
2.2
No Concept Loss L1. Ar,, 1.00
53 + 3.2
89 8912.
0.50 0.0
Table 4: Test performance on DBPedia and AGNews (mean + SEM) for all values of X and \L,
B.2 ADDITIONAL RESULTS ON BEER
As in Lei et al. (2016); Bastings et al. (2019), also train EDUCE the prediction of each of the first three aspects (the 4th aspect they do not experiment on 5 Table 6 reports the precision of gold rationales for that aspect and the total percentage of selected words, values for Lei al. (2016): Bastings et al. (019)) are taken from Bastings et al. 2019). We use C = 10 and = 0.1.
Bastings 2019) call the third aspect Taste hut they actually refer Palate.
12

---

Under review as a conference paper at ICLR 2020
Concept
fruity esters, and/ fruits caramelized pecans, and/ toffee and caramel accents coffee and chocolate flavors/ earthy hop resin.
0
Concept
delicious and with/ enjoyed this beer/ damn good beer/ nice stout that/ good beer to
Concept
creamy and a/ chewy and rich and drinkability/ creamy mouthfeel that/ smooth and just velvety on the/ thick, and
2
Concept
rich malt scents/ aroma is quite hoppy with big citrus/ tons of different sweet malts, toffe/ smells extremely roasty/ boom of grapefruity
3
Concept
octoberfest: say the word and you probably/ 2004 vintage pours/ 12 ounce bottle acquired
4
in trade/ hate to be the guy/ 12oz bottle best before 9/09, sampled 5/16/09
Concept
which fantastic/ delicious/ is intensely packed/ and a lovely/ builds wonderful
5
Concept
m really digging/ a damn good/ a super fresh growler/ the better abitas/ a great traditional
6
Concept
good head and lacing/ beautiful aabber color/ creamy tan head/ nice deep brown color/ proud head has settled, nothing
Concept
easy to drink/ hot summer day/ drinkability easy/ easy to down/ can easily turn this 'strong ale into session
8
Concept
carbonation is graceful/ drinkability is excellent/ mouthfeel is wonderful/ mouthfeel is exemplary/ drinkability: excellent
Table 5: Examples of excerpts extracted on the Beer test set.
Model
Appearance
Smell
Palate
Prec.
Extr.
Prec.
Extr.
Prec.
Extr.
EDUCE
81 + 4.9
+ 0.5
+ 3.1
4.8 + 0.4
50 + 6.6
4.1 - 0.6
Lei al. (2016)
96.3
14
95.1
7
80.2
Bastings 2011
98.1
96.8
89.8
Table 6: Precision (Prec., in %) of gold rationales and percentage of extracted text (Extr. in %) when trained separately on each aspect.
We consider a word selected if any of the concept selects and count one selected even it is selected by multiple concepts. While the results are less convincing than the aforementioned rationales models that are specifically designed for this task, we see that extracted excerpts match gold rationales to certain extent, especially for the aspects Appearance and Smell. For the Palate aspect, manual inspection shows us that for some seeds, the model defines a concept dedicated the notion of numbers (e.g. corresponding to excerpts "12 ounce bottle") that is consistent, yet not part of the gold rationale.
C ADDITIONAL QUALITATIVE EXAMPLES ON DBPEDIA AND AGNEWS
13

---

Under review as a conference paper at ICLR 2020
Class Sports: just imagine what david ortiz could do on a good night" s rest. ortiz spent the night before last with his baby boy, d' angelo, who is barely I month old. he had planned on attending the red sox family day at fenway park yesterday morning, but he had to sleep in after all, ortiz, had a son at home, and he...
Class Sports: foxborough - looking at his ridiculously developed upper body, with huge biceps and hardly ounce of fat, it easy to see why ty law, arguably the best cornerback in football chooses physical play over finesse thhtt s not to imply that he' s &lacking a finesse component, because he can shut down his side of the field much as deion sanders s...
Class Sports: ap american natalie coughlin won olympic gold in the 100-meter backstroke monday night. coughlin, the only woman ever to swim under I minute in the event. . finished first in minute, 0. 37 seconds. kirsty coventry of zimbabwe, who swims at auburn university in alabama, earned the silver in I 00 50. laure manaudou of france took bronze in I 100 88.
Class World: lourdex france- a frail pope john paul ii breathing heavily and gasping at fimes, , celebrated an open-air mass on sunday for several hundred thousand pilgrims, many in wheelchairs, at a shrine to the virgin mary that is associated with miraculous cures. at one point he said help me in polish while struggling through his homity in french.
Class World: najaf= 1 iraq : explosions and gunfire rattled through the city of najaf in armored vehicles and tanks rolled back into the streets here sunday, a day after the collapse of talks- and with them a temporary cease-fire -intended to end the fighting in this holy city... ...
troops
Class World: kabul, , afghanistan = government troops intervened in afghanistan s latest outbreak of deadly fighting between warlords, , flying from the capital to the far west on u. s. and nato airplanes to retake an air base contested in the violence, officials said sunday...
Class Business: london [ reeutrs - the dollar dipped to a four-week low against the euro on monday before rising slightly on profit-taking, but steep oil prices and weak u. 5. data continued to fan worries about the health of the world s largest economy.
Class Business: new york ( renters) - the dollar extended gains against the euro on monday after a report on flows into u. 5. assets showed enough of a rise in foreign investments to offset the current account gap for the month.
Class Business: reuters- apparel retailers are hoping grade amongstyle-connsion teens and young adults this fall, but it couldbe a tough sell, with students and parents keeping tighterhold on their wallets.
to-ooackkkt--chooo fashions will make the
Table 7:
AGNews test examples correctly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.
Business, True Class: Sci/Tech hong kong reuters )- dell inc. &lt dell. o&gt, , the world' s largesi pc maker, said on monday it has left the low-end consumer pc market in china and cut its øverall growth larget for the country this year due to stiff competition in the segment. Predicted: Business, True Class: World supporters and rivals warn of possible fraud government says chavez" s defeat could produce turmoil in world oil market,
Predicted:
Predicted: Sports, True Class: World athens, , greece top american sprinters jason lezak and ian crocker missed the cut in the olympic 100-meter freestyle preliminaries tuesday, a stunning blow for a country that had always done well in the event. pieter van den hoogenband of the netherlands and australian ian thorpe advanced to the evening semifinal a day after dueling teenager michael phelps in the 200 freestyle, , won by thorpe...
Predicted: Sci/Tech, True Class: Business san francisco - in the latest of a series of product delays, intel corp has postponed the launch of a video display chip it had previously planned to introduce by year end. , putting off showdown with texas instruments inc. . in the fast- fast- growing market for high-definition television displays.
Predicted: World, True Class: Sports athens- - the mistakes were so minor. carly patterson s foot scraping the lower of the uneven bars courtney kupets tumbling pass that ended here instead of there. mohini bhardwaj s sllght stumble on the beam.
Predicted: World, True Class: Sports thens, aug. 17 so michael phelps is not going to match the seven gold medals won by mark spitz. . and it is too early to tell if he will match aleksandr dityatin the soviet gymnast who won eight total medals in 1980. but those were not the
Table 8: AGNews test examples uncorrectly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.
14

---

Under review as a conference paper at ICLR 2020
Concept
software services giant moonwalk to home / video display chip downloading music
Concept
upcoming my prerogative video / his ever-growing swimming / launch of video display chip / illegality of downlouding
Concept 2
oil market oil giant sibneft oil prices and ipos public
Concept 3
100-meter freestyle preliminaries tuesday, a stunning blow for a/ the red sox family day at / the winner entire era
Concept
cash settlement of up to #36 50 million / six-year deal worth about $40 million / the dollar dipped four-week low against the euro / five shares, york, light / boston and texas / london reuters newsday #146s
Concept
Concept
olympic 100-meter freestyle sox family olympics should help athletes were already
Concept
gunfire rattled through the election on democratic coordination ed as the / wiretapping internet phones to monitor criminals and terrorists is
Concept
frail pope john paul/ / indian army major shot / unions representing workers gover-
8
ment representatives.
Concept
lourdes, france ap american natalie / new york -/ap it
Table Examples of excerpts that extracted AGNews for all 10 concepts. Colors match the colors used in Table 7 and Table
Concept
lourdes, france david ortiz could do on good night //) monsanto foaf/loaf and bloom filters have
Concept
man who claims gov install shutters outside their windows reuters apparel retailers downloading music the ignorance
Concept
najaf, iraq - explosions and gunfire / latest veterans committee ballot / steep oil prices / billed its ipo
Concept
google inc '/ the pollution. the daylight, the noise video display chip download ing music.. the ignorance
3
Concept
lourdes france / veterans committee ballot/ / intel corp has downloading music.. the ignorance
Concept
sprinters jason lezak and ian crocker missed david ortiz could / the season wireless networking standard up to 10 times the speed oil market. / david ortiz could do good night profiittakiigg. but steep oil prices boston the jury is still out on whether computer
Concept
Concept
pilgrims, many in wheelchairs david ortiz could factory is never / nobel laureate in medicine
Concept
france a frail pope new york (/ world" largest economy / maps figures and endless charts
8
Concept
afp - india s tata iron and steel company / quot it hurt profit-taking, but steep oil / goverment representatives. but hope you find it
Table 10: Examples of excerpts that extracted AGNews for all 10 concepts without No concept Loss. We see the concepts are inconsistent and hard to parse.
15

---

Under review as a conference paper at ICLR 2020
Class Company: transurban manages and develops urban toll road networks in australia and north america it is a top 50 company on the australian securities exchange (axx) ) and has been in business since 1996. in australia transurban has stake in five of sydney" "s nine motorways and in melbourne is the fall owner of citylink which connects three of the city" 's &major freeways. . in the usa transurban has ownership interests in the 495 express lanes on a section of the capital beltway ground washington de.
Class Animal: the red-necked falcon red-headed merlin falco chicquera is bird of prey in the falcon family. this bird is a widespread resident in india and adjacent regions as well as sub- sub-saharan africa. it is sometimes called turunti locally- the red-necked falcon is medium-sized long-winged species with g bright rufous crown and nape. it is on average 30-36 cm in length with wingspan of 85 cm. the sexes are similar except in size males are smaller than females as is usual in falcons.
Class Plant: astroloma an endemic australian genus of around 20 species of flowering plants in the family ericaceae. the majority of the species endemic in western australia but a few species occur in new south wales victoria tasmania and south australia. - species include astroloma baxteri a. cunn. ex de. astroloma cataphractum a a.j. g. wilson ms astroloma ciliatum lindl. ) druce astroloma compuctum r. br. astroloma conostephioides ( (sond. J. muell. ex benth.
Class Album: stars and hank forever was the second and last release in the american composers series by the avant garde band the rexidents. the album was releused in 1986. this particular release featured side of hank williams songs and a medley of john philip sousa marches. this was also the last studio album to feature snakefinger. . kaw-liga samples the rhythm to michael jackson 's billie jean and did well in europe it is as close as the residents ever got to bona fide commercial hit..
Class Written Work: fire ice is the third book in the numa filey series of books co-written by best-selling author clive cussler and paul kemprecos and was published in 2002. the main character of this series is kurt austin. in this novel a russian businessmun with tsarist ambitions masterminds a plot against america which involves triggering a set of earthquakes on the ocean floor creating a number of tsunami to hit the usa coastline.. it is up to kurt and his team and some new allies to stop his plans.
Class Educational Inst.; avon high school is a secondary school for grades 9-12 located in avon phin, its
enrollment neared 1000 as of the 2008-2000 school year with a 2008 graduating class of 215. the school colors purple and gold. the school mascot is an eagle. the avon eagles part of the west shore conference. . they will be moving to the southwestern conference beginning the 2015-201a school year. Class Artist: vicky hamilton [ boonn april 1 1958 j is an american record executive personal manager promoter and club booker writer journalist playwright and screenwriter ) documentary film maker and artist. hamilton is noted for managing the early careers of guns n roses poison and faster pussycal for being a management consultant for mötley criee and stryper 1980s concert promoter on the sunset strip and club booker at bar sinister from 2001 to 2010.1 I...]
Class Athlete: james jim arthur bacon birth registered october-december 1896 in newport district death unknown) was a welsh rugby union and professional rugby league footballer of the 1910s and 20s and coach of the 1920s playing club level rugby union ( ru for cross keys rfc and representative level rugby league ( rl] for great britain and wales and at club level for leeds as wing or centre i. number 2 or or 3 or and coaching club level rugby league rl ) for castleford.
Class Office Holder: jonathan david morris ( october 8 1804 - may 16 1875 was a u.. 8. representative from ohio son of thomas morris and brother of isaac n morris. born in colambia hamilton county ohio morris attended the public schools. . he studied law he was admitted to the bar and commenced practice in batavia ohio he served us clerk of the courts of clermont county- morris was elected as a democrat to the thirtieth congress fill the vacancy caused by the death of thomas I.
Class Mean Of Transp.: german submarine u-32 was a type viia u-boat nazi germany" s kkiegssaarie
during world war ii her keel was laid down on 15 march 1936 by ag weser of bremen as werk 91.3, she was launched on 25 february 1937 and commissioned on 15 april with kapitänleumant ( kptlt. ) werner lott in command. on 15 august 1937 lott was relieved by korvettenkapitän ( krv. . kpt. ) paul büccell and on 12 february 1940 oberleumant zur see oblt. 2. s. hans jenisch took over he was in charge of the boat until her loss.
Class Building: the tootell house also called king "s row hedgerow is g house at 1747 mooresfield road in kingston rhode island that is listed on the national register of historic places, , the two-story wood-shingles
colonial revival house on a 3-acre 12000 m2) ) tract was designed by gunther and beamis associates of house design was by john j. g.
boston for mr. & mrs f. delmont tootell and was built in gunther and elizabeth clark gunther was the landscape architect for the grounds.
Class Village: angamoozhi is a village in pathanamthitta district located in kerala state india angamoozhi is near seethathodu town. geographically angamoozhi is a high-range area it is mainly a plantation township. both state run karte and private operated buses connect angamouzhi to pathanamthitta city. tourist can avail the travelling facility by ksrtc service morning 5 30 from kumili and 11 30 from pathanamhitta between kumili and pathanamthitta via vallakkadavu angamoozhi kakki dam and vadaserikkara and can enjoy the beauty of the forest. I citation needed) ]
Table 11: DBPedia test examples correctly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.
16

---

Under review as a conference paper at ICLR 2020
Concept
top 50 company the regular seasonal movement / the production of port wine / the american composers bollywood film / the third book / a secondary school / an american record executive personal manager promoter / a well-known v8 supercars presenter and commentator a merchant and public official / a type viia u-boat house at / the main fort major poet
Concept
born in philadelphia/ tranquil star ( foaled / instructional documentary produced / bollywood film released in nineteenth instalment secondary school for / born april 1958 / birth registered october-december 1896 october 8 1804 may 16 1875 academy is the volleyball academy of village in pathanamthitta
Concept
( (1800-1870) probably born in / ( 12 april release in is bollywood film released in / an omnibus release from a studio school located in boorn april / gunji 8 july / october 8 1804 may 16 / an iron collier in / of historic places fort also called top 50 company medium-sized
Concept
wingwwnged species with bright rufous crown / endemic australian genus album was released / bollywood film released paul kemprecos and was published inns of court exclusively entitled / debut album explores / lagarto lizard hong kong cross- medsaa creator bird park in / wild blackwater river
Concept
beltway around washington de democratic republic of plants in the party live in / my texas asheville north carolina. ohio its executive he democrat to the south africa. / mooresfield road in kingston rhode island that / north to south / kerala state india
Concept
top 50 company / is a genus of crane fly / is red portuguese wine grape / is the debut studio record / is an abe family original movie / was a regional fashion entertainment and lifestyle publication interactive technologies institute / is a beirut- based lebanese haute couture fashion designer / was a montenegrin serb boxer is the chief technology officer / was a type viia u-boat of historie places the two-story wood-shingled fort ( also / is village
Concept
company on the bird of prey in apple in documentary produced by / film released in / sequel to the / school for grades animator he catcher he cross-media creator who / aircraft corporation to museum was made
Concept 7
the brands club mahindra holidays club mahindra travel club / bird of prey in / an endemic australian genus of around snooker world rankings 1978/1979 the professional world rankings for black and white chinese animation short made in a british rugby league periodical that / a russian youth football academy based in / a turkish cypriot singer and athlete /a welsh rugby union and professional rugby league footballer of planter confederate cavalry general in a type viia u-boat of a volleyball academy / an uninhabited steep rocky island west of street sounds was / medium-sized oong-winged species with bright rufous / rataj places it / ep by linkin park / white chinese animation british schools secondary school for grades argentine painter of the concretist and cubist schools eminent an- glican priest / hospital ships was built/ / national register of historic places uninhabited steep rocky island west village pathanamthitta district
Concept
Concept translation pension fund for adaptation which enables / mopeds produced by / instructional documentary produced by bollywood film released in / book the / river in kwazulu-natal memoir of her boxer in the / presidency of dmitry medvedev / began u-boat of nazi / harbour of helsingør dedicated uninhabited steep rocky island west of
Table 12:
Examples of excerpts that are extracted on DBPedia for all 10 concepts. Colors match the colors used in Table
17

---

Under review as a conference paper at ICLR 2020
D VARYING THE NUMBER OF CONCEPTS ON DBPEDIA
( (the second is zoom to better see the differences between 10, 20 and 50 concepts) show the trade-off between concept accuracy and final accuracy on the DBPedia dataset, for multiple values of C' and different value of A. Each line is a value of C. Each marker is different value of and values are (from left to right markers) {0, 0.01, 0.1, 0.5). Separately. we show examplar excerpts of concepts learned with C = 50 and À = 0.5 (purple dot in Figure 4b) in Table [3]ll 36 concepts are identified as present and we report those) to compare with the ones learned with C' = 10 and À = 0.1 in Table 11 (blue square in Figure 4b With C = 50, the learned concepts qualitatively seem more consistent, even the posteriori concept accuracy 81 .37% for the model with C' 50 which excerpts report in Table 13 which is slightly less than the value (83.90%) for the model with = 10. Hence, learned concepts with C' 50 might be more similar to each others (e.g. less separable). Furthermore, also harder to parse, which is why we consider that the binary intermediate representation should be oowdiimeniional. In future work, we would like conduct large scale experimentation involving human judgments to conclude on the interpretability benefits of different values of C. Note that with too small value of C' (here, C 5) final accuracy is harmed the intermediate representation is too coarse
Figures 4a and 4bd
explanation with more concepts to look and interpret is
EOUCE C-5
C-10
C=20
97.5
90
EDUCE C-50
Outpur AcC. (%)
85
Outgut Acc. (%)
97.0
80
75
96.0
70
EDUCE, C=5
95.5
65
EDUCE, C-20
EDUCT, CH30
95.0
40
60
100
0
20
40
60
90
100
Posterior Concept Acc (%6)
Posteriori Concept
(b) DBPedia test performance, output accuracy vs (a) DBPedia performance, output accuracy posteriori concept accuracy, for multiple values a posteriori concept accuracy, for multiple values of C and multiple values Zoom between 95% of C' and multiple values of X.
and 98% final accuracy.
Figure 4: DBPedia test performance, output accuracy vs posteriori concept accuracy, for multiple values of C and multiple values of X.
18

---

Under review as a conference paper at ICLR 2020
Concept
member of sire records act primate native to / an indictment of policy changes / annexation of mrauk-u kingdom / regents of the university of / litterateur historian and activist/ / baseman first baseman and manager democrat to the thirtieth congress to fill the vacancy caused / lieutenant commander henry crommelin / governor residence and / prime minister of canada arthur meighen who held office committee in ilam district is portuguese wine grape
Concept
Concept
william James young (1800-1870) probably born in april 1983 30 december 1992 ) was a british-bred ramos crepidium ) member brian wilson presents smile / film starring chad everett sarah / robert service then a professor st, george s r. с primary vicky hamilton ( boonn april / james jim arthur bacon ( (birth registered october-december / jonathan david morris october clarence 1. evans ( (
was cannon- cannon-class/ / arthur goodson house
Concept
a tribe of / neighhorhood
village
Concept
correspondents located in most of the district headquarters / forest on the western slopes river range in album from stone temple / forest home of seaside town of southwold and centers/ / whisky distillery. the distillery / favela of capão redondo in bar and moved hospital ships was built house 1747 mooresfield road island west of the dingle peninsula. at village in pathanamthitta district located
Concept 7
shoes and other dance shoes and apparel. racehorse who won at his section macrophylli / players in the man who flees the city and works in / biography of jesse lauriston livermore.. the / painter who lived / artist hamilton noted for managing the / footballer of the 1910s and 20s and coach of singer from finland. elomaa was born in / exhibition at the resident paid for / fan gyhirych in top 50 company / of the five operating divisions of the thomson corporation/ rankings 1978/1979 the professional world rankings / collection of educational dvds tv series music/ independent alternative newspaper leading public university in the field of agriculture / network manager at the national / hong kong cross- moos--media creator / electrical multiple units / artist-run centre located in vancouver be with a vision / supply drinking water / centre of Ørskog municipality
Concept
Concept 10
an international producer / last studio album /a 1971 road movie directed by monte hellman starring songer-songwwiier / a damned poet / fryderyk chopin university of music/ british musician he was bassist / a french passport a canadian record producer music publisher
Concept
top 50 company the / is portuguese wine grape that is used in / is demo tape by / was first broadcast in syndication on was a regional fashion entertainment and / is established under the is an american record executive personal manager promoter and / was an american racecar driver. primarily / was a merchant and public official from / was 4337-ton steamship built by / is house at 1747 mooresfield road in / is located in / is a police station at an institute of the romanian academy cultivar raised the university / guide to british schools/ secondary school / college oxford university he reviewed poetry / an american college/ a degree in chemistry mars-orbiting aeronomy probe the volleyball academy taluka of amravati
Concept 12
Concept 14
university.... the first of the school and it was released on / school during the early 1990s specifically between school for grades 9-12 located in / school later returning to / school... she graduated from dartmouth college in 1998 with / schools he studied law he was admitted to university also known as evolutionsmuseet / high located in skole which covers løken momoen
Concept 15
bird of the arabian nights bird of prey in the falcon family this bird / subfamily within the family cuphorbiaceae / hyena label in 2006 / faunistic studies and biogeography... dolphins. he mammal of africa / fossil collection in scandinavia/ cat about to pounce... bruce
19

---

long-winged species with / is Under review as a conference paper at ICLR 2020
Concept 16
first listed in the considered by the identified by melville listed on the delivered to nearly 8200 homes monday through / built in 1959 and has undergone some modifications from the / 1965 us number 1 hit single for herman powered by miller four-cylinder. it the former first lady of / launched 25 february 1937 and commissioned on / built in 1932-1933. house design was by john / an uninhabited steep rocky island west of the provides public bus transportation
Concept 17
Concept 21
31 march 2014 pfzw / april 1983 30 december 1992 / april 27 1993 / july 6 1975 / 1933 to july 28 1934 / september 29 1962 born april 1 1958 / birth registered october-december 1896 born 5 august 1946 november 18 1985 december 29 1962
Concept 22
business since 1996.. helicominnee commonly called name has been misapplied for / label in 1992. production created by / based in egremont / founded in 2005 / producer based in toronto who has / united states ). company thornaby-on-Be burdick and cook london in sweden established in name could derive from newspaper comic strip reprints but later on original material / first described by / published his first description of in pomographic parodies of / was published in newspaper reported in / romance novels from / published author himself as / newspaper reporter and editor and ( briefly a motorcyclist magazine stated / a copy of / his novel on first written notice about
for
Concept 23
Concept 24
a church established an historical text a building of the trinity church / an american beaux-arts architect 145th state house victorian admiral-class battleship house at / a tourist attraction
Concept 27
single-engined jet the company / light golden color / list of motorcycles scooters and mopeds produced/ hits released only / bollywood film released / whaling ship called trains indian army personnel/ american film and television screenwriter best american racecar driver united states navy commander and type viia u-boat of nazi germany s kriegsmarine submarine museum managed
Concept 28
gecko they claim their cars stick / bird of prey in the falcon family this bird / subfamily within the family japanese phantom thief eagle the avon eagles / brazilian butterfly swimmer monoplane with wire braced wings / subsidiary of sparrow health bird species including ducks great crested grebes swans and geese largest icelandic commercial banks / the only mare / is the pacific ocean water by the spoonful / agua fria high school / karnik ( devanagari / represents the creuse s 2nd constituency mars-orbiting aeronomy probe is a large boulder is an uninhabited steep rocky island west of the dingle is situated on the lap of vindhya ranges
Concept 29
Concept 30
a top 50 company is portuguese wine grape the original soundtrack on the sony bmg label / an american disney comedy franchise a free independent alternative newspaper the statutory and regulatory organization personal manager promoter / an american stock car racing driver / a grain buyer / high-speed electrical multiple units / an artist-run centre / supply drinking water was only newspaper comic / is selection / was released in 1986 this particular release / are 1993 pornographic parodies/ is the third book was the original dj / is the eleventh / is survey launch / is the world tenth longest river
american record executive
Concept 31
Concept 34
british pottery manufacturing firm / a collection of american singer laura / italian film it stars actor gabriele/ / damned poet and somewhat heartbreaking images american comic strip creator poker player pioneer poker theorist / female bodybuilding champion and pop singer scientist astronomer and botanist
Concept 35
ponderosa steakhouse and / is medium
endemic australian genus of around / is a venomous tree-dwelling snake : / is a brazilian butterfly swimmer. rooted in is family of / is a saltwater pond in a naive bayes/ / a bird of prey / an endemic australian genus/ / venomous tree-dwelling snake/ / a young adult / an american sociobiologist brazilian butterfly swimmer / large ruminant mammal the largest and most important cave
Concept 36
20

---

Under review as a conference paper at ICLR 2020
Concept 37
king street sounds / an historical text a small rural public school district / was american beaux-arts architect george heard stone / s eighty-sixth house district / haven class of hospital ships / is a house also called purandhar fort / is a village single-engined jet the company was founded 1998 by mopeds produced by honda/ chinese habitats that simulate generation ships / an argentine model vedette / sprint car driver he drove in the 1936 indianapolis confederate cavalry u-boat of nazi germany kriegsmarine during world maritime museum kulturverftet and helsingør harbour jeep underwater the deepest
Concept 38
Concept 39
species of tropical moss / endemic australian genus of around 20 species of flowering plants / Iloyd mackenzie says "by blending the richness swee pea brookland plantation is plantation / plants and animals
Concept 40
was the albuquerque new mexico- based manufacturer / is member is a collection of american singer / are 1993 pornographic parodies the third book is an american record executive personal manager promoter / was japanese master / was u s representative / was stalwart significant a representative is the most prominent feature
Concept 42
the second largest pension fund is teen pop star aaron carter" 8 first concert / is a bollywood film / a 1923 roman clef / is secondary school / is an american actor film is an american football guard / is the oklahoma senator / as barracks ship / is house also called purandhar fort is separate campus
Concept 43
film magic was founded on / rock monitor it is found in meadows in the video release overall released in film released in 2010 which stars rishi / moon series written by / river in kwazulu-natai south / rock was the film blazing saddles and for starring in the / lake county and heights in the / family situated on the north slope of the catchment within the murray darraay basin is located in the is an optical disc system developed by / is a selection made by / was released in 1986 is video series by is an omnibus release from / the electronic music polo-project of / was on the was a planned plug-in in hybrid electric vehicle from / is joint initiative by
Concept 46
Concept 47
multi-jurisdictionall lottery games american thoroughbred race horse / rifle sport ruthless / champion thoroughbred filly ruffian who went undefeated british rugby league / creek tennis courts track / french dancer choreographer / welsh rugby union and professional rugby league footballer / retired professional female bodybuilding champion sport utility vehicle / galatasaray sports club / hole par-3 golf
Concept 48
mikuláš ružomberok vrútky
Concept 49
company that published comic books beginning in 1933 an album by american comedian comedy writer and radio / a bollywood film released in 2010 which stars / the third book in the american comic strip creator who signed his work professional poker player pioneer poker theorist author of poker / a canadian lawyer professor politician and writer / british motorcycle made between 1959 and 1966 by phelon contemporary art gallery in melbourne australia / government- commissioned geographic study which forms the basis for the
Examples of excerpts that are extracted on DBPedia when using C' 50 concepts. Only 36 concepts are identified as present and we report those.
Table 13:
21

---

Under review as a conference paper at ICLR 2020
Concept 0
Concept
Concept 2
Concept 3
Concept 4
Concept 5
Concept
Concept 7
Concept
Concept
Figure 5: Samples of patterns extracted for each concept in the birds-cats-dogs dataset.
E APPLYING EDUCE PRINCIPLES FOR IMAGE CLASSIFICATION
Having assessed the relevance of our model on text data, we now turn to image data and explore EDUCE is also able to extract meaningful concepts. In that case, the excerpts extracted in textual data are replaced by regions (or patches) in an image, and the recurrent neural architecture processing text is replaced by classical convolutional neural network.
Principles: Each image x is first encoded through a convolutional neural network that output representation h(x) = Wx HxD where the number of output filters. Each vector hs,j (x) RDD corresponds to a particular region of the input image (i.e the receptive field of the convolutional neural network). Then each ft.j. is used to compute Py (s|2, c) through linear model and softmax activation as in Section 2.1 (step 1), and to compute Po (Ze = l[so(z); ((()), c) (Section 2.1, step 2) here again with a linear model and Bernoulli distribution. The semantics homogeneity is ensured by using an image classification network over the region captured by the receptive field the previous convolutional neural networks that ensure that all regions captured from particular concept "look the same" and look differently from the regions captured for other concepts.
Experiments: To ensure the validity of our adaptation of EDUCE images, conduct very simple experiments on a dataset composed of 224 x 224 RGB images split in 3 categories: dogs, cats, and birds in equal proportion. We train on 3. 000 images and test on 3, 000 images. We build our model on top of a pretrained VGG-11 11 model [Simonyan & Zisserman (014) which is used for computing the intermediate representations hax)). Said otherwise, the output of the VGG-11 model is used for both detecting relevant regions and thus building the binary representation 2, but also to ensure the homogeneity extracted regions during training. Figure 5 shows extracted patterns and associated categories. Final classification performance is 91.6% with 10 concepts. In Figure 5 we plot extracted patterns for the 10 concepts and report in Figure 6a the weights of the final classifier. From these two figures, differentiates a dog from a cal or a bird, and support the classifier's prediction the dog category. Moreover, by looking which regions extracted by model (Figure 6b ), we can that EDUCE can focus its attention to discriminative regions of the images.
interpret the model's behavior: concept 8 and concept 9 show what
We construct this dataset by combining random images from the Caltech Bird 200-2011 dataset Wah al. with images of the cats-and-dogs Kaggle dataset (Kaggle 2013
2011
22

---

Under review as a conference paper at ICLR 2020
Cat
Dog
(a) Final Classifier weights, each column a con- cept (0 to 9, from left to right) each row a final category. The lighter, the higher the weight is. Concepts and associated category 'Bird', Concept and 'Dog'.
(b) Examples of classified images (column and
and concepts 8 to
4) and corresponding extracted patterns (column
and 3) for concept detected as present.
Figure 6: Interpretation of EDUCE on images.
23


Reviews:
[{"review_id": "rylzzXLCYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a self-explainable deep net architecture that could be used for text categorization. The main idea is to force the network to extract \"excerpts\", from the input text, each corresponds to a concept, which are also learned for interpretation. The classification is finally made based off of the learned concept, which is a binary vector. All three steps are learned in an end-to-end manner. The learning of concepts is regularized to make sure the concepts are consistent and non-overlapping. The idea sounds interesting and the experimental results support the usefulness of the proposed method on a variety of datasets. My sole concern is about the sensitivity analysis of the explanation, i.e. how robust is the explanation with respect to the perturbations that do not change the classifier prediction. It has been discussed in the literature that many explanation methods suffer from this sensitivity issue. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sklhuih3Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new concept-based interpretability method that lies in the family of self-interpretable models (i.e. it's not a post-hoc method). Self-interpretability is achieved by a two-stage model: First, a concept-extractor finds the related pieces of consecutive words (excerpts) in a given text that are related to a concept among a set of given concepts (if any), then the model makes its predictions solely based on the presence or absence of concepts (binary).  The most useful part of the algorithm is that there is no need for concept annotations. The work then experimentally shows that their method, although does not outperform a non-self-interpretabile baseline but has better performance and interpretation compared to rival methods.\nThe paper is quite well-written. The introduced method is well-justified and the use of concept-based self-interpretable models is very useful to the field. It is also interesting to see that the performance is comparable to non-self-nterpretable baselines which would make a case for the use of self-interpretable models. I have two main concerns with this work. First, there is the novelty concern. The idea of unsupervised extraction of concepts for interpretability was introduced before (https://arxiv.org/pdf/1902.03129.pdf) and is not discussed by the authors (although the utilized terminology is very similar).  Authors should make a much more comprehensive discussion of what already exists in the concept-based interpretability literature and make the contribution of this work more clear (unsupervised concept extraction for a self-interpretable model instead of post-hoc interpretations). Secondly, although some of the objective experimental results (BeerAdvocate results, to some degree) suggest that the method is indeed capable of discovering meaningful, consistent, and separable concepts on its own, there is not enough discussion of why it actually should be the case. It's very easy to assume the introduced training procedure to extract separable but meaningless concepts(i.e. the excerpts of a concept are separable from that of other concepts and are consistent with each other in the eyes of the network while they are not consistent with a concept in the eye of human rationale; both loss terms will be minimized but there is no human-interpretability. One big issue with the experiments section is the A Posteriori Concept Acc metric as it only measures the separability and consistency of discovered concepts; first of all, it can be high while the extracted concepts are meaningless to humans, secondly, using it as a measure of comparison to rival methods is not quite fair as the introduced method is directly optimizing for this metric. The subjective results are interesting but not convincing. Adding a section for human subject experiments (following the previous work in concept-based interpretability) would make the results section much crisper. It also seems unlikely that all of the discovered concepts follow the desiderata and the fact that this is not mentioned makes it even more difficult to assess the amount of cherry-picking in the mentioned subjective results; there should be a discussion of cases of failure and why it happens. My score will change accordingly as the authors address the raised issues.\n\nA few questions and suggestions:\n\n* It would be more interesting to show subjective results of the model for cases of mistake; does the interpretability make sense when the model's prediction is wrong?\n\n* It seems necessary to discuss how much the results are sensitive to selecting C? If the performance is robust, it would be interesting to see what happens for a large C? Does it discover more fine-grained concepts or most of the additional concepts would be null?\n\n* For the model architecture, why did the authors choose to use the straight-through estimator and not other methods (e.g. concrete layers, ...)\n\n* Some of the figures should be larger (much larger)\n\nThanks again for a well-organized and easy-to-follow paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkeHR3J0FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a classification model focused on interpretability. The model, Explaining model Decision through Unsupervised Concepts Extraction (EDUCE), is applied to a text classification task, while the authors argue in the appendix that this is also applicable to a wider problem, such as image classification. \n\nThe model is composed of three parts: the first part is detecting salient spans of text relevant to the text classification problem, the second part assigning each salient span a concept label, and the third part which does the classification task based on the binary concept feature label. The models’ loss is composed of two parts: (A) minimizing the cross-entropy of text classification loss and (B) minimizing the cross-entropy of concept classification loss. For (A), as the first and second part of the model introduce discrete choices, they use a RL with Monte-Carlo approximation of gradients. \n\nThe system is evaluated under two measure: 1) classification accuracy and 2) concept accuracy. They define the concept accuracy as follows: after training, they train a classifier that takes output (in the form of <salient span, their concept label>) of the model from the test portion of the data. They split this output into train and test, and report the test accuracy. This aims to show how consistent is the labeling of the salient spans for different methods: if the concept label set correctly merged together semantically similar spans, this “concept accuracy” would be higher. This is a new metric they are proposing. While it is interesting, I would like to see *some* studies on how this correlates with human’s judgements on how interpretable the model is. The paper is introducing a new measure *and* new model, and it’s hard to be persuaded the model is doing well based on this new measure, when there is little ground to know what this measure really measures. \n\nOverall, I’m not impressed with the models’ performances. The aspect rationale annotated beer sentiment dataset, presented by Lei et al (2016), has provided one of few opportunities to evaluate interpretability / rationale model quantitatively. The paper evaluates on this measure, which is included in the appendix, and the results are pretty disappointing compared to the existing models such as Lei et al’s initial baseline or Bastings et al. While the paper argues this method isn’t necessarily designed for this task unlike the other methods, I’m not sure this is necessarily the case. Bastings et al could be applied to other tasks that model is evaluated on, such as DBPedia and AGNews classification. The difference comes on how easy it is to interpret the methods, as these other rationale-based text processing methods would make use of captured words, while EDUCE would make use of detected “concept” clusters. Currently, the only real baselines are the ablations of its own model. \n\nTable 3 is quite interesting, different “concepts” capture different aspects fairly well. \n\nNot having a concept loss actually helps the classification accuracy. Would the concepts learned without concept loss qualitatively very different? This goes back to my original point that their new measure of \"concept accuracy\" is vague. \n\nOther comments and Q: \n- Figure (3), the visualization is a bit confusing cause it is unclear whether it is each span is a set of spans or a single span. Also, I would recommend making figures colorblind friendly, if possible. \nQ: what kind of classifier was used for the evaluation metric “concept accuracy” classifier? I don’t think it’s mentioned. \nQ: why are you sampling a test set for DBPedia experiments? Is it for efficiency reason?\nQ: how sensitive is model’s performance to the hyper parameters, especially the number of concepts?\nQ: the current baseline classifier is a simple BiLSTM one, which definitely perform a lot worse than recent pre-trained LMs such as BERT. Would it be easy to use this method on top of richer representation such as pertained LM outputs?\nQ: how would this connects to saliency map literature in computer vision? I guess these would be mostly “a posteriori” explanations? Discussion would be helpful. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method for building interpretable classifiers, along with a measure of "concept accuracy" to evaluate interpretability, and primarily applies this method to text models, but includes a proof of concept on images in the appendix.

The main contributions are sensible enough, but the main problems the reviewers had were:
A) The performance of the proposed method
B) The lack of human evaluation of interpretability, and 
C) Lack of background and connections to other work.

The authors improved the paper considerably during the rebuttal period, and might have addressed point C) satisfactorily, but only after several back and forths, and at this point it's too late to re-evaluate the paper.  I expect that a more polished version of this paper would be acceptable in a future conference.

I mostly ignored R1's review as they didn't seem to put much thought into their review and didn't respond to requests for clarifications.
Decision: Reject