Title: EDUCE: EXPLAINING MODEL DECISIONS THROUGH UNSUPERVISED CONCEPTS EXTRACTION

Abstract: Providing explanations along with predictions is crucial in some text processing tasks. Therefore, we propose a new self-interpretable model that performs output prediction and simultaneously provides an explanation in terms of the presence of particular concepts in the input. To do so, our model's prediction relies solely on a low-dimensional binary representation of the input, where each feature denotes the presence or absence of concepts. The presence of a concept is decided from an excerpt i.e. a small sequence of consecutive words in the text. Relevant concepts for the prediction task at hand are automatically defined by our model, avoiding the need for concept-level annotations. To ease interpretability, we enforce that for each concept, the corresponding excerpts share similar semantics and are differentiable from each others. We experimentally demonstrate the relevance of our approach on text classification and multi-sentiment analysis tasks.

1. INTRODUCTION:

While deep learning models are powerful tools to perform a large variety of tasks, their predictive process often remains obscure. Understanding their behavior becomes crucial. This is particularly true with text data, where predicting without justifications has limited applicability. There has been a recent focus on trying to make deep models more interpretable, see for example (Ribeiro et al., 2016; Bach et al., 2015; Shrikumar et al., 2017; Simonyan et al., 2014; Sundararajan et al., 2017) . Specifically, methods have been recently proposed to provide such explanations simultaneously with the prediction. For example, (Lei et al., 2016; Yu et al., 2019; Bastings et al., 2019) select subsets of words in the input text that can account for the model's prediction (called rationales). In this work, we propose a model that provides an explanation based on the absence or presence "concepts" that are automatically discovered in the texts.

Suppose we ask a user to tell what is the category (or class) of the top text of Figure 1 (text x ). She detects that the words the government said relate to a specific concept that is present in x, a concept she also detect in the words made official what in text x . Let us call this concept "politics" in the remainder. She also notes that the words retails sales bounced back refer to a concept different from the previous one. Let us call this other concept "economy". As the text is concerned with politics and economy she infers that its category is Business. Said otherwise, she detects excerpts that relate to particular concepts and decides on the text category based on the detected concepts.

Similarly, our paradigm assumes that an explanation of a model's prediction is understandable if it relies on a few concepts, where each concept relates to parts of text (referred to as excerpts) that are semantically consistent across multiple texts. Our methodology is as follows. First, our model encodes the input text into a binary low-dimensional vector, where each value (0 or 1) is computed from an excerpt and denotes the presence or absence of a specific concept in the input. Then, the prediction is made from this binary vector alone, allowing an easy interpretation of the decision in term of presence/absence of concepts.

While we set a maximal value of the number of concepts (which is the dimensionality of the binary representation), the concepts are unsupervised and not defined a priori. The model automatically determines them in a way that eases interpretation: each concept is encouraged to be semantically consistent and not to overlap with other concepts. Therefore, extracted excerpts for a concept must be discriminative of that concept only and share similar semantics. This is enforced through a concept consistency and separability constraint added as an auxiliary loss in the learning optimization problem. (Zhang & LeCun, 2015) examples. The different steps are explained in the text.

As a result, each discovered concept can be understood from the corresponding excerpts extracted that activate its appearance: in our previous example, the meaning of the first concept the user identifies is inferred from the excerpts she detected for that concept in x and x , i.e. the government said, made official what. Looking at these excerpts, we identify that concept as politics.

Our idea relates to Latent Dirichlet Allocation (LDA) (Blei et al., 2003) , where text documents are described by a set of topics that are semantically consistent. However, LDA builds a probabilistic model of text generation, whereas our goal is to discover and define latent concepts that are relevant for a prediction task at hand. In comparison to rationale-based text processing models (Lei et al., 2016; Yu et al., 2019; Bastings et al., 2019) , we rely on a different paradigm: our model's prediction is based on the absence or presence of discovered concepts, and makes no direct use of the words captured as excerpts. We do so to ease interpretation of the prediction. This makes the interpretation of a different nature than these methods, and simpler to understand for the user.

Our contribution with this work is a new self-interpretable model that predicts solely from the presence/absence of concepts in the input. Concepts are learned unsupervisingly, and described by a semantically consistent set of excerpts. We experiment on three text categorization tasks and a multiaspect sentiment analysis task, compare our model's performance with state-of-the-art prediction models, and demonstrate its interpretability. Note that an instance of our model for image processing is described in the supplementary Section E with illustrative experiments.

2. THE EDUCE MODEL:

We present our model, called EDUCE for Explaining model Decisions through Unsupervised Concepts Extraction, using a multi-class classification task, but our method can be used to perform regression or multi-label classification. We consider a training dataset D of inputs (x 1 , ...., x N ),

x n ∈ X and corresponding labels (y 1 , ..., y N ), y n ∈ Y.

Contrarily to back-box models that use complex computations over low-level input features to predict the output class y, our objective is to enforce the model to map an x input to an easy-to-interpret representation z(x) ∈ {0, 1} C , on which the output prediction relies. EDUCE's inference process consists in the following steps.

Step 1: For each concept c (i.e. each dimension of z(x)), compute p γ (s|x, c): the probability of each excerpt s in x to be selected. Sample a unique excerpt s c (x) ∼ p γ (s|x, c) per concept (per dimension of z(x)). Note that the same excerpt can be selected for multiple concepts.

Step 2: Decide on each value z c (x) of the representation z(x) by sampling from

p α (z c |s c (x), c). Given s c (x), z c (s c (x)) = 1 means concept c is detected as present and z c (s c (x)) = 0 means it is absent. Step 3: Predict the output class y from z(x) = (z 1 (s 1 (x)), ..., z C (s C (x))).

We do not have concept-level annotations. To ensure semantic consistency and prevent overlap of the concepts, we jointly train a concept classifier to recognize, for every excerpt s c (x) such that z c (s c (x)) = 1, the concept (i.e. the dimension) it was extracted for: the label for each 1 illustrates these steps, which we detail below.

s c (x) is simply c ∈ [1, C]. Figure

2.1. GENERATING PREDICTIONS AND EXPLANATIONS:

Step 1: Extract a unique excerpt s c (x) per concept c. Given an input sentence x : w 1 , ..., w M , where each w i is a word, a excerpt s consists of a span of consecutive words of flexible size between 3 and 10 words: s = w k , ..., w k+l , 3 ≤ l ≤ 10. For each concept, we extract a unique excerpt, defined by its first and last word. Our extraction process is very similar to the one proposed in Question-Answering models (e.g Devlin et al. (2019) ). An excerpt is sampled by first sampling its start word and then its end word conditioned on the start word.

We denote the probability over each excerpt s to represent concept c in x as p γ (s|x, c) and we write as the product of (i) p start (w k |x, c), the probability for s's first word to be the start word of the excerpt to extract and (ii) p stop (w k+l |x, c, w k ) the probability of w k+l to be the stop word given the start word w k : p γ (s|x, c) = p γ (w k , ..., w k+l |x, c) = p stop (w k+l |x, c, w k )p start (w k |x, c). We parametrize p start (w k |x, c) and p stop (w k+l |x, c, w k ) using recurrent neural networks. First we feed the entire input sentence x : w 1 , ..., w M through a bidirectional LSTM, and we represent each word w i in the sentence as the concatenation of the forward pass and backward pass hidden states for that word:

h k = [ -→ h k , ← - h k ], ∀k ∈ [1, M ].

Second, each h k is fed to a linear layer with parameters γ start which outputs a score for w k , followed by the softmax activation function over all possible words to have the probability distribution over each word to be the start word:

EQUATION

Using this distribution, we can sample a specific start word w start ∼ p start (w k |x, c). Third, we feed again the vectors h k to another linear layer with parameters γ stop that gives a score for each word to be the stop word. We mask out these scores before taking the softmax, such that the probabilities for w start+l with l < 3 and l > 10 to be the stop word are 0.

p stop (w start+l |x, c, w start ) =    exp(γ stop • h start+l ) 10 l=3 exp(γ stop • h start+l ) , l ≥ 3 or l ≤ 10 0, l < 3 or l > 10.

We then sample a specific stop word w stop ∼ p stop (w start+l |x, c, w start )1 . We now have the start word w start and stop word w stop . The corresponding excerpt is w start , ..., w stop and is represented as a fixed size vector s c (x) by computing the average of (pre-trained, fixed) words embeddings vectors of w start , ..., w stop . Hence sampled excerpts s c (x) belong to R d , where d is the dimension of embedding vectors.

This process is illustrated in Figure 1 : For text x, the excerpt Retails sales bounced back is selected for concept c 1 , the government said for concept c 2 and a midsummer slump for concept c 3 .

Step 2: For each c, from the excerpt s c (x), decide on the value z c (x) denoting the presence/absence of concept c. Each extracted excerpt (one per concept) is processed to decide on the absence or presence of each concept in x. Specifically, for each concept c we take the dot product of a weight vector α c ∈ R d with s c (x), followed by a sigmoid activation function, in order to obtain the Bernoulli probability

p α (z c = 1|s c (x), c) = σ(α c • s c (x))

(2) This is the probability that s c (x) extracted from x for concept c triggers the presence of concept c. The binary vector z(x) is obtained by independently sampling each dimension: ∀c z c ∼ p α (z c |s c (x), c).

This step can be seen as C independent binary classifiers, each of them deciding on the presence of a particular concept. We illustrate this in Step 2 in Figure 1 : For text x, the excerpt Retails sales bounced back activates the presence of concept c 1 , the government said activates the presence of c 2 but a midsummer slump does not activate the presence of c 3 .

Step 3: Predict y from z(x). As shown in Step 3 of Figure 1 , given an input x, the prediction of the output y is made solely from the intermediate binary representation z(x). We use a linear classifier without bias, parametrized by a weight matrix δ ∈ R |Y|×C followed by a softmax activation function, returning p δ (y|z(x))∀y ∈ Y.

Output classification training objective. The parameters {γ, α, δ} are learned in a end-to-end manner by minimizing cross-entropy, which writes for each x as:

EQUATION

We give details about the optimization in Section 2.3.

2.2. UNSUPERVISED DISCOVERY OF LATENT CONCEPTS:

To ease interpretation of the concepts' meaning, we want each dimension z to correspond to semantically consistent concepts. In other words, for a given concept c (i.e. a given dimension z c ), for all inputs where that concept is present (i.e. ∀x s.t. z c (s c (x)) = 1), the corresponding excerpts s c (x) should share common semantics. While the C binary classifiers deciding on the presence of each concept c (Step 2) enforce consistency within the set of extracted excerpts s c , there is no guarantee that the concepts are not overlapping (i.e. are separable). We want concepts to be separable so that each concept captures a particular notion. Excerpts extracted for concept c must be distinguishable from the excerpts extracted for another concept c . If we take the example of Figure 1 

(x)), ∀c ∈ [1, C].

The concept classifier is trained by minimizing cross-entropy, where the label of each excerpt s c (x) the index of the concept for which it was extracted (i.e. c):

EQUATION

= E s(x)∼pγ [E z(x)∼pα [ c -z c (s c (x)) log p θ (c|s c (x))]],

where s(x) refers to the set of excerpts extracted for each concept: s(x) = {s c (x), c = 1, ..., C}.

The loss considers concepts that are present in x (i.e. z c (s c (x)) = 1). The role of the concept classifier is to further enforce concepts consistency and to prevent overlap of concepts, and is jointly train with the rest of the model.

Note that adding a sparsity constraint on the number of concepts present in the inputs enforces semantic consistency, as sparse coding has been shown to induce useful, interpretable representations (Bengio et al., 2013a; Mairal et al., 2010) . We experimentally demonstrate that a sparsity constraint is not sufficient, and can harm output prediction performance. On the opposite, with our concept classifier, sparsity is encouraged as L concept depends on the concepts that are present, but if concepts are consistent and separable L concept can be low without harming task performance.

2.3. EDUCE OBJECTIVE FUNCTION AND OPTIMIZATION:

We jointly learn the concept classifier and output classifier and our objective function is the sum Equations 3 and 4:

EQUATION

where λ controls the strength of the concept consistency constraint wrt the output prediction. The loss L(x) is differentiable, therefore the gradients with respect to the output classifier's weights (δ) and concept classifier's weights (θ) can be computed and back-propagated. However, the gradients with respect to the parameters of the excerpts extraction (γ) and the Bernoulli distribution over presence of concepts (α) pose an issue due to the sampling of these discrete random variables in Steps 1 and 2. As the explicit computation of the expectation involves expensive summations over all possible values of s(x) and z(x), we resort to Monte-Carlo approximations of the gradient (Sutton & Barto, 1998) , with the loss L(x) used as reinforcement signal. The gradient derivation is provided in Supplementary Material. Our code will be released upon acceptance.

Note that the concept classifier encourages concept consistency and concept separability. The discovered concepts are not guaranteed to align with human defined concepts. This is a general challenge in learning explainable/interpretable models in an unsupervised setting. We believe it is a reasonable assumption to employ a linear concept classifier over pre-trained word embeddings such that it linear separates the word embedding space.

3. RELATED WORK:

Our work relates to topic models, especially to Latent Dirichlet Allocation (LDA) (Blei et al., 2003) where each document is modeled as a mixture of topics and each word relates to one or more topics.

While LDA has been recently combined with recurrent neural networks (Zaheer et al., 2017) , the methodology remains different. LDA learns the parameters of a probabilistic graphical model of text generation while our goal is to build an interpretable prediction model that relies on binary presence/absence of concepts.

Some existing works interpret an already trained model, typically using perturbation and gradientbased approach, (Ribeiro et al., 2016; Bach et al., 2015; Shrikumar et al., 2017; Simonyan et al., 2014; Sundararajan et al., 2017) Alvarez-Melis & Jaakkola (2017) design a model that detects inputoutput pairs that are causally related. Contrarily to these methods, our work falls in the domain of self-interpretable models, which produce an explanation simultaneously with their prediction. Bau et al. (2017) to propose a method that generates visual explanations of a classifier, in the form of semantically interpretable components with a corresponding label and heatmap. However, in these models, the final classifier is already trained and fixed while EDUCE is learned end-to-end. More importantly, these works need concepts to be predefined from human annotations, while EDUCE is unsupervised. Ghorbani et al. (2019) propose a method based on unsupervised concept discovery to explain a model a posteriori. We differ from their work as EDUCE is trained end-to-end to classify solely based on the discovered concepts, which presence is discriminant, resulting in a fully interpretable model. Finally, we focus on text data while the aforementioned works experiment on image data.

4. TEXT CLASSIFICATION EXPERIMENTS:

We experiment on three text classification datasets. The DBpedia ontology classification dataset (Zhang et al., 2015) was constructed by picking 14 non-overlapping categories from DBpedia 2014 (Lehmann et al., 2015) . There are 14 classes. We use 56, 000 examples of the train dataset for training, and 56, 000 for validation. For testing, we use 7, 000 examples of the test dataset (using stratified sampling). Also in Zhang et al. (2015) , the AGNews topic classification dataset was constructed from the AG dataset's 4 largest categories. There are 4 classes. We separate the training set into 84, 000 training samples and 24, 000 validation samples and report results on the full test dataset. We also experiment on the Stanford Sentiment Treebank (SST) (Socher et al., 2013) that includes 5 sentiment classes. For DBPedia and AGNews we use fixed word vectors trained on Common Crawl (Grave et al., 2018) . For SST, we use fixed Glove word vectors (Pennington et al., 2014) .

Comparative models. Since we are concerned with building self-interpretable models, we do not compare with the methods presented in Section 3 that explain a model a posteriori. Rather, for comparison, we report the performance of (i) using a Bidirectional LSTM on the full text and predict directly from its hidden state (Baseline) (ii) the No Concept Loss model that uses a binary discrete intermediate representation but no concept classifier, it corresponds to λ = 0 and its training objective is simply L output and (iii) adding a L 1 -norm sparsity constraint to the No Concept Loss model (referred to as No Concept Loss + L 1 ), i.e. Equation 5 changes to: Hyperparameters selection. We tried different values of hyperparameters (details in supplementary Section B.1) and for each set of hyperparameters, we run 5 different random seeds. We choose the best hyperparameters set using its average validation performance over the 5 random seeds.

EQUATION

Metrics. We report the test output accuracy (Output Acc.) over the task. Evaluating the relevance of discovered concepts is a challenge in the unsupervised setting. To measure how well the discovered concepts align with human defined concepts one would need human experiments, which we consider as future work. However, we evaluate how consistent and separable the discovered concepts are, as a proxy to measure how easy it would be for a user to interpret them. For EDUCE, we can compute the accuracy of the concept classifier on the test data. However, this should be low for models No Concept Loss and No Concept Loss + L 1 that do not train with a concept loss. Therefore, to evaluate interpretability, we report an a posteriori concept accuracy: after training, for each model (including EDUCE), we gather the excerpts extracted (z c = 1) in the test data. We separate these excerpts into two sets (training and testing, note that these are both generated from the test data). We train a new, separate linear concept classifier a posteriori on the extracted excerpts to evaluate and we report its performance as a posteriori concept accuracy (A Posteriori Concept Acc.).

Quantitative results on DBPedia and AGNews. Table 1 reports test performance on the DBPedia and AGNews datasets for C = 10 concepts. We report results with values λ = 0.1 and λ L1 = 0.1 that give best trade-off performance between output and concept accuracies. Complete results are available in supplementary Table 4 . For all metrics we report the mean and standard error of the mean (SEM) over the training random seeds. Table 1 shows that the Baseline model outperforms the other models that all employ a binary discrete encoding of the input. This is expected, and shows the trade-off between interpretability and output accuracy. In terms of output accuracy, EDUCE is comparable with its counterpart that does not encourage concept consistency (No Concept Loss), loosing only 0.4% on DBPedia and 0.7% on AGNews, and as expected outperforms it in terms of concept accuracy. Adding an L 1 constraint to the No Concept Loss model increases the consistency of the concepts (as per A posteriori Concept Accuracy values) yet is largely outperformed by EDUCE on that metric, and output accuracy is also lower than EDUCE's.

Interpreting EDUCE on AGNews. We turn to show how EDUCE's output prediction is easily interpretable. The following results were generated with λ = 0.1 and C = 10 concepts. Figure 2a shows examples of the AGNews test set that were correctly classified by EDUCE. The underlined words correspond to the excerpts extracted for different concepts. Separately, in Figure 2b we show, for each concept detected in the examples of Figure 2a , examplar excerpts extracted from others test documents (excerpts are separated by "/"). We interpret the concepts' meaning as follows: concept 0 maps to informatics notions, concept 2 to corporations/petrol, concept 4 to the notions of investments and finance, concept 6 to sports events and concept 8 to governmental/state affaires. Concept 1 is less clearly defined. Note that in Figure 2b Figure 3a shows the empirical frequency of presence of each concept, per output class in the SST dataset, using λ = 0.1 and C = 10 concepts. We see that concepts 3, 5 and 7 are often triggered when a sample is positive or very positive, and on the opposite concept 8 is present in negative/very negative texts. The values of weight matrix δ of the output classifier in Figure 3b proves that the presence of these concepts is responsible for the output classifier's prediction. We can further analyze the concepts: Each word in SST sentences is annotated with a label for the sentiment it expresses, referred to as word label. We do not use them during training or validation, but at test-time this allows us to analyze the repartition of words labels in the set of excerpts extracted for each concept. Figure 3c shows that indeed the excerpts selected for concepts 3, 5 and 7 are mostly composed of positive or very positive words, and the excerpts extracted when concept 8 is triggered are mostly composed of negative or very negative words2 . We train EDUCE on the prediction of the 5-values vector at once (4 aspects and the overall score, there are therefore 5 scalar ratings to predict) using the 260, 000 reviews. We compare EDUCE's performance with the Baseline that accesses the full text. The test MSE is 0.0089 ± 0.0 (mean and SEM accross training seeds) for the Baseline model and 0.0119 ± 0.0 for EDUCE using λ = 0.01 and C = 10 concepts, with a corresponding concept accuracy of 92.8 ± 0.7. Of specific interest to us is the per-concept precision of gold rationales and the total percentage of selected words. Indeed, Table 3 shows that when predicting the 4 aspects and the overall score, some concepts are capturing a specific aspect: for example, the precision of Concept 7 on the Appearance aspect is 97.09%. This is confirmed by Table 2 that shows that excerpts extracted for Concept 7 are mostly related to the color and head (the frothy foam on top of beer) of the beer4 . Supplementary Table 5 reports examplar excerpts for all concepts.

As in Lei et al. (2016) ; Bastings et al. (2019) we also train EDUCE on the prediction of each of the first three aspects separately. Supplementary Table 6 reports the precision of gold rationales for each aspect. While the results are less convincing than the models specifically designed for rationales extraction, extracted excerpts match gold rationales to a certain extent. Table 3 : Per-concept precision of gold rationales (in % for each aspect), trained to predict all 4 aspects and the overall score. In bold we emphasize the concepts which precision is above 50% for an aspect's gold rationales.

B.1 HYPERPARAMETERS CONSIDERED:

In all experiment, we increase the value of λ by 10% every epoch. In order to choose the best set of hyperparameters, we take the average value on the 5 random seeds.

DBPedia and AGNews:

We try the following ranges of hyperparameters λ ∈ {0.0, 0.001, 0.01, 0.1}, λ L1 = {0.0, 0.001, 0.01, 0.1, 1}, r ∈ {0.01, 0.1}, learning rate lr = {0.0005, 0.001}. The hidden state of the LSTM is of size 200. We use Adam optimizer (Kingma & Ba, 2014) , batches of size 64.

For the Baseline model, we try adding weight decay in {0.0, 1e -6} and dropout in {0.0, 0.1, 0.2}.

SST We try the following ranges of hyperparameters λ ∈ {0.0, 0.001, 0.01, 0.1}, λ L1 = {0.0, 0.001, 0.01, 0.1, 1}, r ∈ {0.01, 0.1}, learning rate lr = {0.0002, 0.001} and weight decay in {0.0, 1e -6}. The hidden state of the LSTM is of size 150. We use Adam optimizer (Kingma & Ba, 2014) 

B.2 ADDITIONAL RESULTS ON BEER:

As in Lei et al. (2016) ; Bastings et al. (2019) , we also train EDUCE on the prediction of each of the first three aspects (the 4th aspect they do not experiment on)5 . Table 6 reports the precision of gold rationales for that aspect and the total percentage of selected words, values for Lei et al. (2016) ; Bastings et al. (2019) are taken from Bastings et al. (2019) . We use C = 10 and λ = 0.1. We consider a word as selected if any of the concept selects it, and count it as one selected even if it is selected by multiple concepts. While the results are less convincing than the aforementioned rationales models that are specifically designed for this task, we see that extracted excerpts match gold rationales to a certain extent, especially for the aspects Appearance and Smell. For the Palate aspect, manual inspection shows us that for some seeds, the model defines a concept dedicated to the notion of numbers (e.g. corresponding to excerpts "12 ounce bottle") that is consistent, yet not part of the gold rationale.

C ADDITIONAL QUALITATIVE EXAMPLES ON DBPEDIA AND AGNEWS:

Class Sports: just imagine what david ortiz could do on a good night ' s rest . ortiz spent the night before last with his baby boy , d ' angelo , who is barely 1 month old . he had planned on attending the red sox ' family day at fenway park yesterday morning , but he had to sleep in . after all , ortiz had a son at home , and he . . . Class Sports: foxborough -looking at his ridiculously developed upper body , with huge biceps and hardly an ounce of fat , it ' s easy to see why ty law , arguably the best cornerback in football , chooses physical play over finesse . that ' s not to imply that he ' s lacking a finesse component , because he can shut down his side of the field much as deion sanders . . . Class Sports: ap -american natalie coughlin won olympic gold in the 100-meter backstroke monday night . coughlin , the only woman ever to swim under 1 minute in the event , finished first in 1 minute , 0 . 37 seconds . kirsty coventry of zimbabwe , who swims at auburn university in alabama , earned the silver in 1 00 . 50 . laure manaudou of france took bronze in 1 00 . 88 . Class World: lourdes , france -a frail pope john paul ii , breathing heavily and gasping at times , celebrated an open-air mass on sunday for several hundred thousand pilgrims , many in wheelchairs , at a shrine to the virgin mary that is associated with miraculous cures . at one point he said help me in polish while struggling through his homily in french . . . Class World: najaf , iraq -explosions and gunfire rattled through the city of najaf as u . s . troops in armored vehicles and tanks rolled back into the streets here sunday , a day after the collapse of talks -and with them a temporary cease-fire -intended to end the fighting in this holy city . . . Class World: kabul , afghanistan -government troops intervened in afghanistan ' s latest outbreak of deadly fighting between warlords , flying from the capital to the far west on u . s . and nato airplanes to retake an air base contested in the violence , officials said sunday . . . Class Business: london ( reuters ) -the dollar dipped to a four-week low against the euro on monday before rising slightly on profit-taking , but steep oil prices and weak u . s . data continued to fan worries about the health of the world ' s largest economy .

Class Business: new york ( reuters ) -the dollar extended gains against the euro on monday after a report on flows into u . s . assets showed enough of a rise in foreign investments to offset the current account gap for the month .

Class Business: reuters -apparel retailers are hoping theirback-to-school fashions will make the grade amongstyle-conscious teens and young adults this fall , but it couldbe a tough sell , with students and parents keeping a tighterhold on their wallets . Predicted: Business, True Class: Sci/Tech hong kong ( reuters ) -dell inc . &lt dell . o&gt , the world ' s largest pc maker , said on monday it has left the low-end consumer pc market in china and cut its overall growth target for the country this year due to stiff competition in the segment .

Predicted: Business, True Class: World supporters and rivals warn of possible fraud government says chavez ' s defeat could produce turmoil in world oil market .

Predicted: Sports, True Class: World athens , greece -top american sprinters jason lezak and ian crocker missed the cut in the olympic 100-meter freestyle preliminaries tuesday , a stunning blow for a country that had always done well in the event . pieter van den hoogenband of the netherlands and australian ian thorpe advanced to the evening semifinal a day after dueling teenager michael phelps in the 200 freestyle , won by thorpe . . . Predicted: Sci/Tech, True Class: Business san francisco -in the latest of a series of product delays , intel corp . has postponed the launch of a video display chip it had previously planned to introduce by year end , putting off a showdown with texas instruments inc . in the fast-growing market for high-definition television displays .

Predicted: World, True Class: Sports athens -the mistakes were so minor . carly patterson ' s foot scraping the lower of the uneven bars . courtney kupets ' tumbling pass that ended here instead of there . mohini bhardwaj ' s slight stumble on the beam . Predicted: World, True Class: Sports thens , aug . 17 -so michael phelps is not going to match the seven gold medals won by mark spitz . and it is too early to tell if he will match aleksandr dityatin , the soviet gymnast who won eight total medals in 1980 . but those were not the . . . Class Company: transurban manages and develops urban toll road networks in australia and north america . it is a top 50 company on the australian securities exchange ( asx ) and has been in business since 1996 . in australia transurban has a stake in five of sydney ' s nine motorways and in melbourne it is the full owner of citylink which connects three of the city ' s major freeways . in the usa transurban has ownership interests in the 495 express lanes on a section of the capital beltway around washington dc .

Class Animal: the red-necked falcon or red-headed merlin ( falco chicquera ) is a bird of prey in the falcon family . this bird is a widespread resident in india and adjacent regions as well as sub-saharan africa . it is sometimes called turumti locally . the red-necked falcon is a medium-sized long-winged species with a bright rufous crown and nape . it is on average 30-36 cm in length with a wingspan of 85 cm . the sexes are similar except in size males are smaller than females as is usual in falcons .

Class Plant: astroloma is an endemic australian genus of around 20 species of flowering plants in the family ericaceae . the majority of the species are endemic in western australia but a few species occur in new south wales victoria tasmania and south australia . species include astroloma baxteri a . cunn . ex dc . astroloma cataphractum a . j . g . wilson ms astroloma ciliatum ( lindl . ) druce astroloma compactum r . br . astroloma conostephioides ( sond . ) f . muell . ex benth .

Class Album: stars and hank forever was the second ( and last ) release in the american composers series by the avant garde band the residents . the album was released in 1986 . this particular release featured a side of hank williams songs and a medley of john philip sousa marches . this was also the last studio album to feature snakefinger . kaw-liga samples the rhythm to michael jackson ' s billie jean and did well in europe it is as close as the residents ever got to a bona fide commercial hit .

Class Written Work: fire ice is the third book in the numa files series of books co-written by best-selling author clive cussler and paul kemprecos and was published in 2002 . the main character of this series is kurt austin . in this novel a russian businessman with tsarist ambitions masterminds a plot against america which involves triggering a set of earthquakes on the ocean floor creating a number of tsunami to hit the usa coastline . it is up to kurt and his team and some new allies to stop his plans . 13 (only 36 concepts are identified as present and we report those) to compare with the ones learned with C = 10 and λ = 0.1 in Table 11 (blue square in Figure 4b ). With C = 50, the learned concepts qualitatively seem more consistent, even if the a posteriori concept accuracy is 81.37% for the model with C = 50 which excerpts we report in Table 13 , which is slightly less than the value (83.90%) for the model with C = 10. Hence, learned concepts with C = 50 might be more similar to each others (e.g. less separable). Furthermore, an explanation with more concepts to look at and interpret is also harder to parse, which is why we consider that the binary intermediate representation should be low-dimensional. In future work, we would like to conduct a large scale experimentation involving human judgments to conclude on the interpretability benefits of different values of C. Note that with a too small value of C (here, C = 5) final accuracy is harmed as the intermediate representation is too coarse. ponderosa steakhouse and / is a medium-sized long-winged species with / is an endemic australian genus of around / is a venomous tree-dwelling snake . / is a brazilian butterfly swimmer . rooted in / is a family of / is a saltwater pond in Concept 36 a naive bayes / a bird of prey / an endemic australian genus / a venomous tree-dwelling snake / a young adult / an american sociobiologist / a brazilian butterfly swimmer / a large ruminant mammal / the largest and most important cave

:

We also prevent stop words to be outside of the text length.

We do not show the amount of neutral words selected as it squeezes the histograms.

https://www.beeradvocate.com/

Tables3 and 2reports results of the best seed for the cross-validated hyperparameters.

DISCUSSION AND PERSPECTIVESWe propose a new self-interpretable model, EDUCE, that maps inputs to an easy-to-interpret representation on which the output prediction relies. We experimentally demonstrate its ability (i) to extract semantically meaningful concepts described by consistent excerpts, and (ii) to ground its output prediction on the presence/absence of these concepts in each input. EDUCE generates relevant explanations while retaining high predictive performance. As future direction for research, we contemplate using hierarchical representations, as human notions can often be represented with hierarchical structures, for example as in WordNet(Miller, 1995). Furthemore, our principles can be applied on different types of inputs like images, and we provide initial experiment in Supplementary Section E.

Bastings et al. (2019) call the third aspect Taste but they actually refer to Palate.

We construct this dataset by combining random images from the Caltech Bird 200-2011 dataset(Wah et al., 2011) with images of the cats-and-dogs Kaggle dataset(Kaggle, 2013)

Figures and Tables:

Figure 1: Illustration of EDUCE's prediction process on AGNews topic classification dataset (Zhang & LeCun, 2015) examples. The different steps are explained in the text.

excerpts are consistent yet are extracted in texts from multiple output classes. The excerpts extracted in the examples of Figure 2a are consistent with these interpretations. These results show how easily the classification of any text can be explained by the detection of multiple, relevant, and intelligible concepts. More qualitative examples are in supplementary Section C. Note that supplementary

Figure 2: Interpretation of EDUCE prediction through concept analysis.

(a) Per class concept frequency. (b) Output classifier weight matrix δ. (c) Repartition of words labels for concepts 3, 5, 7, 8. (d) SST test performance, output accuracy vs a posteriori concept accuracy. A posteriori concept accuracy is not applicable for Baseline, Lei et al. (2016) and Bastings et al. (2019).

Figure 3: Results on the SST dataset. 5 MULTI ASPECT SENTIMENT ANALYSIS EXPERIMENT BeerAdvocate dataset While our paradigm is different than Lei et al. (2016); Bastings et al. (2019), we also perform a multi-aspect sentiment regression experiment on the pre-processed subset of the BeerAdvocate 3 dataset (McAuley et al., 2012). It consists of 260, 000 beer reviews where ratings for four aspects (look, smell, palate and taste) are given, as well as an overall rating. The reviews are separated into training/validation sets specific to each aspect, and ratings are mapped to scalars in [0, 1]. This is a regression task, so we modify the Baseline model and EDUCE in the last layer of the prediction with a sigmoid activation function, and use Mean Squared Error (MSE) instead of cross-entropy for L output (y, z(x), δ). McAuley et al. (2012) provided a test set (994 reviews) with sentence-level rationale annotations: different parts of each review are annotated with one (or multiple) aspect label, indicating what aspect it covers (referred to as gold rationales). Importantly, we do not use these gold rationales during training or validation, but use to evaluate our model. Hyperparameters selection is similar to our classification experiments.

Figures4a and 4b(the second is a zoom to better see the differences between 10, 20 and 50 concepts) show the trade-off between concept accuracy and final accuracy on the DBPedia dataset, for multiple values of C and different value of λ. Each line is a value of C. Each marker is a different value of λ, and values are (from left to right markers) {0, 0.01, 0.1, 0.5}. Separately, we show examplar excerpts of concepts learned with C = 50 and λ = 0.5 (purple dot in Figure4b) in Table13(only 36 concepts are identified as present and we report those) to compare with the ones learned with C = 10 and λ = 0.1 in Table11(blue square in Figure4b). With C = 50, the learned concepts qualitatively seem more consistent, even if the a posteriori concept accuracy is 81.37% for the model with C = 50 which excerpts we report in Table13, which is slightly less than the value (83.90%) for the model with C = 10. Hence, learned concepts with C = 50 might be more similar to each others (e.g. less separable). Furthermore, an explanation with more concepts to look at and interpret is also harder to parse, which is why we consider that the binary intermediate representation should be low-dimensional. In future work, we would like to conduct a large scale experimentation involving human judgments to conclude on the interpretability benefits of different values of C. Note that with a too small value of C (here, C = 5) final accuracy is harmed as the intermediate representation is too coarse.

(a) DBPedia test performance, output accuracy vs a posteriori concept accuracy, for multiple values of C and multiple values of λ. (b) DBPedia test performance, output accuracy vs a posteriori concept accuracy, for multiple values of C and multiple values of λ. Zoom between 95% and 98% final accuracy.

Figure 4: DBPedia test performance, output accuracy vs a posteriori concept accuracy, for multiple values of C and multiple values of λ.

: In Step 2 the excerpt Retails sales bounced back is identifiable as relating to economy/finance, contrarily to the government said, made official what which trigger the concept politics.We rephrase our desiderata for consistency and separability as classification of the excerpts, with as many classes as the number of concepts. An external classifier should recognize that elements in the set {s c (x)|∀x z c (s c (x)) = 1} belong to concept c, and {s c (x)|∀x z c (s c (x)) = 1} belong to concept c and not c. Recall that we do not have labels for the concepts. The ground-truth label of each extracted excerpt s c (x) is the index c of the dimension of z(x) for which it was extracted. We learn a linear concept classifier without bias, parametrized by a weight matrix θ ∈ R |C|×d followed by a softmax activation function, returning p θ (c |s c

Table content: <table/>

Lei et al. (2016);Yu et al. (2019);Bastings et al. (2019) develop interpretable models for NLP tasks by selecting rationales, i.e. parts of text, on which a consequent model bases its prediction. A rationale acts as a justification supporting the prediction and is different from our definition of excerpt. In EDUCE, excerpts support the attribution of presence of concepts, and this attribution supports the

Table content: <table/>

Test performance on DBPedia and AGNews (mean ± SEM). binary intermediate representation, but are trained without concept loss.

Table content: <table><tr><td>Data</td><td>Model</td><td>Output Acc. (%)</td><td>A Posteriori Concept Acc. (%)</td></tr><tr><td>DBPedia</td><td colspan="2">EDUCE No Concept Loss No Concept Loss+L1 96.5 ± 0.2 97.0 ± 0.1 97.4 ± 0.1 Baseline 98.75 ± 0.0</td><td>82.4 ± 0.8 25.9 ± 0.6 44 ± 2.6 n/a</td></tr><tr><td>AGNews</td><td colspan="2">EDUCE No Concept Loss No Concept Loss+L1 86.3 ± 0.7 87.5 ± 0.2 88.2 ± 0.1 Baseline 92.08 ± 0.1</td><td>78 ± 6.5 31.0 ± 0.7 56 ± 3.2 n/a</td></tr></table>

Business: sports retailer jjb yesterday reported a near 25 drop in profits and continuing poor sales , and ended shareholders #39 hopes of a takeover by announcing that a potential bidder had walked away . Class World: bangkok , thailand sept . 30 , 2004 -millions of volunteers led by emergency teams fanned out across thailand on thursday in a new drive to fight bird flu after the prime minister gave officials 30 days to eradicate the epidemic . Class Sports: the spanish government responded to diplomatic pressure from britain yesterday by starting a search for fans who racially abused england players during a quot friendly quot football match with spain . Class Sci/Tech: los angeles ( reuters ) -a group of technology companies including texas instruments inc . &lt txn . n&gt , stmicroelectronics &lt stm . pa&gt and broadcom corp . &lt brcm . o&gt , on thursday said they will propose a new wireless networking standard up to 10 times the speed of the current generation . (a) AGNews test examples correctly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept. Concept 0 software services giant / moonwalk to home / video display chip / downloading music . Concept 1 upcoming my prerogative video / his ever-growing swimming / launch of a video display chip / illegality of downloading Concept 2 oil market . / oil giant sibneft / oil prices and / corp . &lt brcm Concept 4 cash settlement of up to #36 50 million / six-year deal worth about $40 million / the dollar dipped to a four-week low against the euro / five shares , Concept 6 olympic 100-meter freestyle / sox ' family / olympics should help / athletes were already Concept 8 frail pope john paul / indian army major shot / unions representing workers / goverment representatives . (b) Examples of excerpts that are extracted accross the test set, corresponding to the concepts detected in Figure 2a. Colors match the colors used in Figure 2a.

Table content: <table/>

Examples of excerpts extracted on the Beer test set.

Table content: <table><tr><td>Concept 0 fruity esters , and/ fruits , caramelized pecans , and/ tof-</td><td colspan="3">Concept Appearance Smell Palate Taste</td></tr><tr><td>fee and caramel accents ,/ coffee and chocolate flavors/</td><td>Concept 0</td><td>0.85</td><td>55.75 1.79 40.60</td></tr><tr><td>earthy hop resin .</td><td>Concept 1</td><td>0.54</td><td>5.48 11.70 30.58</td></tr><tr><td>Concept 2 creamy and a/ chewy and rich and drinkability/ creamy</td><td>Concept 2</td><td>1.22</td><td>0.90 78.18 15.03</td></tr><tr><td>mouthfeel that/ smooth and just velvety on the/ thick ,</td><td>Concept 3</td><td>1.46</td><td>92.39 0.30 5.34</td></tr><tr><td>and</td><td>Concept 4</td><td>45.33</td><td>0.23 0.26 0.16</td></tr><tr><td>Concept 3 rich malt scents/ aroma is quite hoppy with big citrus/</td><td>Concept 5</td><td>7.27</td><td>22.50 2.27 28.86</td></tr><tr><td>tons of different sweet malts , toffe/ smells extremely</td><td>Concept 6</td><td>0.36</td><td>5.86 6.40 28.73</td></tr><tr><td>roasty/ boom of grapefruity</td><td>Concept 7</td><td>97.09</td><td>0.98 0.45 0.53</td></tr><tr><td>Concept 7 good head and lacing/ beautiful golden-amber color/ creamy tan head/ nice deep brown color/ proud head has settled . nothing</td><td>Concept 8 Concept 9</td><td>0.18 0.00</td><td>0.09 18.07 5.93 0.00 68.42 10.53</td></tr><tr><td>Concept 9 carbonation is graceful/ drinkability is excellent/ mouth-</td><td/><td/><td/></tr><tr><td>feel is wonderful/ mouthfeel is exemplary/ drinkability :</td><td/><td/><td/></tr><tr><td>excellent</td><td/><td/><td/></tr></table>

, batches of size 25. For the Baseline model, we followBastings et al. (2019) and use weight decay 1e -6 and dropout {0.5}.BEER We try the following ranges of hyperparameters λ ∈ {0.001, 0.01, 0.1}, r ∈ {0.001, 0.01, 0.1}, learning rate lr = {0.0005, 0.001}. The hidden state of the LSTM is of size 200. We use Adam optimizer(Kingma & Ba, 2014), batches of size 256. We also clip the gradient to the value of 1 and employ an entropy maximization regularization on mean of the Bernoulli distributions ∀c, p α (z c |s c (x), c), weighted by 0.001. We train all models for 250 epochs and use early stopping on the validation set. For the Baseline model on the BEER dataset, we use the same hyperparameters as inBastings et al. (2019) (lr = 0.0004, hidden dimension 200, dropout in {0.1, 0.2} and weight decay 1e -6) and train for 100 epochs. Test performance on DBPedia and AGNews (mean ± SEM) for all values of λ and λ L1 .

Table content: <table><tr><td>Dataset</td><td>Model</td><td>Output Acc. (%)</td><td>A Posteriori Concept Acc. (%)</td><td>Sparsity</td></tr><tr><td/><td>Baseline</td><td>98.75 ± 0.0</td><td>n/a</td><td>n/a</td></tr><tr><td/><td>No Concept Loss</td><td>97.4 ± 0.1</td><td>25.9 ± 0.6</td><td>5.51 ± 0.1</td></tr><tr><td>DBPEDIA</td><td colspan="2">EDUCE λ = 0.001 EDUCE λ = 0.01 EDUCE λ = 0.1 No Concept Loss + L1, λL 1 = 0.001 97.4 ± 0.1 97.3 ± 0.1 97.3 ± 0.1 97.0 ± 0.1</td><td>28 ± 1.7 39 ± 3.5 82.4 ± 0.8 27 ± 1.9</td><td>5.60 ± 0.1 5.39 ± 0.1 3.5 ± 0.2 5.5 ± 0.1</td></tr><tr><td/><td colspan="2">No Concept Loss + L1, λL 1 = 0.01 97.46 ± 0.1</td><td>28 ± 2.1</td><td>5.12 ± 0.0</td></tr><tr><td/><td>No Concept Loss + L1, λL 1 = 0.1</td><td>96.5 ± 0.2</td><td>44 ± 2.6</td><td>3.1 ± 0.1</td></tr><tr><td/><td>No Concept Loss + L1, λL 1 = 1.00</td><td>84 ± 2.0</td><td>77 ± 2.4</td><td>1.31 ± 0.0</td></tr><tr><td/><td>Baseline</td><td>92.08 ± 0.1</td><td>n/a</td><td>n/a</td></tr><tr><td/><td>No Concept Loss</td><td>88.2 ± 0.1</td><td>31.0 ± 0.7</td><td>5.2 ± 0.2</td></tr><tr><td>AGNews</td><td colspan="2">EDUCE λ = 0.001 EDUCE λ = 0.01 EDUCE λ = 0.1 No Concept Loss + L1, λL 1 = 0.001 88.3 ± 0.2 88.4 ± 0.1 88.6 ± 0.1 87.5 ± 0.2</td><td>33 ± 1.6 40 ± 2.1 78 ± 6.5 31.6 ± 0.7</td><td>5.2 ± 0.2 4.6 ± 0.2 2.4 ± 0.2 5.1 ± 0.2</td></tr><tr><td/><td>No Concept Loss + L1, λL 1 = 0.01</td><td>88.7 ± 0.2</td><td>34 ± 1.4</td><td>4.6 ± 0.2</td></tr><tr><td/><td>No Concept Loss + L1, λL 1 = 0.1</td><td>86.3 ± 0.7</td><td>56 ± 3.2</td><td>2.2 ± 0.1</td></tr><tr><td/><td>No Concept Loss + L1, λL 1 = 1.00</td><td>53 ± 3.2</td><td>89 ± 2.8</td><td>0.50 ± 0.0</td></tr></table>

Examples of excerpts extracted on the Beer test set.

Table content: <table><tr><td>Concept</td><td colspan="7">fruity esters , and/ fruits , caramelized pecans , and/ toffee and caramel accents ,/ coffee</td></tr><tr><td>0</td><td colspan="3">and chocolate flavors/ earthy hop resin .</td><td/><td/><td/></tr><tr><td>Concept</td><td colspan="7">delicious and with/ enjoyed this beer/ damn good beer/ nice stout that/ good beer to</td></tr><tr><td>1</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Concept</td><td colspan="7">creamy and a/ chewy and rich and drinkability/ creamy mouthfeel that/ smooth and just</td></tr><tr><td>2</td><td colspan="2">velvety on the/ thick , and</td><td/><td/><td/><td/></tr><tr><td>Concept</td><td colspan="7">rich malt scents/ aroma is quite hoppy with big citrus/ tons of different sweet malts , toffe/</td></tr><tr><td>3</td><td colspan="4">smells extremely roasty/ boom of grapefruity</td><td/><td/></tr><tr><td>Concept</td><td colspan="7">octoberfest : say the word and you probably/ 2004 vintage : pours/ 12 ounce bottle acquired</td></tr><tr><td>4</td><td colspan="7">in a trade/ i hate to be the guy/ 12oz bottle , best before 9/09 , sampled 5/16/09</td></tr><tr><td>Concept</td><td colspan="7">which is fantastic/ it is delicious/ is intensely packed/ and a lovely/ builds a wonderful</td></tr><tr><td>5</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Concept</td><td colspan="7">'m really digging/ a damn good/ a super fresh growler/ the better abitas/ a great traditional</td></tr><tr><td>6</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Concept</td><td colspan="7">good head and lacing/ beautiful golden-amber color/ creamy tan head/ nice deep brown</td></tr><tr><td>7</td><td colspan="3">color/ proud head has settled . nothing</td><td/><td/><td/></tr><tr><td>Concept</td><td colspan="7">easy to drink/ hot summer day/ drinkability easy/ easy to down/ can easily turn this 'strong</td></tr><tr><td>8</td><td>ale ' into a session</td><td/><td/><td/><td/><td/></tr><tr><td>Concept</td><td colspan="7">carbonation is graceful/ drinkability is excellent/ mouthfeel is wonderful/ mouthfeel is</td></tr><tr><td>9</td><td colspan="3">exemplary/ drinkability : excellent</td><td/><td/><td/></tr><tr><td/><td>Model</td><td colspan="2">Appearance</td><td colspan="2">Smell</td><td colspan="2">Palate</td></tr><tr><td/><td/><td>Prec.</td><td>Extr.</td><td>Prec.</td><td>Extr.</td><td>Prec.</td><td>Extr.</td></tr><tr><td/><td>EDUCE</td><td colspan="6">81 ± 6.9 4.9 ± 0.5 79 ± 3.1 4.8 ± 0.4 50 ± 6.6 4.1 ± 0.6</td></tr><tr><td/><td>Lei et al. (2016)</td><td>96.3</td><td>14</td><td>95.1</td><td>7</td><td>80.2</td><td>7</td></tr><tr><td/><td colspan="2">Bastings et al. (2019) 98.1</td><td>13</td><td>96.8</td><td>7</td><td>89.8</td><td>7</td></tr></table>



Table content: <table/>

AGNews test examples correctly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.

Table content: <table/>

AGNews test examples uncorrectly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.

Table content: <table/>

Class Educational Inst.: avon high school is a secondary school for grades 9-12 located in avon ohio . its enrollment neared 1000 as of the 2008-2009 school year with a 2008 graduating class of 215 . the school colors are purple and gold . the school mascot is an eagle . the avon eagles are part of the west shore conference . they will be moving to the southwestern conference beginning in the 2015-2016 school year . Class Artist: vicky hamilton ( born april 1 1958 ) is an american record executive personal manager promoter and club booker writer ( journalist playwright and screenwriter ) documentary film maker and artist . hamilton is noted for managing the early careers of guns n ' roses poison and faster pussycat for being a management consultant for mötley crüe and stryper a 1980s concert promoter on the sunset strip and a club booker at bar sinister from 2001 to 2010 . [...] Class Athlete: james jim arthur bacon ( birth registered october-december 1896 in newport districtdeath unknown ) was a welsh rugby union and professional rugby league footballer of the 1910s and ' 20s and coach of the 1920s playing club level rugby union ( ru ) for cross keys rfc and representative level rugby league ( rl ) for great britain and wales and at club level for leeds as a wing or centre i . e . number 2 or 5 or 3 or 4 and coaching club level rugby league ( rl ) for castleford . Class Office Holder: jonathan david morris ( october 8 1804 -may 16 1875 ) was a u . s . representative from ohio son of thomas morris and brother of isaac n . morris . born in columbia hamilton county ohio morris attended the public schools . he studied law . he was admitted to the bar and commenced practice in batavia ohio . he served as clerk of the courts of clermont county . morris was elected as a democrat to the thirtieth congress to fill the vacancy caused by the death of thomas l . Class Mean Of Transp.: german submarine u-32 was a type viia u-boat of nazi germany ' s kriegsmarine during world war ii . her keel was laid down on 15 march 1936 by ag weser of bremen as werk 913 . she was launched on 25 february 1937 and commissioned on 15 april with kapitänleutnant ( kptlt . ) werner lott in command . on 15 august 1937 lott was relieved by korvettenkapitän ( krv . kpt . ) paul büchel and on 12 february 1940 oberleutnant zur see ( oblt . z . s . ) hans jenisch took over he was in charge of the boat until her loss . Class Building: the tootell house ( also called king ' s row or hedgerow ) is a house at 1747 mooresfield road in kingston rhode island that is listed on the national register of historic places . the two-story wood-shingled colonial revival house on a 3-acre ( 12000 m2 ) tract was designed by gunther and beamis associates of boston for mr . & mrs . f . delmont tootell and was built in 1932-1933 . house design was by john j . g . gunther and elizabeth clark gunther was the landscape architect for the grounds . Class Village: angamoozhi is a village in pathanamthitta district located in kerala state india . angamoozhi is near seethathodu town . geographically angamoozhi is a high-range area . it is mainly a plantation township . both state run ksrtc and private operated buses connect angamoozhi to pathanamthitta city . tourist can avail the travelling facility by ksrtc service ( morning 5 30 from kumili and 11 30 from pathanamthitta ) in between kumili and pathanamthitta via vallakkadavu angamoozhi kakki dam and vadaserikkara and can enjoy the beauty of the forest .[ citation needed ]

Table content: <table/>

DBPedia test examples correctly classified by EDUCE. Underlined set of words are excerpts extracted, one color per concept.

Table content: <table><tr><td>D VARYING THE NUMBER OF CONCEPTS ON DBPEDIA</td></tr><tr><td>Figures 4a and 4b</td></tr></table>

Concept 0 member of sire records act / primate native to / an indictment of policy changes / annexation of mrauk-u kingdom / regents of the university of / litterateur historian and activist / baseman first baseman and manager / democrat to the thirtieth congress to fill the vacancy caused / lieutenant commander henry crommelin / governor ' s residence and / prime minister of canada arthur meighen who held office / committee most of the district headquarters / forest on the western slopes on / river range in / album from stone temple / forest home of / seaside town of southwold and centers / whisky distillery . the distillery / favela of capão redondo in / bar and moved / hospital ships was built / house at 1747 mooresfield road / island west of the dingle peninsula . at / village in pathanamthitta district located Concept 7shoes and other dance shoes and apparel . / racehorse who won at / his section macrophylli / players in the / man who flees the city and works in / biography of jesse lauriston livermore . the / painter who lived in / artist . hamilton is noted for managing the / footballer of the 1910s and ' 20s and coach of / singer from finland . elomaa was born in / exhibition at the / resident paid for / fan gyhirych in the romanian academy / cultivar raised at the university / a guide to british schools / a secondary school / college oxford university he reviewed poetry / an american college / a degree in chemistry / a mars-orbiting aeronomy probe / is the volleyball academy / taluka of amravati Concept 14university . the first of the / school and it was released on / school during the early 1990s specifically between / school for grades 9-12 located in / school later returning to / school . she graduated from dartmouth college in 1998 with / schools . he studied law . he was admitted to / university also known as evolutionsmuseet / high located in / skole which covers løken momoen Concept 15 bird of the arabian nights . / bird of prey in the falcon family . this bird / subfamily within the family euphorbiaceae . / hyena label in 2006 / faunistic studies and biogeography . / dolphins . he / mammal of africa / fossil collection in scandinavia / cat about to pounce . bruce first listed in the / considered by the / identified by melville / listed on the / delivered to nearly 8200 homes monday through / built in 1959 and has undergone some modifications from the / a 1965 us number 1 hit single for herman / powered by a miller four-cylinder . it / the former first lady of new / launched on 25 february 1937 and commissioned on / built in 1932-1933 . house design was by john / an uninhabited steep rocky island west of the . / heliconiinae commonly called / name has been misapplied for / label in 1992 . / production created by / based in egremont / founded in 2005 / producer based in toronto who has / united states ) . / company thornaby-on-tees for burdick and cook in / sweden established in / name could derive from Concept 23 newspaper comic strip reprints but later on original material / first described by / published his first description of it in / pornographic parodies of / was published in / newspaper reported in / romance novels from / published author himself as / newspaper reporter and editor and ( briefly ) a / motorcyclist magazine stated / a copy of / his novel on / first written notice about Concept 24 a church established / an historical text / a building of the trinity church / an american beaux-arts architect / 145th state house / a victorian era admiral-class battleship / a house at / a tourist attraction Concept 27 single-engined jet . the company / light golden color / list of motorcycles scooters and mopeds produced / hits released only / bollywood film released / whaling ship called / trains indian army personnel / american film and television screenwriter best / american racecar driver / united states navy commander and / type viia u-boat of nazi germany ' s kriegsmarine / submarine museum managed Concept 28 gecko as they claim their cars stick / bird of prey in the falcon family . this bird / subfamily within the family / japanese phantom thief / eagle . the avon eagles / brazilian butterfly swimmer / monoplane with wire braced wings / subsidiary of sparrow health / bird species including ducks great crested grebes swans and geese Concept 29 largest icelandic commercial banks / is the only mare / is the pacific ocean / water by the spoonful / agua fria high school / karnik ( devanagari / represents the creuse ' s 2nd constituency / a mars-orbiting aeronomy probe / is a large boulder / is an uninhabited steep rocky island west of the dingle / is situated on the lap of vindhya ranges Concept 30 a top 50 company / is portuguese wine grape / the original soundtrack on the sony bmg label / an american disney comedy franchise / a free independent alternative newspaper / the statutory and regulatory organization / an american record executive personal manager promoter / an american stock car racing driver / a grain buyer / high-speed electrical multiple units / an artist-run centre / to supply drinking water Concept 31 was only newspaper comic / is a selection / was released in 1986 . this particular release / are 1993 pornographic parodies / is the third book / was the original dj / is the eleventh / is a survey launch / is the world ' s tenth longest river Concept 34 british pottery manufacturing firm / a collection of american singer laura / italian film . it stars actor gabriele / damned poet and somewhat heartbreaking images / american comic strip creator / poker player pioneer poker theorist / female bodybuilding champion and pop singer / scientist astronomer and botanist Concept 35

Table content: <table><tr><td>in ilam district</td></tr><tr><td>Concept 1 Concept 2 Concept 4 Concept 6 correspondents located in Concept is portuguese wine grape william james young ( 1800-1870 ) probably born in / april 1983 -30 december 1992 ) was a british-bred / ramos ' crepidium ) is a member / brian wilson presents smile / film starring chad everett sarah / robert service then a professor / st . george ' s r . c . primary / vicky hamilton ( born april / james jim arthur bacon ( birth registered october-december / jonathan david morris ( october / clarence l . evans ( de-113 ) was a cannon-class / arthur goodson house a tribe of / a neighborhood in / a village in 9 top 50 company / of the five operating divisions of the thomson corporation / rankings 1978/1979 the professional world rankings / collection of educational dvds tv series music / independent alternative newspaper / leading public university in the field of agriculture / network manager at the national / hong kong cross-media creator / electrical multiple units / artist-run centre located in vancouver bc with a vision / supply drinking water / centre of Ørskog municipality Concept 10 an international producer / last studio album / a 1971 road movie directed by monte hellman starring singer-songwriter / a damned poet / fryderyk chopin university of music / a british musician . he was a bassist / a french passport / a canadian record producer music publisher Concept 11 is a top 50 company on the / is portuguese wine grape that is used in / is a demo tape by / was first broadcast in syndication on / was a regional fashion entertainment and / is established under the / is an american record executive personal manager promoter and / was an american racecar driver . primarily / was a merchant and public official from / was a 4337-ton steamship built by / is a house at 1747 mooresfield road in / is located in / is a police station at Concept 17 provides public bus transportation Concept 21 31 march 2014 pfzw / april 1983 -30 december 1992 / april 27 1993 / july 6 1975 / 1933 to july 28 1934 / september 29 1962 / born april 1 1958 / birth registered october-december 1896 / born 5 august 1946 / november 18 1985 / december 29 1962 22 an institute of Concept Concept business since 1996</td></tr><tr><td>12</td></tr></table>

Peer review (iclr2020):
The paper introduces a new concept-based interpretability method that lies in the family of self-interpretable models (i.e. it*s not a post-hoc method). Self-interpretability is achieved by a two-stage model: First, a concept-extractor finds the related pieces of consecutive words (excerpts) in a given text that are related to a concept among a set of given concepts (if any), then the model makes its predictions solely based on the presence or absence of concepts (binary). The most useful part of the algorithm is that there is no need for concept annotations. The work then experimentally shows that their method, although does not outperform a non-self-interpretabile baseline but has better performance and interpretation compared to rival methods. The paper is quite well-written. The introduced method is well-justified and the use of concept-based self-interpretable models is very useful to the field. It is also interesting to see that the performance is comparable to non-self-nterpretable baselines which would make a case for the use of self-interpretable models. I have two main concerns with this work. First, there is the novelty concern. The idea of unsupervised extraction of concepts for interpretability was introduced before (https://arxiv.org/pdf/1902.03129.pdf) and is not discussed by the authors (although the utilized terminology is very similar). Authors should make a much more comprehensive discussion of what already exists in the concept-based interpretability literature and make the contribution of this work more clear (unsupervised concept extraction for a self-interpretable model instead of post-hoc interpretations). Secondly, although some of the objective experimental results (BeerAdvocate results, to some degree) suggest that the method is indeed capable of discovering meaningful, consistent, and separable concepts on its own, there is not enough discussion of why it actually should be the case. It*s very easy to assume the introduced training procedure to extract separable but meaningless concepts(i.e. the excerpts of a concept are separable from that of other concepts and are consistent with each other in the eyes of the network while they are not consistent with a concept in the eye of human rationale; both loss terms will be minimized but there is no human-interpretability. One big issue with the experiments section is the A Posteriori Concept Acc metric as it only measures the separability and consistency of discovered concepts; first of all, it can be high while the extracted concepts are meaningless to humans, secondly, using it as a measure of comparison to rival methods is not quite fair as the introduced method is directly optimizing for this metric. The subjective results are interesting but not convincing. Adding a section for human subject experiments (following the previous work in concept-based interpretability) would make the results section much crisper. It also seems unlikely that all of the discovered concepts follow the desiderata and the fact that this is not mentioned makes it even more difficult to assess the amount of cherry-picking in the mentioned subjective results; there should be a discussion of cases of failure and why it happens. My score will change accordingly as the authors address the raised issues. A few questions and suggestions: * It would be more interesting to show subjective results of the model for cases of mistake; does the interpretability make sense when the model*s prediction is wrong? * It seems necessary to discuss how much the results are sensitive to selecting C? If the performance is robust, it would be interesting to see what happens for a large C? Does it discover more fine-grained concepts or most of the additional concepts would be null? * For the model architecture, why did the authors choose to use the straight-through estimator and not other methods (e.g. concrete layers, ...) * Some of the figures should be larger (much larger) Thanks again for a well-organized and easy-to-follow paper.