Title: LEARNING CORRESPONDED RATIONALES FOR TEXT MATCHING

Abstract: The ability to predict matches between two sources of text has a number of applications including natural language inference (NLI) and question answering (QA). While flexible neural models have become effective tools in solving these tasks, they are rarely transparent in terms of the mechanism that mediates the prediction. In this paper, we propose a self-explaining architecture where the model is forced to highlight, in a dependent manner, how spans of one side of the input match corresponding segments of the other side in order to arrive at the overall decision. The text spans are regularized to be coherent and concise, and their correspondence is captured explicitly. The text spans -rationales -are learned entirely as latent mechanisms, guided only by the distal supervision from the end-to-end task. We evaluate our model on both NLI and QA using three publicly available datasets. Experimental results demonstrate quantitatively and qualitatively that our method delivers interpretable justification of the prediction without sacrificing state-of-the-art performance. Our code and data split will be publicly available.

1. INTRODUCTION:

Text matching serves as a key subroutine facilitating many natural language processing (NLP) tasks including natural language inference (NLI) (Bowman et al., 2015; Wang & Jiang, 2016a; Khot et al., 2018) , paraphrase detection (Wang et al., 2017c) , question answering (QA) (Andreas et al., 2016a; Wang et al., 2017b; Chen et al., 2017) , and others. Much of the progress across such tasks has come from the use of flexible neural architectures that can be trained to achieve high performance. However, absent attention or rationale mechanisms, the resulting models lack transparency about how the decisions are reached. Soft attention keeps the model differentiable, typically operates on a word-by-word basis (Parikh et al., 2016; Seo et al., 2016; Chen et al., 2016b; Wang et al., 2016; Wang & Jiang, 2016b) , but does not strictly confer importance as weakly attended words can impact decisions. In contrast, rationales (Lei et al., 2016) as hard selections are challenging to train but provide a certificate of exclusion of any unattended part. We opt for rationales in this paper as our goal is to identify clearly necessary text spans as opposed to individual words, and to do so across the matched texts in a dependent way.

Our work builds on and relates to many recent advances in explaining neural predictions for NLP (Lei et al., 2016; Li et al., 2016; Sharp et al., 2017; Koh & Liang, 2017; Alvarez-Melis & Jaakkola, 2017; 2018) but aims for rationalizing textual matching. Consider answer reranking or selection in the state-of-the-art open domain QA pipeline1 (Wang et al., 2017b) . The end goal in this case is to classify whether the supporting passage indeed has the answer to the specific question. Figure 1 illustrates the problem with two answer candidates and their supporting passages. A rationale that explains the correct match should not only highlight "Galileo Galilei" in the passage but also include two additional facts from the question and their corresponding evidence in the passage (highlighted in red and blue). In this paper, we seek rationales that identify necessary textual spans from each question and, in a dependent way, their short and sufficient corresponding segments in the supporting Figure 1 : Examples of the corresponded rationales for QA. Words highlighted in the same color from the question and a passage is a paired rationale. "Galileo Galilei" is the correct answer since two facts from the question are matched in its passage. On the other hand, the passage of "Isaac Newton" does not cover the second fact, which makes it a wrong answer.

passages. Only the selected text spans in aggregate -the rationales -are used as the basis for the final decision about matching.

We do not assume that any rationale annotations are given. Instead, the rationales are learned as latent selections based on the distal supervision (Mintz et al., 2009) from the downstream text matching task. The rationale mechanism builds on the generator-predictor framework Lei et al. (2016) introduced for sentiment analysis where the generator selects text spans as rationales that are then fed to the predictor. In our case, the generator is tasked with a combinatorial selection across the texts being matched. To make the combinatorial selection easier to learn we first identify key pieces from a single side of the input (e.g. question or hypothesis) and then look for the corresponding spans from the other side in a dependent manner. The predictor finally maps the extracted rationales in aggregate to task-specific values. The entire model along with the rationales is trained in an end-to-end fashion through policy gradient (Williams, 1992) .

We consider both NLI and QA as the evaluation tasks. Extensive experimental results on three publicly available datasets show that the model delivers human-interpretable information while preserving comparable performance to the state-of-the-art. Further, to quantify the quality of generated rationales, we contrast the rationales arising from clean vs adversarial inputs. The generated rationales (as they operated over spans) are fairly robust against inserting irrelevant texts.

2. RELATED WORK:

Learning interpretable models Improving interpretability has been approached primarily from two different angles: learning self-explainable models and post-hoc prediction analysis. The former encodes explainability in the architecture itself, for example, by generating understandable intermediate results, while the latter aims to explain the mechanism of prediction in already learned models.

An example of learning with self-explainable architecture is the neural module network, introduced in Andreas et al. (2016b) , and later extended Andreas et al. (2016a) , Johnson et al. (2017) , and Mascharka et al. (2018) . The interpretability comes from the mechanism of composing appropriate modules following the logical program produced by a natural language component. The restriction to a small set of pre-defined programs currently limits their applicability.

Input-level selection is another popular way to generate human interpretable explanations. These approaches often integrate explanation generation as part of the learning problem. Lei et al. (2016) highlighted fragments of the input text as a rationale. In contrasts, Li et al. (2016) finds the minimal set of words that need to be erased before the decision changes. Our model is inspired by (Lei et al., 2016) but extends the ideas towards combinatorial corresponding rationales across matched texts. In other words, our rationales are structured, over a pair of inputs. Sharp et al. (2017) and Choi et al. (2017) also address interpretability of QA but focus on document selections instead of reasoning about the matching problem at the finer level. Ni et al. (2018) learns how to transform a naturallanguage question into a query for an IR model by identifying essential terms. But they assume that ground-truth selections are provided.

Efforts in learning post-hoc explanations include analyzing and visualizing neural activations (Hermans & Schrauwen, 2013; Li et al., 2015; Karpathy et al., 2015; Bau et al., 2017; Selvaraju et al., 2017) , converting word embeddings into sparse or binary interpretable vectors (Faruqui et al., 2015) , and approximating the model via locally interpretable forms (Ribeiro et al., 2016; Alvarez-Melis & Jaakkola, 2017; Lundberg & Lee, 2017; Lee et al., 2018) . Recently, Chen et al. (2018) trained an instance selection model based on the class distributions from the model to be explained, which could be considered as a model-agnostic variant of (Lei et al., 2016) . The approaches here provide a posteriori explanations for previously trained models while we focus on learning explanations in the course of model training.

Attention models Attention-based models (Bahdanau et al., 2014; Xu et al., 2015; Luong et al., 2015; Rush et al., 2015) offer another means to elucidate the inner workings of neural models. These models have been successfully applied to many tasks of sequence comparison. Parikh et al. (2016) proposed word-by-word attention model for NLI and later extended by Seo et al. (2016) for QA. Similar ideas also appeared in the work of (Chen et al., 2016b; Wang et al., 2016; 2017c) . Kim et al. (2017) proposed a structured attention mechanism, which induces a latent tree structure and computes the alignment jointly. Later, based on the idea of comparing two sentences using their tree structures (Chen et al., 2016a; Zhao et al., 2017) , Liu et al. (2018) extended structured attention for matching sentences.

The methods discussed here are primarily based on soft-attention instead of hard-selection. Although such attention is easier to train and softly simulates the alignment, it serves as a proxy in understanding how portions of the input contribute to final prediction. Absent additional regularization, any small attention score can still in principle contribute significantly to later computations through re-amplification in later stages. For the purpose of understanding neural predictions we opt for hard-selections despite the more challenging training (Deng et al., 2018) .

3. RATIONALIZING TEXT MATCHING:

We formalize here the problem of learning corresponding rationales. Consider a pair of word sequences, namely q = {q 1 , • • • , q m } and p = {p 1 , • • • , p n }, where q i , p j ∈ R d denote vector representations. The learning problem is to output a task-specific prediction l based on matching q and p. For example, in NLI, the task is to classify the hypothesis-premise pair as entailment, contradiction or neutral. On the other hand, the prediction task for QA could be in the form of answer generation, classification or ranking.

Our goal is to generate human-interpretable information to gain insight into the mechanism of prediction. Specifically, we aim to select a list of pairwise text-spans as corresponding rationales, namely R = {(x k , y k )}, where x k and y k are the k th rationale pair selected from sequence q and p, respectively.

x k = {x k 1 , • • • , x k m } ∈ {0, 1} m and y k = {y k 1 , • • • , y k n } ∈ {0, 1}

n are binary indicator sequences of lengths m and n where each nonzero entry denotes a selected word. The cardinality of the set R varies with the input pair. In order for these pairwise text-spans to qualify as rationales, we ask them to satisfy the following desiderata.

• Coherent and concise: spans selected from each side of q and p need to be short and coherent compared to the original text. • Correspondent: for every pair of text spans, the selected words in x k and y k should be relevant and correspond to each other in a task-specific manner toward the end decision. • Sufficient: the selected words in the rationale collection R should serve as a substitute of the full original texts for the predictive task without degrading performance For example, in QA, each x k indicates a key factor that a particular question is asking, while y k is the corresponding supporting evidence in the passage. We call the selection of words as the rationale generator.

In this work, we assume that no rationale annotations are given. Thus, the rationale generation is learned entirely in an unsupervised manner, guided only by the distal supervision from the end task, in accordance with the above desiderata. The rationales are latent variables where selections from q and p are the combinatorial objects. The supervision from text matching guides the model to indirectly learn to choose the word combinations that provide good explanations for predictions that 

LSTM Encoder:

Galileo Galilei was an Italian physicist , mathematician , astronomer , and philosopher who played a major role in the Scientific Revolution . Galileo Galilei is credited with discovering the first four moons of Jupiter . are themselves learned concurrently. We will next describe the generator and predictor in greater detail.

3.1. RATIONALE GENERATOR:

Selecting words from q and p with desired properties is a combinatorial problem. It would be infeasible to consider all possible selections across the sequences jointly. Instead, we propose a dependent selection mechanism that follows a generation-encoding-generation scheme. Concretely, we first identify all rationales x k from sequence q. For every x k we re-encode the selected words to construct a standalone rationale vector. These vectors are then used to find corresponding rationales from p. Figure 2 illustrates the architecture of our rationale generator.

Identifying rationales from q We aim to obtain a set of rationales

{x k } from q. Let z = {z 1 , • • • , z m } ∈ {0,

1} m be the super-set of highlighted words from all x k . We generate {x k } by first generating z, followed by partitioning. The problem of generating z is cast as sequence tagging, where z ∼ p(z|q). In a simple case, we parameterize the probability distribution with a bi-directional long short-term memory (LSTM) network (Hochreiter & Schmidhuber, 1997) . Each z i is selected conditionally independently given q. Thus joint probability factors according to

EQUATION

where h q i is the i th hidden state of the LSTM taking input q, and σ(•) denotes the sigmoid function. Once z is selected, we partition the sequence by grouping successive ones to yield a list of new binary sequences {x k }, all of the same length. The newly generated sequences satisfy the following two properties: 1) each x k contains only localized group of ones without intervening zeros; and 2) z = k x k . For example, the partition outcome for the sequence "00111011" will be {00111000, 00000011}. It is worth mentioning that by construction, there is no textual overlap between different rationale selections from q, and that the number of rationales varies for different inputs.

Rationale re-encoding In order to find the corresponding rationale y k for each x k , we reencode the selected words of each rationale to a vector representation. This is done by feeding the element-wise product of x k and q to the previously used LSTM. The LSTM's hidden states

h q * x k = {h q * x k 1 , • • • , h q * x k

m } are further masked by x k followed by max pooling to obtain the rationale vector. We denote the re-encoded rationale as r k = max(h q * x k * x k ), and * is the element-wise product. One may argue whether it is necessary to re-encode, i.e., feed q * x k to the LSTM to get new hidden states so as to calculate r k . An alternative way would be to directly use the hidden states from the LSTM of q to obtain max(h q * x k ), which requires less effort. However, even though the hidden states would be masked by x k , we couldn't prevent information from other unselected words influencing r k . This is due to the nature of bi-directional LSTM, where any hidden state might potentially (and often does) encode information beyond the immediate context. A similar argument would hold if we used a convolutional encoder assuming the length of consecutive rationale words were smaller than the width of the receptive field. Thus, re-encoding step ensures that the rationale vectors only contain information from the actually selected words.

Generating corresponding rationales from p For each rationale vector r k generated from q, our goal is to find the corresponding text-spans in p. We encode the input sequence p using the same sentence encoder, where the hidden states are now

h p = {h p 1 , • • • , h p n }.

To generate rationales, we could either use a tagging model as before or a boundary model (Vinyals et al., 2015; Wang & Jiang, 2016b) which marks the start and end tokens. Here, we introduce and use the boundary model, where the words in between the two tokens are used as the rationale. The probability of selection is simply

EQUATION

We produce the start and end tokens by first generating two head vectors r k start = ReLU (Wstartr k + bstart), and r k end = ReLU (W end r k + b end ).

(3) followed by

EQUATION

where •, • denotes the inner product. We sample the end token y k end similarly, but conditioned on the selection of y k start so as to ensure that the end selection appears only at or after the location of y k start . Compared to a tagging model, the rationale selected by a boundary model is by construction consecutive. In addition, the boundary model has the flexibility of not choosing anything if p does not contain information corresponding to r k . This can be achieved by selecting both start and end tokens at the "EOS" symbol in p.

3.2. MODEL PREDICTOR:

For simplicity, let us consider classification as the guiding task to instantiate the predictor, where both NLI and answer selection fall within the setup. The framework itself generalizes to other textmatching tasks. Given the list of generated rationale pairs R referring to input texts p and q, our goal is to make these rationales alone useful for the final prediction. For each k, we first re-encode the rationale y k (as we did for x k ) and obtain LSTM hidden states h p * y k . We then construct a sparse matrix S k ∈ R m×n as follows:

EQUATION

where ⊗ is the outer product. The (i, j) entry of matrix S k is nonzero if and only if words q i and p j are both selected in the corresponded rationale pair, i.e., x k i = 0 and y k j = 0. To predict the final label, we follow a strategy reminiscent of Match-LSTM (Wang & Jiang, 2016a) . Concretely, for each S k , we first normalize it using softmax in a row-by-row fashion to construct an attention matrix A k . If q i is a rationale word in x k , then j s.t., y k j =0 A k ij = 1. Otherwise, the values in the corresponding row are all zeros. Next, we introduce aggregate representations for each q i as hq *

x k i = j A k ij * h p * y k i

. This can be interpreted as a measure of how q i can be matched by p.

By construction, for each entry i for which x k i = 0, hq * x k i is also zero. Following (Wang & Jiang, 2016a), we construct

EQUATION

where [•; •] represents vector concatenation. We repeat the aforementioned construction for every k and aggregate these representations to a unified vector h = |R| k=1 h k . The distribution over labels l is then obtained by max-pooling on h (per dimension) followed by a softmax linear prediction layer.

3.3. JOINT TRAINING:

We minimize the discrepancy between l and l via a cross-entropy loss L(p, q, R, l), which makes generated rationales corresponded and sufficient. In addition, we want the generator to realize short and coherent rationales which is encouraged by two regularizers over selections {x k } and {y k } separately. For example

EQUATION

where the first term penalizes the number of selected words, the second term makes the selections coherent and controls the total number of rationales generated from q. The regularizer is normalized by the length of the sequence. We introduce an analogous regularization for {y k }. Note that the loss function depends on the sampled rationales in R and therefore we minimize the expected cost:

EQUATION

)

where θ is the set of parameters and D is the collection of training data. Directly minimizing the expected cost is hard due to discrete rationale selections and thus we appeal to policy gradient method (Williams, 1992) to derive stochastic gradients. Due to the loss and the regularization, the generated rationales in R are encouraged to satisfy all the three desired properties. It is worth mentioning that although the Gumble reparameterization trick (Jang et al., 2016) can alleviate training difficulties in similar settings, it won't offer the same certificate of exclusion of unselected parts. For this reason (clarity), we opt for keeping the hard selections also during training.

4. EXPERIMENTS 4.1 SETTINGS:

Datasets To evaluate the proposed method, we conduct experiments on three representative datasets. Specifically, we consider SciTail (Khot et al., 2018) as the benchmark for NLI and two QA datasets that are AskUbuntu (Dos Santos et al., 2015) and SearchQA (Dunn et al., 2017) .

AskUbuntu is a non-factoid answer selection benchmark, which has been previously used for analyzing model interpretability (Lei et al., 2016) . We follow the same setting and data split using both question body and title. On the other hands, SearchQA is an open-domain QA benchmark. The questions are Jeopardy-style queries, which are mostly clued sub-clauses and phrases separated by commas. Since the question is not in the format of natural language, we directly split the question with commas and take each segment as a question rationale. In other words, we assume the question rationale is given on SearchQA and the goal is to generate corresponded passage rationales.

For simplicity, we only consider the answer selection problem in the multi-step open-domain QA pipeline (Wang et al., 2017b) , where all candidate answers are generated by an existing reading comprehension model (Wang et al., 2017a) .

Since SciTail provides data in form of both natural language and OpenIE tuple, we consider two setups: 1) we aim to generate corresponded rationales from the hypothesis-premise pair in natural language; 2) a similar setup as SearchQA that assume rationales on the hypothesis side is pre-defined base on the OpenIE results. We term these two settings as SciTail and SciTail factwise , respectively. For SciTail factwise , since the OpenIE extractions could be nested, we segment the provided tuple with the largest text coverage of the hypothesis and treat each argument as a separate rationale.

Baselines We compare our approach to the following methods:

• Independent: The model treats two sequences q and p as independent texts. The generator first generates {x k } based on equation (1). And then, for each x k , the generator generates y k regardless of the selection. After that, we use the same re-encoding procedure and the predictor. • No rationalization: This method uses the entire text from both q and p without rationale generation, which reduces to the standard Match-LSTM algorithm (Wang & Jiang, 2016a ). • No rationalization w/ re-encoding: This method is only applicable to these experiments that the rationales of one side of text are given. Similar to "No rationalization", the method uses all the words as input. However, it re-encodes the rationales from q and generates hidden representations based on each rationale assignments. Consequently, each piece of question/hypothesis rationale is encoded independently of the other pieces.

It worth mentioning that the last two baselines do not satisfy the purpose of learning corresponded rationales. However, their predictive accuracy serves as the upper bound of our proposed method.

Implementation Details Unless specified otherwise, all the models are implemented with Py-Torch (Paszke et al., 2017) . The size of the hidden states of the LSTM encoder is 200 and the batch size is set to 40 pairs of texts. All models use fixed Glove 100-dimension word embeddings (Pennington et al., 2014) , except for the AskUbuntu where we used the same word embeddings as (Lei et al., 2016) . The maximum number of rationales we considered is three. For the case that the generator identifies more than three rationales on q, we extract the first three and discard the rest. We use Adam optimizer (Kingma & Ba, 2014) to train our model. We choose to use the tagging based model to generate rationales on p except for the SciTail dataset. Other hyperparameters including the learning rate are tuned and selected according to the development set. We report the testing set results corresponding to the best development set settings. In addition, since both development and test sets of AskUbuntu are small (∼200 questions), we report the averaged results over five runs.

Due to the training difficult of policy gradient, we also study an additional regularizer, which aims to make the rationale generator benefit from the soft-attention scores between words in q and p. Specifically, we calculate a word-by-word similarity matrix using LSTM contexts and then normalized it in a row-by-row fashion. We denote the newly constructed attention matrix A. Then for each rationale x k , we select the row of A that x k i = 1. We rank all entries and pick the words in p with top K scores to form a new binary sequence ŷk with length n. K is set to be i x k i . We found that although ŷk is insufficient to serve as rationales, it provides indirect supervision to help the model training. Λ({y k }) = λ 3 1 m |R| k=1 |y k -ŷk | used as the additional regularizer for these experiments that the rationales on q are provided. According to the development results, this regularizer helps SciTail factwise since it introduces contrastive information among words in p. But it fails to improve SearchQA. Thus, all the reported results on SciTail factwise include the additional regularizer.

4.2. MAIN RESULTS:

We demonstrate that the proposed framework is able to match spans from on side of the input to corresponding segments by first reporting the performances of SearchQA and SciTail factwise . We report the accuracy of both datasets. In addition, we also calculate the mean average precision (MAP) for SearchQA. We report performances at different highlight percentages on the alignment matrix, which is defined as R k=1 x k ⊗ y k ∈ R m×n . With a proper re-encoding process, this quantity characterizes how much information is utilized for matching two sequences based on the corresponded information only. The sparsity control is achieved by constraining the percentage of selected words2 in p for each rationale generated in q.

Table 1 shows that our proposed method manages to select a small fragment of the alignment matrix while maintaining the accuracy. Furthermore, when selecting the same level of corresponded words, our dependent selection performs consistently better than the independent one. Surprisingly, our approach even outperforms the model without rationale generation on SearchQA when 40% of the alignment matrix is selected. This is might due to the fact that the passages are often redundant. Some examples of the generated rationales by the proposed model are shown in Figure 3 . Given rationales in q, rationales in p are selected as consecutive spans that with similar meanings. Such corresponded matches provide an interpretable way about how the model makes the prediction.

The next, we test the performance of our pipeline on AskUbuntu and SciTail and report results in Table 2 . Since the average length of the question and passage for AskUbuntu is much larger, we are able to select much fewer words to predict the answer. Specifically, our dependent generation manages to use only 8% of the matchings to achieve 50.9 MAP. On the other hands, we select 30% for the shorter sentences in SciTail. The proposed rationalization approach outperforms the independent baseline, however, there is still a small gap compared to the Match-LSTM algorithm without any rationalization. In addition, for the SciTail dataset, we also investigate the choice of the maximum number of rationale we generate from q. This is denoted as the |R| in the table.

We observe that by considering a larger number of rationale pairs, the overall performance gets improved. This because the larger number makes the sampling process of z more robust.

4.3. FURTHER ANALYSES:

To analyze the quality of generated rationale in a quantitative way, we design an experiment that adds non-relevant sentences to the premise using the SciTail factwise dataset. Specifically, for each (q, p) pair, we select a non-relevant premise sentence p . We make sure that the corresponding hypothesis of p is not same to q. We construct a new premise by concatenating two sequences [p; p ], and then feed the new pair (q, [p; p ]) to a previously trained model for inference. Please note that the testing model does not see any adversary inputs with additive noise during its training. We hope the selected spans on the premise side should not cover any text in p . To quantify the measure, for any rationale x k in q, we check if the corresponding rationale y k cover any words in the span of p .

We conduct the aforementioned experiment on our pre-trained model. We first consider (q, [p; p ]) pairs that q, p is a "entail" relation. We observe that among these inputs, our dependent rationalization model is fairly robust. Consider all selected words in {y k }, there are 13.1% of words come from the irrelevant piece p . Furthermore, in total, 13.0 % of the words in p are accidentally selected for at least once. These two numbers serve as a similar flavor of the "precision" and "recall" measures. Moreover, for these pairs that have "contradict" relations, we observe performance drops. Particularly, 16.7% of our selections are from p and 17.5% of the words in p are selected as rationales for at least once. The drop of performance might due to the difficulty of policy gradient optimization, which the selection model is hard to learn the option of not choosing any words for those rationales facts that are not covered by p. We will investigate the problem in our future work. Qualitative results of the above experiments can be found in Appendix A.

5. CONCLUSION:

In this paper, we propose a novel self-explaining architecture to predict matches between two sequences of texts. Specifically, we introduce the notion of corresponded rationales and learn to extract them by the distal supervision from the downstream task. To evaluate, we compare our method to state-of-the-art on both NLI and QA. We show quantitatively and qualitatively that our algorithm delivers human interpretable justification while preserving high predictive performances.

:

The multi-step QA pipeline that follows the process of search, read and re-rank. In brief, an information retrieval model first coarsely selects relevant passages to a given question. Then, a reading comprehension model infers a list of candidate answers from the passages, followed by a re-ranking algorithm that reorders the answer candidates based on aggregated evidence from different passages towards the final prediction.

One way is to replace the 1 norm in equation (7) to a hinge-loss that controls the percentage of selection to be less than a fixed level.

Figures and Tables:

Figure2: The architecture of the rationale generator. The model is comprised of (a) a module for identifying rationales on q, (b) a rationale re-encoding module, and (c) a component for generating corresponding rationales on p. The sentence encoder is shared across all modules.

Experiment results on SearchQA and SciTail factwise .

Table content: <table><tr><td/><td/><td>SearchQA</td><td/><td colspan="2">SciTail factwise</td></tr><tr><td/><td colspan="5">% Alignment Test MAP Test Acc % Alignment Test Acc</td></tr><tr><td>Independent</td><td>∼ 10%</td><td>58.3</td><td>49.6</td><td>∼ 20%</td><td>71.9</td></tr><tr><td/><td>∼ 40%</td><td>59.4</td><td>51.5</td><td>∼ 40%</td><td>75.1</td></tr><tr><td>Ours</td><td>∼ 10%</td><td>59.6</td><td>51.8</td><td>∼ 20%</td><td>74.2</td></tr><tr><td/><td>∼ 40%</td><td>60.2</td><td>52.9</td><td>∼ 40%</td><td>76.9</td></tr><tr><td>No Rationalization w/ re-encoding</td><td>100%</td><td>59.9</td><td>52.3</td><td>100%</td><td>78.0</td></tr></table>

Experiment results on AskUbuntu and SciTail. Two light nuclei combine to produce a heavier nucleus and great energy in the nuclear fusion process Premise: Nuclear fusion is when two or more light nuclei join together to form a heavier nucleus , releasing energy in the process Hypothesis: protein is an important part of a healthy diet because it is needed to repair tissue Premise: protein is needed in the diet for muscle and tissue growth and repair Figure3: Examples of the corresponded rationales extracted from the SciTail factwise dataset. Rationales are highlighted in different color. Text spans from the hypothesis and premise with the same color indicate a corresponded match. If a word in the premise is selected by two rationale pairs (e.g. red and yellow color), we change the text color (to red) and highlight (to yellow) accordingly.

Table content: <table><tr><td/><td/><td>AskUbuntu</td><td/><td/><td>SciTail</td><td/></tr><tr><td/><td colspan="6">% Alignment Test MAP Test Acc |R| % Alignment Test Acc</td></tr><tr><td>Independent</td><td>∼ 8%</td><td>50.3</td><td>49.4</td><td>5</td><td>∼ 30%</td><td>74.9</td></tr><tr><td>Ours</td><td>∼ 2%</td><td>50.9</td><td>48.5</td><td>3</td><td>∼ 30%</td><td>76.0</td></tr><tr><td/><td>∼ 8%</td><td>51.5</td><td>50.0</td><td>5</td><td>∼ 30%</td><td>79.4</td></tr><tr><td>No Rationalization</td><td>100%</td><td>53.5</td><td>51.7</td><td>-</td><td>100%</td><td>82.2</td></tr></table>

Peer review (iclr2019):
 This paper proposes an approach to introduce interpretability in NLP tasks involving text matching. However, the evaluation is not evaluated using human input, thus it is not clear whether the model is indeed meeting this important goal. Furthermore, there is no direct comparison against related work on the same topic, so it is not possible to assess the contributions over the state of the art on the topic. In more detail: - There are versions of attention mechanisms that are spare and differentiable. See here: From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification André F. T. Martins, Ramón Fernandez Astudillo - Why is "rationalizing textual matching" different than other approaches to explaining the predictions of a model? As far as I can tell, thresholding on existing attention would give the same output. I am not arguing that there is nothing different, but there should be a direct comparison, especially since eventually the method proposed is thresholded as well by limiting the number of highlights. - A key assumption in the paper is that the method identifies rationales that humans would find useful as explanations. However there is no evaluation of this assumption. For a recent example of how such human evaluation could be done see: D. Nguyen. Comparing automatic and human evaluation of local explanations for text classification. NAACL 2018 http://www.dongnguyen.nl/publications/nguyen-naacl2018.pdf - I don*t agree that explanations are sufficient if removing them doesn*t degrade performance. While these two are related concepts, the quality of the explanation to a human is different to a system. In fact, more text can degrade performance when it is unrelated. See the experiments of this paper: Adversarial Examples for Evaluating Reading Comprehension Systems. Robin Jia and Percy Liang. EMNLP 2017: http://stanford.edu/~robinjia/pdf/emnlp2017-adversarial.pdf - Reducing the selection of rationales to sequence tagging eventually done as classification is suboptimal compared to work on submodular optimization (cited in the introduction) if being concise is important. A comparison is needed. - There is an argument that the training objective makes generated rationales corresponded and sufficient. This requires some evidence to support it. - What is the "certificate of exclusion of unselected parts" that the proposed method has? - An important argument is that the performance does not degrade. However there is no comparison against state of the art models to verfiy it.