Learning Corresponded Rationales for Text Matching

1 INTRODUCTION . Text matching serves as a key subroutine facilitating many natural language processing ( NLP ) tasks including natural language inference ( NLI ) ( Bowman et al. , 2015 ; Wang & Jiang , 2016a ; Khot et al. , 2018 ) , paraphrase detection ( Wang et al. , 2017c ) , question answering ( QA ) ( Andreas et al. , 2016a ; Wang et al. , 2017b ; Chen et al. , 2017 ) , and others . Much of the progress across such tasks has come from the use of flexible neural architectures that can be trained to achieve high performance . However , absent attention or rationale mechanisms , the resulting models lack transparency about how the decisions are reached . Soft attention keeps the model differentiable , typically operates on a word-by-word basis ( Parikh et al. , 2016 ; Seo et al. , 2016 ; Chen et al. , 2016b ; Wang et al. , 2016 ; Wang & Jiang , 2016b ) , but does not strictly confer importance as weakly attended words can impact decisions . In contrast , rationales ( Lei et al. , 2016 ) as hard selections are challenging to train but provide a certificate of exclusion of any unattended part . We opt for rationales in this paper as our goal is to identify clearly necessary text spans as opposed to individual words , and to do so across the matched texts in a dependent way . Our work builds on and relates to many recent advances in explaining neural predictions for NLP ( Lei et al. , 2016 ; Li et al. , 2016 ; Sharp et al. , 2017 ; Koh & Liang , 2017 ; Alvarez-Melis & Jaakkola , 2017 ; 2018 ) but aims for rationalizing textual matching . Consider answer reranking or selection in the state-of-the-art open domain QA pipeline1 ( Wang et al. , 2017b ) . The end goal in this case is to classify whether the supporting passage indeed has the answer to the specific question . Figure 1 illustrates the problem with two answer candidates and their supporting passages . A rationale that explains the correct match should not only highlight “ Galileo Galilei ” in the passage but also include two additional facts from the question and their corresponding evidence in the passage ( highlighted in red and blue ) . In this paper , we seek rationales that identify necessary textual spans from each question and , in a dependent way , their short and sufficient corresponding segments in the supporting 1The multi-step QA pipeline that follows the process of search , read and re-rank . In brief , an information retrieval model first coarsely selects relevant passages to a given question . Then , a reading comprehension model infers a list of candidate answers from the passages , followed by a re-ranking algorithm that reorders the answer candidates based on aggregated evidence from different passages towards the final prediction . passages . Only the selected text spans in aggregate – the rationales – are used as the basis for the final decision about matching . We do not assume that any rationale annotations are given . Instead , the rationales are learned as latent selections based on the distal supervision ( Mintz et al. , 2009 ) from the downstream text matching task . The rationale mechanism builds on the generator-predictor framework Lei et al . ( 2016 ) introduced for sentiment analysis where the generator selects text spans as rationales that are then fed to the predictor . In our case , the generator is tasked with a combinatorial selection across the texts being matched . To make the combinatorial selection easier to learn we first identify key pieces from a single side of the input ( e.g . question or hypothesis ) and then look for the corresponding spans from the other side in a dependent manner . The predictor finally maps the extracted rationales in aggregate to task-specific values . The entire model along with the rationales is trained in an end-to-end fashion through policy gradient ( Williams , 1992 ) . We consider both NLI and QA as the evaluation tasks . Extensive experimental results on three publicly available datasets show that the model delivers human-interpretable information while preserving comparable performance to the state-of-the-art . Further , to quantify the quality of generated rationales , we contrast the rationales arising from clean vs adversarial inputs . The generated rationales ( as they operated over spans ) are fairly robust against inserting irrelevant texts . 2 RELATED WORK . Learning interpretable models Improving interpretability has been approached primarily from two different angles : learning self-explainable models and post-hoc prediction analysis . The former encodes explainability in the architecture itself , for example , by generating understandable intermediate results , while the latter aims to explain the mechanism of prediction in already learned models . An example of learning with self-explainable architecture is the neural module network , introduced in Andreas et al . ( 2016b ) , and later extended Andreas et al . ( 2016a ) , Johnson et al . ( 2017 ) , and Mascharka et al . ( 2018 ) . The interpretability comes from the mechanism of composing appropriate modules following the logical program produced by a natural language component . The restriction to a small set of pre-defined programs currently limits their applicability . Input-level selection is another popular way to generate human interpretable explanations . These approaches often integrate explanation generation as part of the learning problem . Lei et al . ( 2016 ) highlighted fragments of the input text as a rationale . In contrasts , Li et al . ( 2016 ) finds the minimal set of words that need to be erased before the decision changes . Our model is inspired by ( Lei et al. , 2016 ) but extends the ideas towards combinatorial corresponding rationales across matched texts . In other words , our rationales are structured , over a pair of inputs . Sharp et al . ( 2017 ) and Choi et al . ( 2017 ) also address interpretability of QA but focus on document selections instead of reasoning about the matching problem at the finer level . Ni et al . ( 2018 ) learns how to transform a naturallanguage question into a query for an IR model by identifying essential terms . But they assume that ground-truth selections are provided . Efforts in learning post-hoc explanations include analyzing and visualizing neural activations ( Hermans & Schrauwen , 2013 ; Li et al. , 2015 ; Karpathy et al. , 2015 ; Bau et al. , 2017 ; Selvaraju et al. , 2017 ) , converting word embeddings into sparse or binary interpretable vectors ( Faruqui et al. , 2015 ) , and approximating the model via locally interpretable forms ( Ribeiro et al. , 2016 ; Alvarez-Melis & Jaakkola , 2017 ; Lundberg & Lee , 2017 ; Lee et al. , 2018 ) . Recently , Chen et al . ( 2018 ) trained an instance selection model based on the class distributions from the model to be explained , which could be considered as a model-agnostic variant of ( Lei et al. , 2016 ) . The approaches here provide a posteriori explanations for previously trained models while we focus on learning explanations in the course of model training . Attention models Attention-based models ( Bahdanau et al. , 2014 ; Xu et al. , 2015 ; Luong et al. , 2015 ; Rush et al. , 2015 ) offer another means to elucidate the inner workings of neural models . These models have been successfully applied to many tasks of sequence comparison . Parikh et al . ( 2016 ) proposed word-by-word attention model for NLI and later extended by Seo et al . ( 2016 ) for QA . Similar ideas also appeared in the work of ( Chen et al. , 2016b ; Wang et al. , 2016 ; 2017c ) . Kim et al . ( 2017 ) proposed a structured attention mechanism , which induces a latent tree structure and computes the alignment jointly . Later , based on the idea of comparing two sentences using their tree structures ( Chen et al. , 2016a ; Zhao et al. , 2017 ) , Liu et al . ( 2018 ) extended structured attention for matching sentences . The methods discussed here are primarily based on soft-attention instead of hard-selection . Although such attention is easier to train and softly simulates the alignment , it serves as a proxy in understanding how portions of the input contribute to final prediction . Absent additional regularization , any small attention score can still in principle contribute significantly to later computations through re-amplification in later stages . For the purpose of understanding neural predictions we opt for hard-selections despite the more challenging training ( Deng et al. , 2018 ) . 3 RATIONALIZING TEXT MATCHING . We formalize here the problem of learning corresponding rationales . Consider a pair of word sequences , namely q = { q1 , · · · , qm } and p = { p1 , · · · , pn } , where qi , pj ∈ Rd denote vector representations . The learning problem is to output a task-specific prediction l based on matching q and p. For example , in NLI , the task is to classify the hypothesis-premise pair as entailment , contradiction or neutral . On the other hand , the prediction task for QA could be in the form of answer generation , classification or ranking . Our goal is to generate human-interpretable information to gain insight into the mechanism of prediction . Specifically , we aim to select a list of pairwise text-spans as corresponding rationales , namelyR = { ( xk , yk ) } , where xk and yk are the kth rationale pair selected from sequence q and p , respectively . xk = { xk1 , · · · , xkm } ∈ { 0 , 1 } m and yk = { yk1 , · · · , ykn } ∈ { 0 , 1 } n are binary indicator sequences of lengths m and n where each nonzero entry denotes a selected word . The cardinality of the setR varies with the input pair . In order for these pairwise text-spans to qualify as rationales , we ask them to satisfy the following desiderata . • Coherent and concise : spans selected from each side of q and p need to be short and coherent compared to the original text . • Correspondent : for every pair of text spans , the selected words in xk and yk should be relevant and correspond to each other in a task-specific manner toward the end decision . • Sufficient : the selected words in the rationale collectionR should serve as a substitute of the full original texts for the predictive task without degrading performance For example , in QA , each xk indicates a key factor that a particular question is asking , while yk is the corresponding supporting evidence in the passage . We call the selection of words as the rationale generator . In this work , we assume that no rationale annotations are given . Thus , the rationale generation is learned entirely in an unsupervised manner , guided only by the distal supervision from the end task , in accordance with the above desiderata . The rationales are latent variables where selections from q and p are the combinatorial objects . The supervision from text matching guides the model to indirectly learn to choose the word combinations that provide good explanations for predictions that are themselves learned concurrently . We will next describe the generator and predictor in greater detail .

Summary: This paper proposes an approach to introduce interpretability in NLP tasks involving text matching. However, the evaluation is not evaluated using human input, thus it is not clear whether the model is indeed meeting this important goal. Furthermore, there is no direct comparison against related work on the same topic, so it is not possible to assess the contributions over the state of the art on the topic. In more detail: