NeurQuRI: Neural Question Requirement Inspector for Answerability Prediction in Machine Reading Comprehension

1 INTRODUCTION . Machine reading comprehension ( MRC ) , where a machine understands a given document and answers a question , is a challenging task , but it has a significant impact in real-world applications such as dialog systems . In practice , given a user-initiated question , potentially relevant paragraphs ( often called contexts ) are first retrieved from a search engine , which may or may not contain an actual answer . In this case , it is important for an MRC model ( or in short , a reader ) to be able to determine whether the retrieved context contains the answer before actually predicting the answer . In most previous MRC tasks and datasets , such an answerability issue was out of scope as the provided context was guaranteed to contain an answer for a given question . Recently , a new dataset called SQuAD 2.0 ( Rajpurkar et al. , 2018 ) was released , containing instances with unanswerable questions for a given context , so that models can be properly trained to classify this case . Additionally , this dataset also contains information about plausible answers in the context when the question is unanswerable , which can be used to prevent our model from wrongly predicting it as an answer . Previously , Liu et al . ( 2018 ) addressed the problem of classifying unanswerable cases by adding an auxiliary no-answer classifier to the last layer of the MRC model . Clark & Gardner ( 2018 ) tackled answerability classification through a joint softmax layer of the answerability score as well as the scores of all possible answer spans . Hu et al . ( 2019 ) attempted to verify the question against the sentence ( s ) containing the candidate answer . The answerability score from the verifier and the score from the reader were combined to generate the final score of having no answer . However , these existing approaches do not pinpoint where the mismatch occurs between the question and the candidate answer in the unanswerable case , thus being prone to choosing a plausible but wrong answer . This task often becomes tricky when particular conditions from the question are not met . For example , in a question “ What was the projection of sea level increases in the fourth assessment report ? ” , the answer should properly satisfy several conditions , such as “ increases ” ( but not decreases ) and “ fourth ” ( but not third ) . Motivated by this , we propose a novel neural inspector model that forms a list of conditions from the question , each of which should be satisfied by the candidate answer generated by the reader . To check whether each condition is met , we leverage and extend the idea proposed by Kiddon et al . ( 2016 ) , which introduced a recurrent unit that records the used ingredients of cooking recipes by accumulating an attention mechanism during the generation of the recipe in a natural language text . They encourage the model to use all the ingredients by the end of the recipe text generation . Extending this idea , we present a novel condition-checking module that determines whether the candidate answer satisfies all the conditions from the question . Furthermore , we propose a novel regularization method that can properly train our condition-checking model , leading to a correct candidate answer . Finally , we evaluate our proposed model on SQuAD 2.0 , NewsQA ( Trischler et al. , 2017 ) and MS MARCO ( Bajaj et al. , 2016 ) datasets . Our experimental results show consistent improvements across a wide range of MRC models and also demonstrate the explainability of our model regarding which conditions of a given question are not met , or a reason why our model classified the question as unanswerable in a given context . 2 PROPOSED METHOD . This section discusses the details of our proposed method called Neural Question Requirement Inspector ( NeurQuRI ) . As shown in Fig . 1- ( A ) , NeurQuRI calculates answerability by taking a candidate answer and the question as input . To create the candidate answer , the reader takes a context ( or paragraphs ) and a question as input ( Fig . 1- 1 ) , chooses the most probable candidate answer span ( Fig . 1- 2 , 3 ) and gives its contextualized word-level representation as input to NeurQuRI ( Fig . 1- 4 ) . NeurQuRI then determines answerability by checking whether all conditions from the question are met by the candidate answer ( Fig . 1- 5 , 6 ) . Intuitively , a wrong candidate answer will not satisfy at least one condition given by the question . To develop this idea , we propose a novel architecture for the inspector encoder as well as a condition satisfaction loss to properly train it . 2.1 NEURAL QUESTION REQUIREMENT INSPECTOR ( NEURQURI ) . In NeurQuRI , inspired by the idea of using an ingredient word as a condition in the checklist ( Kiddon et al. , 2016 ) , each question word works as a condition to be satisfied by a candidate answer . Additionally , we use the question itself as the pseudo-answer that trivially contains all the words in the question and thus exemplifies an indication of full satisfaction during training . NeurQuRI creates and compares an inspection vector of the candidate answer with that of the question to check whether all the word-level meanings in the question are covered in the candidate answer . NeurQuRI is largely composed of two parts : an inspection encoder and an inspection comparator . The inspection encoder encodes an input embedding sequence ( seq ) , either a candidate answer or a question , into an inspection vector hseq→q by using the question ’ s condition satisfaction with the input embedding sequence ( seq ) . Using candidate answer ( x ) and question ( q ) as input sequence ( seq ) , we calculate two inspection vectors , hx→q and hq→q ( Fig . 1- 5 ) . Next , the inspection comparator compares these two vectors and finally computes an answerability score ( Fig . 1- 6 ) . Inspection Encoder . Let us denote a sequence of contextual embeddings of question words as Q = { qi } mi=1 ∈ Rm×dq and that of candidate answer words as X = { xt } kt=1 ∈ Rk×dx , where dq and dx are the input dimensions of question and candidate answer words , respectively , and m and k are their respective sequence lengths . GivenQ andX , the inspection encoder generates ( 1 ) an inspection vector h , which encodes information on the conditions given by Q that are satisfied by X , and ( 2 ) the cumulative satisfaction score vector a ∈ Rm , each element of which indicates how much the condition corresponding to the question word is satisfied , as a value between zero and one . In detail , we extend a gated recurrent unit ( GRU ) ( Cho et al. , 2014 ) , which sequentially takes a candidate answer word xt at time t and generates an inspection vector ht and the cumulative satisfaction score vector at , i.e. , at , ht = InspectionEnc ( xt , at−1 , ht−1 , Q ) , where vectors a and h are initialized with zeros . First , we linearly transform Q and X as Q̃ = { q̃i } mi=1 ∈ Rm×d and X̃ = { x̃t } kt=1 ∈ Rk×d to have the target dimension d in common . We then calculate Q̃newt = { q̃newt , i } mi=1 , which represents unmet conditions by X from Q until time t , i.e. , Q̃newt = ( ( 1m − at−1 ) ⊗ 1d ) ◦ Q̃ ∈ Rm×d , where ◦ indicates element-wise multiplication and ⊗1d stacks the source d times ( Fig . 2- A ) . Second , we multiply the t-th word vector of a candidate answer , x̃t , with a gating vector gt so that we can selectively ignore part of information contained in x̃t , yielding the vector x̃ g t as gt = σ ( Wgx̃t + Ught−1 ) ∈ Rd x̃gt = gt ◦ x̃t ∈ Rd , where Wg ∈ Rd×d and Ug ∈ Rd×d are linear transformation matrices and σ represents a sigmoid function ( Fig . 2- B ) . Ignoring such partial information prevents our model from considering the redundant information in the previous candidate answer words . Third , we compute the satisfaction score at time step t , αt , by using x̃ g t as a query vector and each question word ( or condition ) vector in Q̃newt , i.e. , ( αt ) i = σ ( fα ( [ x̃ g t ; q̃ new t , i ; x̃ g t − q̃newt , i ; x̃ g t ◦ q̃newt , i ] ) ) , where fα is a fully connected unit with two hidden layers of dimensions d/2 and 1 , respectively ( Fig . 2- C ) . We used independent sigmoid outputs here instead of integrated softmax outputs to allow multiple high satisfaction scores for all met conditions . For example , word ‘ Lincoln ’ in the candidate answer can simultaneously satisfy the words ‘ Who ’ and ‘ President ’ in the question . We obtain the vector et as the information of the satisfied condition ( Fig . 2- D ) by max-pooling over Q̃newt multiplied with the satisfaction score α , i.e. , et = max-pooling ( { αiq̃newt , i } mi=1 ) ∈ Rd . Finally , we use et as an additional feature to compute the update vector in GRU , h̃t ( Fig . 2- E ) and obtain the inspection vector ht as zt = σ ( Wzx̃t + Uzht−1 ) ∈ Rd rt = σ ( Wrx̃t + Urht−1 ) ∈ Rd h̃t = tanh ( Whx̃t + Uh ( rt ◦ ht−1 ) + Vhet ) ∈ Rd ht = ( 1− zt ) ◦ ht−1 + zt ◦ h̃t ∈ Rd , where zt and rt are an update and a reset gate , respectively , and Wz , Uz , Wr , Ur , Wh , Uh , Vh ∈ Rd×d . We also update the cumulative satisfaction score vector at ( Fig . 2- F ) , with the current satisfaction score vector α as at = min ( at−1 + α,1 m ) ∈ Rm , where 1m is an m dimensional ones vector so that each element of at is clipped between 0 and 1. at represents how much its corresponding question word as a condition is satisfied by the candidate answer . Inspection Comparator . The inspection comparator compares the inspection vector of the candidate answer with the inspection vector of the question itself to check whether all the word-level meanings in the question are involved in the candidate answer . We use the question itself as the pseudo-answer that trivially contains all the words in the question . To be specific , we compare the ‘ candidate-answer-to-question ’ inspection vector hx→q with the ‘ question-to-question ’ inspection vector hq→q , where the latter can be considered as a fully satisfied reference to the question . We compare these vectors using a fully-connected layer to generate an answerability score . First , we calculate the inspection vector ht and the question ’ s satisfaction score at with respect to the question representation Q , i.e. , ax→qt , h x→q t = InspectionEnc ( xt , a x→q t−1 , h x→q t−1 , Q ) aq→qt , h q→q t = InspectionEnc ( qt , a q→q t−1 , h q→q t−1 , Q ) . Afterwards , we compute the answerability score zi by combining them , i.e. , zi = fβ ( [ h x→q Tx ; hq→qTq ; h x→q Tx − hq→qTq ; h x→q Tx ◦ hq→qTq ) , where Tx and Tq indicate the last time steps of the two sequences , respectively , and fβ is a fully connected unit with two hidden layers that have have dimensions as d/2 and 1 , respectively .

Summary: This paper incorporates an answer inspection encoder verifying whether the answers selected by the Machine Reading Comprehension (MRC) component is valid, into the MRC reader. For training, the model includes two additional losses and trained jointly. Evaluation is on SQuAD V2.0 which is constructed from Wikipedia and contains unanswerable questions generated by crowd sources. The approaches are verified in the settings with/without including BERT to show the generalization of the answer inspector.