Title: SDNET: CONTEXTUALIZED ATTENTION-BASED DEEP NETWORK FOR CONVERSATIONAL QUESTION AN-SWERING

Abstract: Conversational question answering (CQA) is a novel QA task that requires the understanding of dialogue context. Different from traditional single-turn machine reading comprehension (MRC), CQA is a comprehensive task comprised of passage reading, coreference resolution, and contextual understanding. In this paper, we propose an innovative contextualized attention-based deep neural network, SDNet, to fuse context into traditional MRC models. Our model leverages both inter-attention and self-attention to comprehend the conversation and passage. Furthermore, we demonstrate a novel method to integrate the BERT contextual model as a sub-module in our network. Empirical results show the effectiveness of SDNet. On the CoQA leaderboard, it outperforms the previous best model's F 1 score by 1.6%. Our ensemble model further improves the F 1 score by 2.7%.

1. INTRODUCTION:

Machine reading comprehension (MRC) is a core NLP task in which a machine reads a passage and then answers related questions. It requires a deep understanding of both the article and the question, as well as the ability to reason about the passage and make inferences. These capabilities are essential in applications like search engines and conversational agents. In recent years, there have been numerous studies in this field (Huang et al., 2017; Seo et al., 2016; Chen et al., 2017; Liu et al., 2017) , with various innovations in text encoding, attention mechanisms and answer verification.

However, traditional MRC tasks often take the form of single-turn question answering. In other words, there is no connection between different questions and answers to the same passage. This oversimplifies the conversational manner humans naturally take when probing a passage, where question turns are assumed to be remembered as context to subsequent queries. Figure 1 demonstrates an example of conversational question answering in which one needs to correctly refer "she" in the last two rounds of questions to its antecedent in the first question, "Cotton." To accomplish this kind of task, the machine must comprehend both the current round's question and previous rounds of utterances in order to perform coreference resolution, pragmatic reasoning and semantic implication.

To facilitate research in conversation question answering (CQA), several public datasets have been published that evaluate a model's efficacy in this field, such as CoQA (Reddy et al., 2018) , QuAC (Choi et al., 2018) and QBLink (Elgohary et al., 2018) . In these datasets, to generate correct responses, models need to fully understand the given passage as well as the dialogue context. Thus, traditional MRC models are not suitable to be directly applied to this scenario.

Therefore, a number of models have been proposed to tackle the conversational QA task. DrQA+PGNet (Reddy et al., 2018) combines evidence finding and answer generation to produce answers. BiDAF++ (Yatskar, 2018) achieves better results by employing answer marking and contextualized word embeddings on the MRC model BiDAF (Seo et al., 2016) . FlowQA (Huang et al., 2018) leverages a recurrent neural network over previous rounds of questions and answers to absorb information from its history context.

Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton. Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept. But Cotton wasn't alone in her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sisters...

Q1::

What color was Cotton? A1: white Q2: Where did she live? A2: in a barn Q3: Did she live alone? A3: no Figure 1 : Example passage and first three rounds of question and answers from CoQA dataset (Reddy et al., 2018) . Pronouns requiring coreference resolution is marked in bold.

In this paper, we propose SDNet, a contextual attention-based deep neural network for the conversational question answering task. Our network stems from machine reading comprehension models, but it has several unique characteristics to tackle context understanding. First, we apply both interattention and self-attention on the passage and question to obtain a more effective understanding of the passage and dialogue history. Second, we prepend previous rounds of questions and answers to the current question to incorporate contextual information. Third, SDNet leverages the latest breakthrough in NLP: BERT contextual embeddings (Devlin et al., 2018) .

Different from the canonical way of employing BERT as a monolithic structure with a thin linear task-specific layer, we utilize BERT as a contextualized embedder and absorb its structure into our network. To accomplish this, we align the traditional tokenizer with the Byte Pair Encoding (BPE) tokenizer in BERT. Furthermore, instead of using only the last layer's output from BERT (Devlin et al., 2018) , we employ a weighted sum of BERT layer outputs to take advantage of all levels of semantic abstraction. Finally, we lock the internal parameters of BERT during training, which saves considerable computational cost. These techniques are also applicable to other NLP tasks.

We evaluate SDNet on the CoQA dataset, and it improves on the previous state-of-the-art F 1 score by 1.6% (from 75.0% to 76.6%). The ensemble model further increases the F 1 score to 79.3%.

2. APPROACH:

In this section, we propose our neural model, SDNet, for the conversational question answering task. We first formulate the problem and then present an overview of the model before delving into the details of the model structure.

2.1. PROBLEM FORMULATION:

Given a passage/context C, and question-answer pairs from previous rounds of conversation

Q 1 , A 1 , Q 2 , A 2 , ..., Q k-1 , A k-1 , the task is to generate response A k given the latest question Q k .

The response is dependent on both the passage and historic questions and answers.

To incorporate conversation history into response generation, we employ the idea from DrQA+PGNet (Reddy et al., 2018) to prepend the latest N rounds of QAs to the current question Q k . The problem is then converted into a single-turn machine reading comprehension task, where the reformulated question is

Q k = {Q k-N ; A k-N ; ..., Q k-1 ; A k-1 ; Q k } 1 .

2.2. MODEL OVERVIEW:

Encoding layer encodes each token in passage and question into a fixed-length vector, which includes both word embeddings and contextualized embeddings. For contextualized embedding, we utilize the pretrained language understanding model BERT (Devlin et al., 2018) . Different from previous work, we fix the parameters in BERT model and use the linear combination of embeddings from different layers in BERT.

Integration layer uses multi-layer recurrent neural networks (RNN) to capture contextual information within passage and question. To characterize the relationship between passage and question, we conduct word-level attention from question to passage both before and after the RNNs. We employ the idea of history-of-word from FusionNet (Huang et al., 2017) to reduce the dimension of output hidden vectors. Furthermore, we conduct self-attention to extract relationship between words at different positions of context and question.

Output layer computes the final answer span. It uses attention to condense the question into a fixedlength vector, which is then used in a bilinear projection to obtain the probability that the answer should start and end at each position.

An illustration of our model SDNet is in Figure 2 .

2.3. ENCODING LAYER:

We first use GloVe (Pennington et al., 2014) embedding for each word in the context and question. Additionally, we compute a feature vector f w for each context word, following the approach in DrQA (Chen et al., 2017) . This feature vector contains a 12-dim POS embedding, an 8-dim NER embedding, a 3-dim exact matching vector em i indicating whether this word, its lower-case form or its stem appears in the question, and a 1-dim normalized term frequency.

BERT as Contextual Embedder. We design a number of methods to leverage BERT (Devlin et al., 2018) as a contextualized embedder in our model.

First, because BERT uses Byte Pair Encoding (BPE) (Sennrich et al., 2015) as the tokenizer, the generated tokens are sub-words and may not align with traditional tokenizer results. To incorporate BERT into our network, we first use a conventional tokenizer (e.g. spaCy) to get word sequences, and then apply the BPE tokenizer from BERT to partition each word w in the sequence into subwords w = (b 1 , ..., b s ). This alignment makes it possible to concurrently use BERT embeddings and other word-level features. The contextual embedding of w is defined to be the averaged BERT embedding of all sub-words b j , 1 ≤ j ≤ s.

Second, Devlin et al. (2018) proposes the method to append thin task-specific linear layers to BERT, which takes the result from the last transformer layer as input. However, as BERT contains multiple layers, we employ a weighted sum of these layer outputs to take advantage of information from all levels of semantic abstraction. This can help boost the performance compared with using only the last transformer's output.

Third, as BERT contains hundreds of millions of parameters, it takes a lot of time and space to compute and store their gradients during optimization. To tackle this problem, we lock the internal weights of BERT during training, only updating the linear combination weights. This can significantly increase the efficiency during training, which can be especially useful when computing resource is limited.

To summarize, suppose a word w is tokenized to s BPE tokens w = (b 1 , b 2 , ..., b s ), and BERT has L layers that generate L embedding outputs for each BPE token, h l t , 1 ≤ l ≤ L, 1 ≤ t ≤ s. The contextual embedding BERT w for word w is computed as:

EQUATION

where α 1 , ..., α L are trainable parameters.

2.4. INTEGRATION LAYER:

Word-level Inter-Attention. We conduct attention from question to context (passage) based on GloVe word embeddings. Suppose the context word embeddings are {h C 1 , ..., h C m } ⊂ R d , and the question word embeddings are {h

Q 1 , ..., h Q n } ⊂ R d .

Then the attended vectors from question to context are { ĥC 1 , ..., ĥC m }:

EQUATION

EQUATION

EQUATION

where D ∈ R k×k is a diagonal matrix and U ∈ R d×k , k is the attention hidden size.

To simplify notation, we denote the above attention function as Attn(A, B, C), which linearly combines the vector set C using attention scores computed from vector sets A and B. This resembles the definition of attention in transformer (Vaswani et al., 2017) . It follows that the word-level interattention can be rewritten as Attn({h

C i } m i=1 , {h Q i } n i=1 }, {h Q i } n i=1 }

). Therefore, the input vector for each context word and question word is:

EQUATION

EQUATION

RNN. In this component, we use two separate bidirectional LSTMs (Hochreiter & Schmidhuber, 1997) to form the contextualized understanding for C and Q:

EQUATION

where h C,0 i = wC i , h Q,0 i = wQ i , 1 ≤ k ≤ K and K is the number of RNN layers. We use variational dropout (Kingma et al., 2015) for the input vector to each layer of RNN, i.e. the dropout mask is shared over different timesteps.

Question Understanding. For each question word in Q, we employ one more RNN layer to generate a higher level of understanding of the question.

EQUATION

EQUATION

Self-Attention on Question. As the question has integrated previous utterances, the model needs to directly relate the previously mentioned concept with the current question for context understanding. Therefore we employ self-attention on question:

{u Q i } n i=1 = Attn({h Q,K+1 i } n i=1 , {h Q,K+1 i } n i=1 , {h Q,K+1 i } n i=1 ) (11) {u Q i } n i=1

is the final representation of question words. Multilevel Inter-Attention. After multiple RNN layers extract different levels of semantic abstraction, we conduct inter-attention from question to context based on these representations.

However, the cumulative output dimensions from all previous layers can be very large and computationally inefficient. Here we leverage the history-of-word idea from FusionNet (Huang et al., 2017) : the attention uses all previous layers to compute scores, but only linearly combines one RNN layer output.

In detail, we conduct K + 1 times of multilevel inter-attention from each RNN layer output of question to context {m

(k),C i } m i=1 = Attn({HoW C i } m i=1 , {HoW Q i } n i=1 , {h Q,k i } n i=1 ), 1 ≤ k ≤ K + 1,

where HoW is the history-of-word vector:

EQUATION

An additional RNN layer is added to context C:

EQUATION

EQUATION

Self Attention on the Context. Similar to questions, SDNet applies self-attention to the context. Again, it uses the history-of-word concept to reduce the output dimensionality:

EQUATION

)

EQUATION

)

The self-attention is followed by an additional RNN layer to generate the final representation of context words:

EQUATION

2.5 OUTPUT LAYER Question Condensation. The question is condensed into a single representation vector:

EQUATION

)

EQUATION

)

where w is a trainable vector.

Generating answer span. As SDNet outputs answers of interval forms, the output layer generates the probability that the answer starts and ends at the i-th context word, 1 ≤ i ≤ m:

EQUATION

EQUATION

EQUATION

where W S , W E are parameters. The use of GRU is to transfer information from start position to end position computation. Special answer types. SDNet can also output special types of answer, such as affirmation "yes", negation "no" or no answer "unknown". We separately generate the probabilities of these three answers: P Y , P N , P U . For instance, the probability that the answer is "yes", P Y , is computed as:

EQUATION

EQUATION

where W Y and w Y are parametrized matrix and vector, respectively.

2.6. TRAINING AND INFERENCE:

During training, all rounds of questions and answers for the same passage form a batch. The goal is to maximize the probability of the ground-truth answer, including span start/end position, affirmation, negation and no-answer situations. Therefore, we minimize the cross-entropy loss function

EQUATION

where i s k and i e k are the ground-truth span start and end position for the k-th question.

I S k , I Y k , I N k , I U k

indicate whether the k-th ground-truth answer is a passage span, "yes", "no" and "unknown", respectively.

During inference, we pick the largest span/yes/no/unknown probability. The span is constrained to have a maximum length of 15.

3. EXPERIMENTS:

We evaluated our model on CoQA (Reddy et al., 2018) , a large-scale conversational question answering dataset. In CoQA, many questions require understanding of both the passage and previous rounds of questions and answers, which poses challenge to conventional machine reading models. Table 1 summarizes the domain distribution in CoQA. As shown, CoQA contains passages from multiple domains, and the average number of question answering turns is more than 15 per passage.

For each in-domain dataset, 100 passages are in the development set, and 100 passages are in the test set. The rest in-domain dataset are in the training set. The test set also includes all of the out-of-domain passages.

3.1. IMPLEMENTATION DETAILS:

We use spaCy for word tokenization and employ the uncased BERT-large model to generate contextual embedding.

During training, we use a dropout rate of 0.4 for BERT layer outputs and 0.3 for other layers. We use Adamax (Kingma & Ba, 2014) as the optimizer, with a learning rate of α = 0.002, β = (0.9, 0.999) and = 10 -8 . We train the model for 30 epochs. The gradient is clipped at 10. The word-level attention has a hidden size of 300. The self attention layer for question words has a hidden size of 300. The RNNs for question and context have K = 2 layers and each layer has a hidden size of 125. The multilevel attention from question to context has a hidden size of 250. The self attention layer for context has a hidden size of 250. The final RNN layer for context words has a hidden size of 125.

3.2. BASELINE MODELS AND RESULTS:

We compare SDNet2 with the following baseline models: DrQA+PGNet (Reddy et al., 2018) , BiDAF++ (Yatskar, 2018) and FlowQA (Huang et al., 2018) . Aligned with the official leaderboard, we use F 1 as the evaluation metric, which is the harmonic mean of precision and recall at word level between the predicted answer and ground truth. 3Table 2 shows the performance of SDNet and baseline models. 4 As shown, SDNet achieves significantly better results than baseline models. In detail, the single SDNet model improves overall F 1 by 1.6%, compared with previous state-of-art model on CoQA, FlowQA. We also trained an ensemble model consisting of 12 SDNet models with the same structure but different random seeds for initialization. The ensemble model uses the answer from the most number of models as its predicted answer. Ensemble SDNet model further improves overall F 1 score by 2.7%.

Figure 3 shows the F 1 score of SDNet on development set during training. As seen, SDNet overpasses all but one baseline models after the second epoch, and achieves state-of-the-art results after 8 epochs.

Ablation Studies. We conduct ablation studies on SDNet to verify the effectiveness of different parts of the model. As Table 3 shows, our proposed weighted sum of per-layer output from BERT is crucial, boosting the performance by 1.75% compared with the canonical method of using only the last layer's output. This shows that the output from each layer in BERT is useful in downstream tasks. Using BERT-base instead of the BERT-large pretrained model hurts the F 1 score by 2.61%.Variational dropout and self attention can each improve the performance by 0.24% and 0.75% respectively.

Contextual history. In SDNet, we utilize conversation history via prepending the current question with previous N rounds of questions and ground-truth answers. We experimented with the effect of N and present the result in Table 4 .

As shown, excluding dialogue history (N = 0) can reduce the F 1 score by as much as 8.56%, manifesting the importance of contextual information in conversational QA task. The performance of our model peaks when N = 2, which was used in the final SDNet model. q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qSDNet (78.0) 

FlowQA

4. CONCLUSIONS:

In this paper, we propose a novel contextual attention-based deep neural network, SDNet, to tackle the conversational question answering task. By leveraging inter-attention and self-attention on passage and conversation history, the model is able to comprehend dialogue flow and the passage. Furthermore, we leverage the latest breakthrough in NLP, BERT, as a contextual embedder. We design the alignment of tokenizers, linear combination and weight-locking techniques to adapt BERT into our model in a computation-efficient way. SDNet achieves superior results over previous approaches. On the public dataset CoQA, SDNet outperforms previous state-of-the-art model by 1.6% in overall F 1 score and the ensemble model further improves the F 1 by 2.7%.

Our future work is to apply this model to open-domain multiturn QA problem with large corpus or knowledge base, where the target passage may not be directly available. This will be a more realistic setting to human question answering.

:

To differentiate between question and answering, we add a special word Q before each question and A before each answer.

We have open-sourced our SDNet code. For anonymity, we will share the link afterwards.

According to official evaluation of CoQA, when there are more than one ground-truth answers, the final score is the average of max F1 against all-but-one ground-truth answers.

Result was taken from official CoQA leaderboard onNov. 30, 2018.

Figures and Tables:

Figure 2: SDNet model structure.

Figure 3: F 1 score on CoQA dev set over training epochs. Note that for BERT-base model, we use the number on test set from the leaderboard.

Domain distribution in CoQA dataset.

Table content: <table><tr><td>Domain</td><td>#Passage</td><td>#QA turn</td></tr><tr><td>Child Story</td><td>750</td><td>14.0</td></tr><tr><td>Literature</td><td>1,815</td><td>15.6</td></tr><tr><td>Mid/High Sc.</td><td>1,911</td><td>15.0</td></tr><tr><td>News</td><td>1,902</td><td>15.1</td></tr><tr><td>Wikipedia</td><td>1,821</td><td>15.4</td></tr><tr><td/><td>Out of domain</td><td/></tr><tr><td>Science</td><td>100</td><td>15.3</td></tr><tr><td>Reddit</td><td>100</td><td>16.6</td></tr><tr><td>Total</td><td>8,399</td><td>15.2</td></tr></table>

Model and human performance (% in F1 score) on the CoQA test set.

Table content: <table><tr><td/><td colspan="5">Child. Liter. Mid-High. News Wiki Reddit Science Overall</td></tr><tr><td>PGNet</td><td>49.0 43.3</td><td>47.5</td><td>47.5 45.1 38.6</td><td>38.1</td><td>44.1</td></tr><tr><td>DrQA</td><td>46.7 53.9</td><td>54.1</td><td>57.8 59.4 45.0</td><td>51.0</td><td>52.6</td></tr><tr><td>DrQA+PGNet</td><td>64.2 63.7</td><td>67.1</td><td>68.3 71.4 57.8</td><td>63.1</td><td>65.1</td></tr><tr><td>BiDAF++</td><td>66.5 65.7</td><td>70.2</td><td>71.6 72.6 60.8</td><td>67.1</td><td>67.8</td></tr><tr><td>FlowQA</td><td>73.7 71.6</td><td>76.8</td><td>79.0 80.2 67.8</td><td>76.1</td><td>75.0</td></tr><tr><td>SDNet (single)</td><td>75.4 73.9</td><td>77.1</td><td>80.3 83.1 69.8</td><td>76.8</td><td>76.6</td></tr><tr><td colspan="2">SDNet (ensemble) 78.7 77.1</td><td>80.2</td><td>81.9 85.2 72.3</td><td>79.7</td><td>79.3</td></tr><tr><td>Human</td><td>90.2 88.4</td><td>89.8</td><td>88.6 89.9 86.7</td><td>88.1</td><td>88.8</td></tr></table>

Ablation study of SDNet on CoQA development dataset.

Table content: <table><tr><td>Model</td><td>F 1</td></tr><tr><td>SDNet</td><td>77.99</td></tr><tr><td>-Variational dropout</td><td>77.75</td></tr><tr><td>-Question self attention</td><td>77.24</td></tr><tr><td>Using last layer of BERT output</td><td/></tr><tr><td>(no weighted sum)</td><td>76.24</td></tr><tr><td>BERT-base</td><td>75.38</td></tr></table>

Performance of SDNet on development set when prepending different number of rounds of history questions and answers to the question. The model uses BERT-Large contextual embedding and fixes BERT's weights.

Table content: <table><tr><td>#previous QA rounds N</td><td>F 1</td></tr><tr><td>0</td><td>69.43</td></tr><tr><td>1</td><td>76.70</td></tr><tr><td>2</td><td>77.99</td></tr><tr><td>3</td><td>77.39</td></tr></table>

Peer review (iclr2020):
The authors study the problem of conversational QA, proposing SDNet, a neural architecture based on RNNs, several levels/types of attention (as with many QA models), and a novel module for incorporating the contextualized embeddings of BERT. Specifically, as in DrQA+PGNet, previous dialogue turns are prepended to the current question and passed into an encoding layer than includes static word embeddings (GloVe) and pre-trained contextualized embeddings. The resulting representation is passed to an integration layer based on RNNs with self-attention on the context/question representations and inter-attention between then. Finally, the output layers relies on a bilinear projection to compute the probability of candidate answer spans. [Obviously, there are technical details (that are described fairly well in, and is the bulk of, the paper)]. Experiments are conducted on CoQA, showing improvements over FlowQA (and other baselines), presumably the state-of-the-art when this paper was written. Ablation studies show that the specific method of incorporating BERT (weighted sum per-layer) and incorporating dialogue history are important in achieving the stated performance. Given when this work was completed (and the paper written), it was ostensibly the state-of-the-art at the time. However, this is ~9 months from when this paper was submitted — so some of the impact of the empirical results have been attenuating in publishing. When written, the most significant novel contribution was likely the method used for incorporating BERT and combining different embeddings. However, given new contextualized embeddings developed since then, many works incorporating contextualized embeddings with several different strategies, and now knowing that ~1.5% isn’t that large of a jump on this dataset makes this contribution seem more marginal than when published. I am fairly certain that this could have been revisited between the original SDNet submission and now (and likely improving performance). Beyond this, the other neural components were fairly standard at the time of development and now are standard procedure — such that this doesn’t seem very novel at this point. While it is somewhat understandable that the performance isn’t state-of-the-art at the time of submission, it is likely necessary to contextualize this work within more recently developed work. Secondly, I would expect the authors to also evaluate their method on QuAC and complete a more thorough error analysis. Unfortunately, at this point, this paper has likely already had some impact, but seems to have missed its (time-sensitive) window for getting into a top-flight conference without continuing to innovate and be near the state-of-the-art at the time of submission and be appropriate situated. Thus, I recommend rejecting from ICLR as the paper simply reads like a resubmission that has missed its time (even if it is a nice paper in many regards).