SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering	Conversational question answering (CQA) is a novel QA task that requires understanding of dialogue context. Different from traditional single-turn machine reading comprehension (MRC) tasks, CQA includes passage comprehension, coreference resolution, and contextual understanding. In this paper, we propose an innovated contextualized attention-based deep neural network, SDNet, to fuse context into traditional MRC models. Our model leverages both inter-attention and self-attention to comprehend conversation context and extract relevant information from passage. Furthermore, we demonstrated a novel method to integrate the latest BERT contextual model. Empirical results show the effectiveness of our model, which sets the new state of the art result in CoQA leaderboard, outperforming the previous best model by 1.6% F1. Our ensemble model further improves the result by 2.7% F1.

$
{h}_1^{Q,K+1}, ..., {h}_n^{Q,K+1} = \operatorname{\mbox{BiLSTM}}{({h}_1^{Q}, ..., {h}_n^{Q})},
$ $
{h}_i^{Q} = [{h}_i^{Q,1};...;{h}_i^{Q,K}]
$  Self-Attention on Question. As the question has integrated previous utterances, the model needs to directly relate previously mentioned concept with the current question. This is helpful for concept carry-over and coreference resolution. We thus employ self-attention on question. The formula is the same as word-level attention, except that we are attending a question to itself: $\lbrace {u}_i^Q\rbrace _{i=1}^n=\mbox{Attn}(\lbrace {h}_i^{Q,K+1}\rbrace _{i=1}^n, \lbrace {h}_i^{Q,K+1}\rbrace _{i=1}^n, \lbrace {h}_i^{Q,K+1}\rbrace _{i=1}^n)$ .

Conclusions	In this paper, we propose a novel contextual attention-based deep neural network, SDNet, to tackle conversational question answering task. By leveraging inter-attention and self-attention on passage and conversation history, the model is able to comprehend dialogue flow and fuse it with the digestion of passage content. Furthermore, we incorporate the latest breakthrough in NLP, BERT, and leverage it in an innovative way. SDNet achieves superior results over previous approaches. On the public dataset CoQA, SDNet outperforms previous state-of-the-art model by 1.6% in overall $F_1$ metric. Our future work is to apply this model to open-domain multiturn QA problem with large corpus or knowledge base, where the target passage may not be directly available. This will be an even more realistic setting to human question answering.

Firstly, we apply both inter-attention and self-attention on passage and question to obtain a more effective understanding of the passage and dialogue history. Secondly, SDNet leverages the latest breakthrough in NLP: BERT contextual embedding  . Different from the canonical way of appending a thin layer after BERT structure according to  , we innovatively employed a weighted sum of BERT layer outputs, with locked BERT parameters. Thirdly, we prepend previous rounds of questions and answers to the current question to incorporate contextual information. Empirical results show that each of these components has substantial gains in prediction accuracy. We evaluated SDNet on CoQA dataset, which improves the previous state-of-the-art model's result by 1.6% (from 75.0% to 76.6%) overall $F_1$ score. The ensemble model further increase the $F_1$ score to $79.3\%$ . Moreover, SDNet is the first model ever to pass $80\%$ on CoQA's in-domain dataset.

Introduction	Traditional machine reading comprehension (MRC) tasks share the single-turn setting of answering a single question related to a passage. There is usually no connection between different questions and answers to the same passage. However, the most natural way humans seek answers is via conversation, which carries over context through the dialogue flow. To incorporate conversation into reading comprehension, recently there are several public datasets that evaluate QA model's efficacy in conversational setting, such as CoQA  , QuAC  and QBLink  . In these datasets, to generate correct responses, models are required to fully understand the given passage as well as the context of previous questions and answers. Thus, traditional neural MRC models are not suitable to be directly applied to this scenario. Existing approaches to conversational QA tasks include BiDAF++  , FlowQA  , DrQA+PGNet  , which all try to find the optimal answer span given the passage and dialogue history. In this paper, we propose SDNet, a contextual attention-based deep neural network for the task of conversational question answering. Our network stems from machine reading comprehension models, but has several unique characteristics to tackle contextual understanding during conversation.

Approach	In this section, we propose the neural model, SDNet, for the conversational question answering task, which is formulated as follows. Given a passage $\mathcal {C}$ , and history question and answer utterances $Q_1, A_1, Q_2, A_2, ..., Q_{k-1}, A_{k-1}$ , the task is to generate response $A_k$ given the latest question $Q_k$ . The response is dependent on both the passage and history utterances. To incorporate conversation history into response generation, we employ the idea from DrQA+PGNet  to prepend the latest $N$ rounds of utterances to the current question $Q_k$ . The problem is then converted into a machine reading comprehension task. In other words, the reformulate question is $\mathcal {Q}_k=\lbrace Q_{k-N}; A_{k-N}; ..., Q_{k-1}; A_{k-1}; Q_k\rbrace $ . To differentiate between question and answering, we add symbol $\langle Q \rangle $ before each question and $\langle A \rangle $ before each answer in the experiment.

table:mainresult report the performance of SDNet and baseline models. As shown, SDNet achieves significantly better results than baseline models. In detail, the single SDNet model improves overall $F_1$ by 1.6%, compared with previous state-of-art model on CoQA, FlowQA. Ensemble SDNet model further improves overall $F_1$ score by 2.7%, and it's the first model to achieve over 80% $F_1$ score on in-domain datasets (80.7%). fig:epoch shows the $F_1$ score on development set over epochs. As seen, SDNet overpasses all but one baseline models after the second epoch, and achieves state-of-the-art results only after 8 epochs. Ablation Studies. We conduct ablation studies on SDNet model and display the results in table:ablation. The results show that removing BERT can reduce the $F_1$ score on development set by $7.15\%$ . Our proposed weight sum of per-layer output from BERT is crucial, which can boost the performance by $1.75\%$ , compared with using only last layer's output.

This shows that the output from each layer in BERT is useful in downstream tasks. This technique can also be applied to other NLP tasks. Using BERT-base instead of BERT-large pretrained model hurts the $F_1$ score by $2.61\%$ , which manifests the superiority of BERT-large model. Variational dropout and self attention can each improve the performance by 0.24% and 0.75%, respectively. Contextual history. In SDNet, we utilize conversation history via prepending the current question with previous $N$ rounds of questions and ground-truth answers. We experimented the effect of $N$ and show the result in table:context. Excluding dialogue history ( $N=0$ ) can reduce the $F_1$ score by as much as $8.56\%$ , showing the importance of contextual information in conversational QA task. The performance of our model peaks when $N=2$ , which was used in the final SDNet model.

Experiments	We evaluated our model on CoQA  , a large-scale conversational question answering dataset. In CoQA, many questions require understanding of both the passage and previous questions and answers, which poses challenge to conventional machine reading models. table:coqa summarizes the domain distribution in CoQA. As shown, CoQA contains passages from multiple domains, and the average number of question answering turns is more than 15 per passage. Many questions require contextual understanding to generate the correct answer. For each in-domain dataset, 100 passages are in the development set, and 100 passages are in the test set. The rest in-domain dataset are in the training set. The test set also includes all of the out-of-domain passages. Baseline models and metrics. We compare SDNet with the following baseline models: PGNet (Seq2Seq with copy mechanism)  , DrQA  , DrQA+PGNet  , BiDAF++  and FlowQA  . Aligned with the official leaderboard, we use $F_1$ as the evaluation metric, which is the harmonic mean of precision and recall at word level between the predicted answer and ground truth. Results.

Model Overview	Encoding layer encodes each token in passage and question into a fixed-length vector, which includes both word embeddings and contextualized embeddings. For contextualized embedding, we utilize the latest result from BERT  . Different from previous work, we fix the parameters in BERT model and use the linear combination of embeddings from different layers in BERT. Integration layer uses multi-layer recurrent neural networks (RNN) to capture contextual information within passage and question. To characterize the relationship between passage and question, we conduct word-level attention from question to passage both before and after the RNNs. We employ the idea of history-of-word from FusionNet  to reduce the dimension of output hidden vectors. Furthermore, we conduct self-attention to extract relationship between words at different positions of context and question. Output layer computes the final answer span. It uses attention to condense the question into a fixed-length vector, which is then used in a bilinear projection to obtain the probability that the answer should start and end at each position. An illustration of our model SDNet is in fig:model.

Implementation Details	We use spaCy for tokenization. As BERT use BPE as the tokenizer, we did BPE tokenization for each token generated by spaCy. In case a token in spaCy corresponds to multiple BPE sub-tokens, we average the BERT embeddings of these BPE sub-tokens as the embedding for the token. We fix the BERT weights and use the BERT-Large-Uncased model. During training, we use a dropout rate of 0.4 for BERT layer outputs and 0.3 for other layers. We use variational dropout  , which shares the dropout mask over timesteps in RNN. We batch the data according to passages, so all questions and answers from the same passage make one batch. We use Adamax  as the optimizer, with a learning rate of $\alpha =0.002, \beta =(0.9, 0.999)$ and $\epsilon =10^{-8}$ . We train the model using 30 epochs, with each epoch going over the data once. We clip the gradient at length 10. The word-level attention has a hidden size of 300.

Encoding layer	We use 300-dim GloVe  embedding and contextualized embedding for each word in context and question. We employ BERT  as contextualized embedding. Instead of adding a scoring layer to BERT structure as proposed in  , we use the transformer output from BERT as contextualized embedding in our encoding layer. BERT generates $L$ layers of hidden states for all BPE tokens  in a sentence/passage and we employ a weighted sum of these hidden states to obtain contextualized embedding. Furthermore, we lock BERT's internal weights, setting their gradients to zero. In ablation studies, we will show that this weighted sum and weight-locking mechanism can significantly boost the model's performance. In detail, suppose a word $w$ is tokenized to $s$ BPE tokens $w=\lbrace b_1, b_2, ..., b_s\rbrace $ , and BERT generates $L$ hidden states for each BPE token, $\mathbf {h^l_t}, 1\le l \le L, 1\le t \le s$ .

The flow module has a hidden size of 300. The question self attention has a hidden size of 300. The RNN for both question and context has $K=2$ layers and each layer has a hidden size of 125. The multilevel attention from question to context has a hidden size of 250. The context self attention has a hidden size of 250. The final layer of RNN for context has a hidden size of 125.

Therefore, the input vector for each context word is $\tilde{{w}}_i^C=[\operatorname{GloVe}(w_i^C); \operatorname{\mbox{BERT}}_{w_i^C}; \hat{{h}}^C_i; f_{w_i^C}]$ ; the input vector for each question word is $\tilde{{w}}_i^Q=[\operatorname{GloVe}(w_i^Q); \operatorname{\mbox{BERT}}_{w_i^Q}]$ . RNN. In this component, we use two separate bidirectional RNNs (BiLSTMs  ) to form the contextualized understanding for $\mathcal {C}$ and $\mathcal {Q}$ .

Integration layer	Word-level Inter-Attention. We conduct attention from question to context (passage) based on GloVe word embeddings. Suppose the context word embeddings are $\lbrace {h}^C_1, ..., {h}^C_m\rbrace \subset \mathbb {R}^d$ , and the question word embeddings are $\lbrace {h}^Q_1, ..., {h}^Q_n\rbrace \subset \mathbb {R}^d$ . Then the attended vectors from question to context are $\lbrace \hat{{h}}^C_1, ..., \hat{{h}}^C_m\rbrace $ , defined as, $S_{ij} = \operatornamewithlimits{ReLU}(Uh^C_i)D\operatornamewithlimits{ReLU}(Uh^Q_j),$ $\alpha _{ij} \propto {exp(S_{ij})},$  where $D\in \mathbb {R}^{k\times k}$ is a diagonal matrix and $U\in \mathbb {R}^{d\times k}$ , $k$ is the attention hidden size.

To simplify notation, we define the attention function above as $\mbox{Attn}({A}, {B}, {C})$ , meaning we compute the attention score $\alpha _{ij}$ based on two sets of vectors ${A}$ and ${B}$ , and use that to linearly combine vector set ${C}$ . So the word-level attention above can be simplified as $\mbox{Attn}(\lbrace {h}^C_i\rbrace _{i=1}^m, \lbrace {h}^Q_i\rbrace _{i=1}^n\rbrace , \lbrace {h}^Q_i\rbrace _{i=1}^n\rbrace )$ . For each context word in $\mathcal {C}$ , we also include a feature vector $f_w$ including 12-dim POS embedding, 8-dim NER embedding, a 3-dim exact matching vector $em_i$ indicating whether each context word appears in the question, and a normalized term frequency, following the approach in DrQA  .

The final question representation is thus $\lbrace {u}_i^Q\rbrace _{i=1}^n$ . Multilevel Inter-Attention. After multiple layers of RNN extract different levels of understanding of each word, we conduct multilevel attention from question to context based on all layers of generated representations. However, the aggregated dimensions can be very large, which is computationally inefficient. We thus leverage the history-of-word idea from FusionNet  : we use all previous levels to compute attentions scores, but only linearly combine RNN outputs. In detail, we conduct $K+1$ times of multilevel attention from each RNN layer output of question to context. $
\lbrace {m}_i^{(k),C}\rbrace _{i=1}^m=\mbox{Attn}(\lbrace \mbox{HoW}_i^C\rbrace _{i=1}^m, \lbrace \mbox{HoW}_i^Q\rbrace _{i=1}^n,\lbrace {h}_i^{Q,k}\rbrace _{i=1}^n),

$
{h}_1^{C,k}, ..., {h}_m^{C,k} = \operatorname{\mbox{BiLSTM}}{({h}_1^{C,k-1}, ..., {h}_m^{C,k-1})},
$ $
{h}_1^{Q,k}, ..., {h}_n^{Q,k} = \operatorname{\mbox{BiLSTM}}{({h}_1^{Q,k-1}, ..., {h}_n^{Q,k-1})},
$  where $1\le k \le K$ and $K$ is the number of RNN layers. We use variational dropout  for input vector to each layer of RNN, i.e. the dropout mask is shared over different timesteps. Question Understanding. For each question word in $\mathcal {Q}$ , we employ one more layer of RNN to generate a higher level of understanding of the question.

1\le k \le K+1
$  where history-of-word vectors are defined as $\mbox{HoW}_i^C = [\operatorname{GloVe}(w_i^C); \operatorname{\mbox{BERT}}_{w_i^C}; {h}_i^{C,1}; . {h}_i^{C,k}],$ $\mbox{HoW}_i^Q = [\operatorname{GloVe}(w_i^Q); \operatorname{\mbox{BERT}}_{w_i^Q}; {h}_i^{Q,1}; . {h}_i^{Q,k}].$  An additional RNN layer is applied to obtain the contextualized representation ${v}_i^C$ for each word in $\mathcal {C}$ .

$
{y}_i^C = [{h}_i^{C,1}; ...; {h}_i^{C,k}; {m}_i^{(1),C}; ...; {m}_i^{(K+1),C}],
$ $
{v}_1^{C}, ..., {v}_m^{C} = \operatorname{\mbox{BiLSTM}}{({y}_1^{C}, ..., {y}_n^{C})},
$  Self Attention on Context. Similar to questions, we conduct self attention on context to establish direct correlations between all pairs of words in $\mathcal {C}$ . Again, we use the history of word concept to reduce the output dimension by linearly combining ${v}_i^C$ . $
{s}_i^C = &[\operatorname{GloVe}(w_i^C); \operatorname{\mbox{BERT}}_{w_i^C}; {h}_i^{C,1}; . {h}_i^{C,k}; {m}_i^{(1),Q}; . {m}_i^{(K+1),Q};

{v}_i^C]
$ $\lbrace \tilde{{v}}_i^C\rbrace _{i=1}^m=\mbox{Attn}(\lbrace {s}_i^C\rbrace _{i=1}^m, \lbrace {s}_i^C\rbrace _{i=1}^m, \lbrace {v}_i^C\rbrace _{i=1}^m)$  The self-attention is followed by an additional layer of RNN to generate the final representation of context: $\lbrace {u}_i^C\rbrace _{i=1}^m = \operatorname{\mbox{BiLSTM}}{([{v}_1^C; \tilde{{v}}_1^C], . [{v}_m^C; \tilde{{v}}_m^C])}$

Output layer	Generating Answer Span. This component is to generate two scores for each context word corresponding to the probability that the answer starts and ends at this word, respectively. Firstly, we condense the question representation into one vector: ${u}^Q=\sum _i{\beta _i}{u}_i^Q$ , where $\beta _i\propto {\exp {({w}^T{u}_i^Q)}}$ and ${w}$ is a parametrized vector. Secondly, we compute the probability that the answer span should start at the $i$ -th word: $P_i^S\propto {\exp {(({u}^Q)^TW_S{u}_i^C)}},$  where $W_S$ is a parametrized matrix. We further fuse the start-position probability into the computation of end-position probability via a GRU, ${t}^Q = \operatorname{GRU}{({u}^Q, \sum _i P_i^S{u}_i^C)}$ .

Thus, the probability that the answer span should end at the $i$ -th word is: $P_i^E\propto {\exp {(({t}^Q)^TW_E{u}_i^C)}},$  where $W_E$ is another parametrized matrix. For CoQA dataset, the answer could be affirmation “yes”, negation “no” or no answer “unknown”. We separately generate three probabilities corresponding to these three scenarios, $P_Y, P_N, P_U$ , respectively. For instance, to generate the probability that the answer is “yes”, $P_Y$ , we use: $P_i^{Y}\propto {\exp {(({u}^Q)^T W_{Y}{u}_i^C})},$ $P_{Y} = (\sum _i P_i^{Y}{u}_i^C)^T{w}_{Y},$  where $W_Y$ and ${w}_Y$ are parametrized matrix and vector, respectively. Training. For training, we use all questions/answers for one passage as a batch.

The goal is to maximize the probability of the ground-truth answer, including span start/end position, affirmation, negation and no-answer situations. Equivalently, we minimize the negative log-likelihood function $\mathcal {L}$ : $
\mathcal {L} =& \sum _k I^S_k(\mbox{log}(P^S_{i_k^s}) + \mbox{log}(P^E_{i_k^e})) + I^Y_k\mbox{log}P^Y_k+I^N_k\mbox{log}P^N_k + I^U_k\mbox{log}P^U_k,
$   where $i_k^s$ and $i_k^e$ are the ground-truth span start and end position for the $k$ -th question. $I^S_k, I^Y_k, I^N_k, I^U_k$ indicate whether the $k$ -th ground-truth answer is a passage span, “yes”, “no” and “unknown”, respectively. More implementation details are in Appendix. Prediction. During inference, we pick the largest span/yes/no/unknown probability.

The span is constrained to have a maximum length of 15.

The contextual embedding $\operatorname{\mbox{BERT}}_w$ for word $w$ is then a per-layer weighted sum of average BERT embedding, with weights $\alpha _1, ..., \alpha _L$ . $\operatorname{\mbox{BERT}}_w = \sum _{l=1}^L \alpha _l \frac{\sum _{t=1}^s \mathbf {h}^l_t}{s}
$

Question: Does the model incorporate coreference and entailment?
Answer: As the question has integrated previous utterances, the model needs to directly relate previously mentioned concept with the current question. This is helpful for concept carry-over and coreference resolution.