Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering	The last several years have seen intensive interest in exploring neural-network-based models for machine comprehension (MC) and question answering (QA). In this paper, we approach the problems by closely modelling questions in a neural network framework. We first introduce syntactic information to help encode questions. We then view and model different types of questions and the information shared among them as an adaptation task and proposed adaptation models for them. On the Stanford Question Answering Dataset (SQuAD), we show that these approaches can help attain better results over a competitive baseline.

More specifically, the adaptation algorithm contains two steps: adapting and updating, which is detailed as follows: Adapting In the adapting step, we first compute the similarity score between an input question vector $\mathbf {x}\in \mathbb {R} ^{h}$ and each centroid vector of $K$ clusters $~\mathbf {\bar{x}}\in \mathbb {R} ^{K \times h}$ . Each cluster here models a question type. Unlike the explicit question type modeling discussed above, here we do not specify what question types we are modeling but let the system to learn. Specifically, we only need to pre-specific how many clusters, $K$ , we are modeling. The similarity between an input question and cluster centroid can be used to compute similarity weight $\mathbf {w}^a$ :  $$w_k^a = softmax(cos\_sim(\mathbf {x}, \mathbf {\bar{x}}_k), \alpha ), \forall k \in [1, \dots , K]$$   (Eq.

When letting the number of hidden question types ( $K$ ) to be 20, the performance improves to 68.73%/77.74% on EM and F1, respectively, which corresponds to the results of our model reported in Table 1 . Furthermore, after submitted our result, we have experimented with a large value of $K$ and found that when $K=100$ , we can achieve a better performance of 69.10%/78.38% on the development set. Figure UID61 shows the EM/F1 scores of different question types while Figure UID62 is the question type amount distribution on the development set. In Figure UID61 we can see that the average EM/F1 of the "when" question is highest and those of the "why" question is the lowest. From Figure UID62 we can see the "what" question is the major class. Figure 5 shows the composition of F1 score. Take our best model as an example, we observed a 78.38% F1 score on the whole development set, which can be separated into two parts: one is where F1 score equals to 100%, which means an exact match.

focused on word representation and presented a fine-grained gating mechanism to dynamically combine word-level and character-level representations based on the properties of words.  proposed a multi-perspective context matching (MPCM) model, which matched an encoded document and question from multiple perspectives.  proposed a dynamic decoder and so-called highway maxout network to improve the effectiveness of the decoder. The bi-directional attention flow (BIDAF)  used the bi-directional attention to obtain a question-aware context representation. In this paper, we introduce syntactic information to encode questions with a specific form of recursive neural networks  ,  ,  ,  . More specifically, we explore a tree-structured LSTM  ,  which extends the linear-chain long short-term memory (LSTM)  to a recursive structure, which has the potential to capture long-distance interactions over the structures. Different types of questions are often used to seek for different types of information. For example, a "what" question could have very different property from that of a "why" question, while they may share information and need to be trained together instead of separately.

For example, a "when" question seeks for different types of information (i.e., temporal information) than those for a "why" question. Different types of questions and the corresponding answers could potentially have different distributional regularity. The previous models are often trained for all questions without explicitly discriminating different question types; however, for a target question, both the common features shared by all questions and the specific features for a specific type of question are further considered in this paper, as they could potentially obey different distributions. In this paper we further explicitly model different types of questions in the end-to-end training. We start from a simple way to first analyze the word frequency of all questions, and obtain top-10 most frequent question types: what, how, who, when, which, where, why, be, whose, and whom, in which be stands for the questions beginning with different forms of the word be such as is, am, and are. We explicitly encode question-type information to be an 11-dimensional one-hot vector (the top-10 question types and "other" question type). Each question type is with a trainable embedding vector.

Conclusions	Closely modelling questions could be of importance for question answering and machine reading. In this paper, we introduce syntactic information to help encode questions in neural networks. We view and model different types of questions and the information shared among them as an adaptation task and proposed adaptation models for them. On the Stanford Question Answering Dataset (SQuAD), we show that these approaches can help attain better results over a competitive baseline.

23)  $$P(e+)=softmax(W_{e+} \cdot I^3 + W_{h+} \cdot h_{s+})$$   (Eq. 24)  where $\mathbf {I}^3$ is inference layer output, $\mathbf {h}_{s+}$ is the hidden state of the first step, and all $\mathbf {W}$ are trainable matrices. We also perform this by predicting the end position first and then the starting position:  $$P(e-)=softmax(W_{e-}\cdot I^3)$$   (Eq. 25)  $$P(s-)=softmax(W_{s-} \cdot I^3 + W_{h-} \cdot h_{e-})$$   (Eq. 26)  We finally identify the span of an answer with the following equation:  $$P(s)=pooling([P(s+), P(s-)])$$   (Eq. 27)  $$P(e)=pooling([P(e+), P(e-)])$$   (Eq.

We use three BiGRU layers to model the process that aggregates local information to make the global decision to find the answer spans. We found a residual architecture  as described in Figure 2 is very effective in this aggregation process:  $$\mathbf {I}^1_i=\text{BiGRU}(\mathbf {I}_i)$$   (Eq. 18)  $$\mathbf {I}^2_i=\mathbf {I}^1_i + \text{BiGRU}(\mathbf {I}^1_i)$$   (Eq. 19)  The SQuAD QA task requires a span of text to answer a question. We use a pointer network  to predict the starting and end position of answers as in  . Different from their methods, we use a two-directional prediction to obtain the positions. For one direction, we first predict the starting position of the answer span followed by predicting the end position, which is implemented with the following equations:  $$P(s+)=softmax(W_{s+}\cdot I^3)$$   (Eq.

14)  Through concatenating the document representation $\mathbf {D}^c$ , word-level Q-code $\mathbf {Q}^w$ and question-filtered document $\mathbf {D}^f$ , we can finally obtain the alignment layer representation:  $$\mathbf {I}=[\mathbf {D}^c, \mathbf {Q}^w,\mathbf {D}^c \circ \mathbf {Q}^w,\mathbf {D}^c - \mathbf {Q}^w, \mathbf {D}^f, \mathbf {b}^{f_{max}}, \mathbf {b}^{f_{mean}}] \in \mathbb {R} ^{M \times (6d_c+2)}$$   (Eq. 16)  where " $\circ $ " stands for element-wise multiplication and " $-$ " is simply the vector subtraction. After acquiring the local alignment representation, key information in document and question has been collected, and the aggregation layer is then performed to find answers.

Then the document softly filtered based on the corresponding question $\mathbf {D}^f$ can be calculated by:  $$\mathbf {D}_j^{f_{max}}=b^{f_{max}}_j \mathbf {D}_j^{c}, \forall j \in [1, \dots , M]$$   (Eq. 13)  $$\mathbf {D}_j^{f_{mean}}=b^{f_{mean}}_j \mathbf {D}_j^{c}, \forall j \in [1, \dots , M]$$   (Eq.

In the basic form of question-based filtering, for each question word $w_i$ , we find which words in the document are associated. Similar to $\mathbf {a}_j$ discussed above, we can obtain the attention weights on document words for each question word $w_i$ :  $$\mathbf {b}_i=softmax(\mathbf {U}_{i:})\in \mathbb {R} ^{M}, \forall i \in [1, \dots , N]$$   (Eq. 10)  By pooling $\mathbf {b}\in \mathbb {R} ^{N\times M}$ , we can obtain a question-based filtering weight $\mathbf {b}^f$ :  $$\mathbf {b}^f=norm(pooling(\mathbf {b})) \in \mathbb {R} ^{M}$$   (Eq. 11)  $$norm(\mathbf {x})=\frac{\mathbf {x}}{\sum _i x_i}$$   (Eq. 12)  where the specific pooling function we used include max-pooling and mean-pooling.

Questions and documents interact closely. As in most previous work, our framework use both soft attention over questions and that over documents to capture the interaction between them. More specifically, in this soft-alignment layer, we first feed the contextual representation matrix $\mathbf {Q}^c$ and $\mathbf {D}^c$ to obtain alignment matrix $\mathbf {U} \in \mathbb {R} ^{N\times M}$ :  $$\mathbf {U}_{ij} =\mathbf {Q}_i^c \cdot \mathbf {D}_j^{c\mathrm {T}}, \forall i \in [1, \dots , N], \forall j \in [1, \dots , M]$$   (Eq. 7)  Each $\mathbf {U}_{ij}$ represents the similarity between a question word $\mathbf {Q}_i^c$ and a document word $\mathbf {D}_j^c$ . Word-level Q-code Similar as in  , we obtain a word-level Q-code. Specifically, for each document word $w_j$ , we find which words in the question are relevant to it.

To this end, $\mathbf {a}_j\in \mathbb {R} ^{N}$ is computed with the following equation and used as a soft attention weight:  $$\mathbf {a}_j = softmax(\mathbf {U}_{:j}), \forall j \in [1, \dots , M]$$   (Eq. 8)  With the attention weights computed, we obtain the encoding of the question for each document word $w_j$ as follows, which we call word-level Q-code in this paper:  $$\mathbf {Q}^w=\mathbf {a}^{\mathrm {T}} \cdot \mathbf {Q}^{c} \in \mathbb {R} ^{M\times d_c}$$   (Eq. 9)  Question-based filtering To better explore question understanding, we design this question-based filtering layer. As detailed later, different question representation can be easily incorporated to this layer in addition to being used as a filter to find key information in the document based on the question. This layer is expandable with more complicated question modeling.

Question Understanding and Adaptation	The interplay of syntax and semantics of natural language questions is of interest for question representation. We attempt to incorporate syntactic information in questions representation with TreeLSTM  ,  . In general a TreeLSTM could perform semantic composition over given syntactic structures. Unlike the chain-structured LSTM  , the TreeLSTM captures long-distance interaction on a tree. The update of a TreeLSTM node is described at a high level with Equation ( 31 ), and the detailed computation is described in (–). Specifically, the input of a TreeLSTM node is used to configure four gates: the input gate $\mathbf {i}_t$ , output gate $\mathbf {o}_t$ , and the two forget gates $\mathbf {f}_t^L$ for the left child input and $\mathbf {f}_t^R$ for the right.

The above word representation focuses on representing individual words, and an input encoder here employs recurrent neural networks to obtain the representation of a word under its context. We use bi-directional GRU (BiGRU)  for both documents and questions. $${\mathbf {Q}^c_i}&=\text{BiGRU}(\mathbf {Q}^e_i,i),\forall i \in [1, \dots , N] \\
{\mathbf {D}^c_j}&=\text{BiGRU}(\mathbf {D}^e_j,j),\forall j \in [1, \dots , M]$$   (Eq. 5)  A BiGRU runs a forward and backward GRU on a sequence starting from the left and the right end, respectively. By concatenating the hidden states of these two GRUs for each word, we obtain the a representation for a question or document: $\mathbf {Q}^c \in \mathbb {R} ^{N\times d_c}$ for a question and $\mathbf {D}^c \in \mathbb {R} ^{M\times d_c}$ for a document.

The Baseline Model	Our baseline model is composed of the following typical components: word embedding, input encoder, alignment, aggregation, and prediction. Below we discuss these components in more details. We concatenate embedding at two levels to represent a word: the character composition and word-level embedding. The character composition feeds all characters of a word into a convolutional neural network (CNN)  to obtain a representation for the word. And we use the pre-trained 300-D GloVe vectors  (see the experiment section for details) to initialize our word-level embedding. Each word is therefore represented as the concatenation of the character-composition vector and word-level embedding. This is performed on both questions and documents, resulting in two matrices: the $\mathbf {Q}^e \in \mathbb {R} ^{N\times d_w}$ for a question and the $\mathbf {D}^e \in \mathbb {R} ^{M\times d_w}$ for a document, where $N$ is the question length (number of word tokens), $M$ is the document length, and $d_w$ is the embedding dimensionality.

We view this as a "adaptation" problem to let different types of questions share a basic model but still discriminate them when needed. Specifically, we are motivated by the ideas "i-vector"  in speech recognition, where neural network based adaptation is performed among different (groups) of speakers and we focused instead on different types of questions here.

Related Work	Recent advance on reading comprehension and question answering has been closely associated with the availability of various datasets.  released the MCTest data consisting of 500 short, fictional open-domain stories and 2000 questions. The CNN/Daily Mail dataset  contains news articles for close style machine comprehension, in which only entities are removed and tested for comprehension. Children's Book Test (CBT)  leverages named entities, common nouns, verbs, and prepositions to test reading comprehension. The Stanford Question Answering Dataset (SQuAD)  is more recently released dataset, which consists of more than 100,000 questions for documents taken from Wikipedia across a wide range of topics. The question-answer pairs are annotated through crowdsourcing. Answers are spans of text marked in the original documents. In this paper, we use SQuAD to evaluate our models. Many neural network models have been studied on the SQuAD task.  proposed match LSTM to associate documents and questions and adapted the so-called pointer Network  to determine the positions of the answer text spans.  proposed a dynamic chunk reader to extract and rank a set of answer candidates.

28)  We use the mean-pooling here as it is more effective on the development set than the alternatives such as the max-pooling.

\\\mathbf {c}_t &= \mathbf {f}_t^L \circ \mathbf {c}_{t-1}^L + \mathbf {f}_t^R \circ \mathbf {c}_{t-1}^R + \mathbf {i}_t \circ \mathbf {u}_t, \\\mathbf {f}_t^L &= \sigma (\mathbf {W}_f \mathbf {x}_t + \mathbf {U}_f^{LL} \mathbf {h}_{t-1}^L + \mathbf {U}_f^{LR} \mathbf {h}_{t-1}^R),\\
\mathbf {f}_t^R &= \sigma (\mathbf {W}_f \mathbf {x}_t + \mathbf {U}_f^{RL} \mathbf {h}_{t-1}^L + \mathbf {U}_f^{RR} \mathbf {h}_{t-1}^R),

The memory cell $\mathbf {c}_t$ considers each child's cell vector, $\mathbf {c}_{t-1}^L$ and $\mathbf {c}_{t-1}^R$ , which are gated by the left forget gate $\mathbf {f}_t^L$ and right forget gate $\mathbf {f}_t^R$ , respectively. $$\mathbf {h}_t &= \text{TreeLSTM}(\mathbf {x}_t, \mathbf {h}_{t-1}^L, \mathbf {h}_{t-1}^R), \\

\mathbf {h}_t &= \mathbf {o}_t \circ \tanh (\mathbf {c}_{t}),\\
\mathbf {o}_t &= \sigma (\mathbf {W}_o \mathbf {x}_t + \mathbf {U}_o^L \mathbf {h}_{t-1}^L + \mathbf {U}_o^R \mathbf {h}_{t-1}^R),

This part accounts for 69.10% of the entire development set. And the other part accounts for 30.90%, of which the average F1 score is 30.03%. For the latter, we can further divide it into two sub-parts: one is where the F1 score equals to 0%, which means that predict answer is totally wrong. This part occupies 14.89% of the total development set. The other part accounts for 16.01% of the development set, of which average F1 score is 57.96%. From this analysis we can see that reducing the zero F1 score (14.89%) is potentially an important direction to further improve the system.

\\\mathbf {i}_t &= \sigma (\mathbf {W}_i \mathbf {x}_t + \mathbf {U}_i^L \mathbf {h}_{t-1}^L + \mathbf {U}_i^R \mathbf {h}_{t-1}^R), \\\mathbf {u}_t &= \tanh (\mathbf {W}_c \mathbf {x}_t + \mathbf {U}_c^L \mathbf {h}_{t-1}^L + \mathbf {U}_c^R \mathbf {h}_{t-1}^R),$$   (Eq. 31)  where $\sigma $ is the sigmoid function, $\circ $ is the element-wise multiplication of two vectors, and all $\mathbf {W}$ , $\mathbf {U}$ are trainable matrices. To obtain the parse tree information, we use Stanford CoreNLP (PCFG Parser)  ,  to produce a binarized constituency parse for each question and build the TreeLSTM based on the parse tree. The root node of TreeLSTM is used as the representation for the whole question.

More specifically, we use it as TreeLSTM Q-code $\mathbf {Q}^{TL}\in \mathbb {R} ^{d_c}$ , by not only simply concatenating it to the alignment layer output but also using it as a question filter, just as we discussed in the question-based filtering section:  $$\mathbf {Q}^{TL}=\text{TreeLSTM}(\mathbf {Q}^e) \in \mathbb {R} ^{d_c}$$   (Eq. 32)  $$\mathbf {b}^{TL}=norm(\mathbf {Q}^{TL} \cdot \mathbf {D}^{c\mathrm {T}}) \in \mathbb {R} ^{M}$$   (Eq. 33)  where $\mathbf {I}_{new}$ is the new output of alignment layer, and function $repmat$ copies $\mathbf {Q}^{TL}$ for M times to fit with $\mathbf {I}$ . Questions by nature are often composed to fulfill different types of information needs.

We call this explicit question type code, $\mathbf {ET}\in \mathbb {R} ^{d_{ET}}$ . Then the vector for each question type is tuned during training, and is added to the system with the following equation:  $$\mathbf {I}_{new}=[\mathbf {I}, repmat(\mathbf {ET})]$$   (Eq. 38)  As discussed, different types of questions and their answers may share common regularity and have separate property at the same time. We also view this as an adaptation problem in order to let different types of questions share a basic model but still discriminate them when needed. Specifically, we borrow ideas from speaker adaptation  in speech recognition, where neural-network-based adaptation is performed among different groups of speakers. Conceptually we regard a type of questions as a group of acoustically similar speakers. Specifically we propose a question discriminative block or simply called a discriminative block (Figure 3 ) below to perform question adaptation.

The main idea is described below:  $$\mathbf {x^\prime } = f([\mathbf {x}, \mathbf {\bar{x}}^c, \mathbf {\delta _x}])$$   (Eq. 40)  For each input question $\mathbf {x}$ , we can decompose it to two parts: the cluster it belong(i.e., question type) and the diverse in the cluster. The information of the cluster is encoded in a vector $\mathbf {\bar{x}}^c$ . In order to keep calculation differentiable, we compute the weight of all the clusters based on the distances of $\mathbf {x}$ and each cluster center vector, in stead of just choosing the closest cluster. Then the discriminative vector $\mathbf {\delta _x}$ with regard to these most relevant clusters are computed. All this information is combined to obtain the discriminative information. In order to keep the full information of input, we also copy the input question $\mathbf {x}$ , together with the acquired discriminative information, to a feed-forward layer to obtain a new representation $\mathbf {x^\prime }$ for the question.

43)  $$cos\_sim(\mathbf {u}, \mathbf {v}) = \frac{<\mathbf {u},\mathbf {v}>}{||\mathbf {u}|| \cdot ||\mathbf {v}||}$$   (Eq. 44)  We set $\alpha $ equals 50 to make sure only closest class will have a high weight while maintain differentiable. Then we acquire a soft class-center vector $\mathbf {\bar{x}}^c$ :  $$\mathbf {\bar{x}}^c = \sum _k w^a_k \mathbf {\bar{x}}_k \in \mathbb {R} ^{h}$$   (Eq. 46)  We then compute a discriminative vector $\mathbf {\delta _x}$ between the input question with regard to the soft class-center vector:  $$\mathbf {\delta _x} = \mathbf {x} - \mathbf {\bar{x}}^c$$   (Eq.

47)  Note that $\bar{\mathbf {x}}^c$ here models the cluster information and $\mathbf {\delta _x}$ represents the discriminative information in the cluster. By feeding $\mathbf {x}$ , $\bar{\mathbf {x}}^c$ and $\mathbf {\delta _x}$ into feedforward layer with Relu, we obtain $\mathbf {x^{\prime }}\in \mathbb {R} ^{K}$ :  $$\mathbf {x^{\prime }} = Relu(\mathbf {W} \cdot [\mathbf {x},\bar{\mathbf {x}}^c,\mathbf {\delta _x}])$$   (Eq. 48)  With $\mathbf {x^{\prime }}$ ready, we can apply Discriminative Block to any question code and obtain its adaptation Q-code. In this paper, we use TreeLSTM Q-code as the input vector $\mathbf {x}$ , and obtain TreeLSTM adaptation Q-code $\mathbf {Q}^{TLa}\in \mathbb {R} ^{d_c}$ .

Similar to TreeLSTM Q-code $\mathbf {Q}^{TL}$ , we concatenate $\mathbf {Q}^{TLa}$ to alignment output $\mathbf {I}$ and also use it as a question filter:  $$\mathbf {Q}^{TLa} = Relu(\mathbf {W} \cdot [\mathbf {Q}^{TL},\overline{\mathbf {Q}^{TL}}^c,\mathbf {\delta _{\mathbf {Q}^{TL}}}])$$   (Eq. 49)  $$\mathbf {b}^{TLa}=norm(\mathbf {Q}^{TLa} \cdot \mathbf {D}^{c\mathrm {T}}) \in \mathbb {R} ^{M}$$   (Eq. 50)  Updating The updating stage attempts to modify the center vectors of the $K$ clusters in order to fit each cluster to model different types of questions.

The updating is performed according to the following formula:  $$\mathbf {\bar{x}^{\prime }}_k = (1-\beta \text{w}_k^a)\mathbf {\bar{x}}_k+\beta \text{w}_k^a\mathbf {x}, \forall k \in [1, \dots , K]$$   (Eq. 54)  In the equation, $\beta $ is an updating rate used to control the amount of each updating, and we set it to 0.01. When $\mathbf {x}$ is far away from $K$ -th cluster center $\mathbf {\bar{x}}_k$ , $\text{w}_k^a$ is close to be value 0 and the $k$ -th cluster center $\mathbf {\bar{x}}_k$ tends not to be updated.

If $\mathbf {x}$ is instead close to the $j$ -th cluster center $\mathbf {\bar{x}}_j$ , $\mathbf {x}$0 is close to the value 1 and the centroid of the $\mathbf {x}$1 -th cluster $\mathbf {x}$2 will be updated more aggressively using $\mathbf {x}$3 .

Set-Up	We test our models on Stanford Question Answering Dataset (SQuAD)  . The SQuAD dataset consists of more than 100,000 questions annotated by crowdsourcing workers on a selected set of Wikipedia articles, and the answer to each question is a span of text in the Wikipedia articles. Training data includes 87,599 instances and validation set has 10,570 instances. The test data is hidden and kept by the organizer. The evaluation of SQuAD is Exact Match (EM) and F1 score. We use pre-trained 300-D Glove 840B vectors  to initialize our word embeddings. Out-of-vocabulary (OOV) words are initialized randomly with Gaussian samples. CharCNN filter length is 1,3,5, each is 50 dimensions. All vectors including word embedding are updated during training. The cluster number K in discriminative block is 100. The Adam method  is used for optimization. And the first momentum is set to be 0.9 and the second 0.999. The initial learning rate is 0.0004 and the batch size is 32.

We will half learning rate when meet a bad iteration, and the patience is 7. Our early stop evaluation is the EM and F1 score of validation set. All hidden states of GRUs, and TreeLSTMs are 500 dimensions, while word-level embedding $d_w$ is 300 dimensions. We set max length of document to 500, and drop the question-document pairs beyond this on training set. Explicit question-type dimension $d_{ET}$ is 50. We apply dropout to the Encoder layer and aggregation layer with a dropout rate of 0.5.

Results	Table 1 shows the official leaderboard on SQuAD test set when we submitted our system. Our model achieves a 68.73% EM score and 77.39% F1 score, which is ranked among the state of the art single models (without model ensembling). Table 2 shows the ablation performances of various Q-code on the development set. Note that since the testset is hidden from us, we can only perform such an analysis on the development set. Our baseline model using no Q-code achieved a 68.00% and 77.36% EM and F1 scores, respectively. When we added the explicit question type T-code into the baseline model, the performance was improved slightly to 68.16%(EM) and 77.58%(F1). We then used TreeLSTM introduce syntactic parses for question representation and understanding (replacing simple question type as question understanding Q-code), which consistently shows further improvement. We further incorporated the soft adaptation.

Introduction	Enabling computers to understand given documents and answer questions about their content has recently attracted intensive interest, including but not limited to the efforts as in  ,  ,  ,  ,  ,  . Many specific problems such as machine comprehension and question answering often involve modeling such question-document pairs. The recent availability of relatively large training datasets (see Section "Related Work" for more details) has made it more feasible to train and estimate rather complex models in an end-to-end fashion for these problems, in which a whole model is fit directly with given question-answer tuples and the resulting model has shown to be rather effective. In this paper, we take a closer look at modeling questions in such an end-to-end neural network framework, since we regard question understanding is of importance for such problems. We first introduced syntactic information to help encode questions. We then viewed and modelled different types of questions and the information shared among them as an adaptation problem and proposed adaptation models for them. On the Stanford Question Answering Dataset (SQuAD), we show that these approaches can help attain better results on our competitive baselines.

Question: What MC abbreviate for?
Answer: machine comprehension