Title: DYNAMIC COATTENTION NETWORKS FOR QUESTION ANSWERING

Abstract: Several deep learning models have been proposed for question answering. However, due to their single-pass nature, they have no way to recover from local maxima corresponding to incorrect answers. To address this problem, we introduce the Dynamic Coattention Network (DCN) for question answering. The DCN first fuses co-dependent representations of the question and the document in order to focus on relevant parts of both. Then a dynamic pointing decoder iterates over potential answer spans. This iterative procedure enables the model to recover from initial local maxima corresponding to incorrect answers. On the Stanford question answering dataset, a single DCN model improves the previous state of the art from 71.0% F1 to 75.9%, while a DCN ensemble obtains 80.4% F1. * Equal contribution 1 As of Nov. 3 2016. See https://rajpurkar.github.io/SQuAD-

1. INTRODUCTION:

Question answering (QA) is a crucial task in natural language processing that requires both natural language understanding and world knowledge. Previous QA datasets tend to be high in quality due to human annotation, but small in size (Berant et al., 2014; Richardson et al., 2013) . Hence, they did not allow for training data-intensive, expressive models such as deep neural networks.

To address this problem, researchers have developed large-scale datasets through semi-automated techniques (Hermann et al., 2015; Hill et al., 2016) . Compared to their smaller, hand-annotated counterparts, these QA datasets allow the training of more expressive models. However, it has been shown that they differ from more natural, human annotated datasets in the types of reasoning required to answer the questions (Chen et al., 2016) .

Recently, Rajpurkar et al. (2016) released the Stanford Question Answering dataset (SQuAD), which is orders of magnitude larger than all previous hand-annotated datasets and has a variety of qualities that culminate in a natural QA task. SQuAD has the desirable quality that answers are spans in a reference document. This constrains answers to the space of all possible spans. However, Rajpurkar et al. (2016) show that the dataset retains a diverse set of answers and requires different forms of logical reasoning, including multi-sentence reasoning.

We introduce the Dynamic Coattention Network (DCN), illustrated in Fig. 1 , an end-to-end neural network for question answering. The model consists of a coattentive encoder that captures the interactions between the question and the document, as well as a dynamic pointing decoder that alternates between estimating the start and end of the answer span. Our single model obtains an F1 of 75.9% compared to the best published result of 71.0% (Yu et al., 2016) . In addition, our ensemble model obtains an F1 of 80.4% compared to the second best result of 78.1% on the official SQuAD leaderboard. 1 Figure 1 illustrates an overview of the DCN. We first describe the encoders for the document and the question, followed by the coattention mechanism and the dynamic decoder which produces the answer span.

Document encoder:

Question encoder

What plants create most electric power?

Coattention encoder:

The weight of boilers and condensers generally makes the power-to-weight ... However, most electric power is generated using steam turbine plants, so that indirectly the world's industry is ... 2.1 DOCUMENT AND QUESTION ENCODER

Dynamic pointer decoder:

Let (x Q 1 , x Q 2 , . . . , x Q n )

denote the sequence of word vectors corresponding to words in the question and (x D 1 , x D 2 , . . . , x D m ) denote the same for words in the document. Using an LSTM (Hochreiter & Schmidhuber, 1997) , we encode the document as:

d t = LSTM enc d t-1 , x D t .

We define the document encoding matrix as (m+1) . We also add a sentinel vector d ∅ (Merity et al., 2016) , which we later show allows the model to not attend to any particular word in the input.

D = [d 1 . . . d m d ∅ ] ∈ R ×

The question embeddings are computed with the same LSTM to share representation power: q t = LSTM enc q t-1 , x Q t . We define an intermediate question representation (n+1) . To allow for variation between the question encoding space and the document encoding space, we introduce a non-linear projection layer on top of the question encoding. The final representation for the question becomes:

Q = [q 1 . . . q n q ∅ ] ∈ R ×

Q = tanh W (Q) Q + b (Q) ∈ R ×(n+1) .

2.2. COATTENTION ENCODER:

We propose a coattention mechanism that attends to the question and document simultaneously, similar to (Lu et al., 2016) , and finally fuses both attention contexts. Figure 2 provides an illustration of the coattention encoder.

We first compute the affinity matrix, which contains affinity scores corresponding to all pairs of document words and question words: n+1) . The affinity matrix is normalized row-wise to produce the attention weights A Q across the document for each word in the question, and column-wise to produce the attention weights A D across the question for each word in the document:

L = D Q ∈ R (m+1)×(

A Q = softmax (L) ∈ R (m+1)×(n+1) and A D = softmax L ∈ R (n+1)×(m+1)

(1)

Next, we compute the summaries, or attention contexts, of the document in light of each word of the question. n+1) .

C Q = DA Q ∈ R ×(

(2)

A Q A D document product concat product bi-LSTM bi-LSTM bi-LSTM bi-LSTM bi-LSTM concat n+1 m+1 D: Q: C Q C D u t U:

`Figure 2 : Coattention encoder. The affinity matrix L is not shown here. We instead directly show the normalized attention weights A D and A Q .

We similarly compute the summaries QA D of the question in light of each word of the document. Similar to Cui et al. (2016) , we also compute the summaries C Q A D of the previous attention contexts in light of each word of the document. These two operations can be done in parallel, as is shown in Eq. 3. One possible interpretation for the operation C Q A D is the mapping of question encoding into space of document encodings.

EQUATION

We define C D , a co-dependent representation of the question and document, as the coattention context. We use the notation [a; b] for concatenating the vectors a and b horizontally.

The last step is the fusion of temporal information to the coattention context via a bidirectional LSTM:

EQUATION

)

We define U = [u 1 , . . . , u m ] ∈ R 2 ×m , which provides a foundation for selecting which span may be the best possible answer, as the coattention encoding.

2.3. DYNAMIC POINTING DECODER:

Due to the nature of SQuAD, an intuitive method for producing the answer span is by predicting the start and end points of the span (Wang & Jiang, 2016b) . However, given a question-document pair, there may exist several intuitive answer spans within the document, each corresponding to a local maxima. We propose an iterative technique to select an answer span by alternating between predicting the start point and predicting the end point. This iterative procedure allows the model to recover from initial local maxima corresponding to incorrect answer spans.

Figure 3 provides an illustration of the Dynamic Decoder, which is similar to a state machine whose state is maintained by an LSTM-based sequential model. During each iteration, the decoder updates its state taking into account the coattention encoding corresponding to current estimates of the start and end positions, and produces, via a multilayer neural network, new estimates of the start and end positions.

Let h i , s i , and e i denote the hidden state of the LSTM, the estimate of the position, and the estimate of the end position during iteration i. The LSTM state update is then described by Eq. 5.

EQUATION

where u si-1 and u ei-1 are the representations corresponding to the previous estimate of the start and end positions in the coattention encoding U . Given the current hidden state h i , previous start position u si-1 , and previous end position u ei-1 , we estimate the current start position and end position via Eq. 6 and Eq. 7.

EQUATION

EQUATION

where α t and β t represent the start score and end score corresponding to the tth word in the document. We compute α t and β t with separate neural networks. These networks have the same architecture but do not share parameters.

Based on the strong empirical performance of Maxout Networks (Goodfellow et al., 2013) and Highway Networks (Srivastava et al., 2015) , especially with regards to deep architectures, we propose a Highway Maxout Network (HMN) to compute α t as described by Eq. 8. The intuition behind using such model is that the QA task consists of multiple question types and document topics. These variations may require different models to estimate the answer span. Maxout provides a simple and effective way to pool across multiple model variations.

α t = HMN start u t , h i , u si-1 , u ei-1 (8)

Here, u t is the coattention encoding corresponding to the tth word in the document. HMN start is illustrated in Figure 4 . The end score, β t , is computed similarly to the start score α t , but using a separate HMN end .

We now describe the HMN model: where r ∈ R is a non-linear projection of the current state with parameters

HMN u t , h i , u si-1 , u ei-1 = max W (3) m (1) t ; m (2) t + b (3) (9) r = tanh W (D) h i ; u si-1 ; u ei-1 (10) m (1) t = max W (1) [u t ; r] + b (1) (11) m (2) t = max W (2) m (1) t + b (2) (12)

u 48 u 49 u 50 u 51 u 52 MAXOUT MLP u s i 1 u e i 1 h i MAXOUT MAXOUT … … r m (1) m (2) ↵ 49 ↵ 48 ↵ 50 ↵ 51 ↵ 52

EQUATION

t is the output of the first maxout layer with parameters W (1) ∈ R p× ×3 and b (1) ∈ R p× , and m

(2) t is the output of the second maxout layer with parameters W (2) ∈ R p× × and b (2) ∈ R p× . m

(1) t and m

(2) t are fed into the final maxout layer, which has parameters W (3) ∈ R p×1×2 , and b (3) ∈ R p . p is the pooling size of each maxout layer. The max operation computes the maximum value over the first dimension of a tensor. We note that there is highway connection between the output of the first maxout layer and the last maxout layer.

To train the network, we minimize the cumulative softmax cross entropy of the start and end points across all iterations. The iterative procedure halts when both the estimate of the start position and the estimate of the end position no longer change, or when a maximum number of iterations is reached. Details can be found in Section 4.1

3. RELATED WORK:

Statistical QA Traditional approaches to question answering typically involve rule-based algorithms or linear classifiers over hand-engineered feature sets. Richardson et al. (2013) proposed two baselines, one that uses simple lexical features such as a sliding window to match bags of words, and another that uses word-distances between words in the question and in the document. Berant et al. (2014) proposed an alternative approach in which one first learns a structured representation of the entities and relations in the document in the form of a knowledge base, then converts the question to a structured query with which to match the content of the knowledge base. Wang et al. (2015) described a statistical model using frame semantic features as well as syntactic features such as part of speech tags and dependency parses. Chen et al. (2016) proposed a competitive statistical baseline using a variety of carefully crafted lexical, syntactic, and word order features.

Neural QA Neural attention models have been widely applied for machine comprehension or question-answering in NLP. Hermann et al. (2015) proposed an AttentiveReader model with the release of the CNN/Daily Mail cloze-style question answering dataset. Hill et al. (2016) released another dataset steming from the children's book and proposed a window-based memory network. Kadlec et al. (2016) presented a pointer-style attention mechanism but performs only one attention step. Sordoni et al. (2016) introduced an iterative neural attention model and applied it to cloze-style machine comprehension tasks.

Recently, Rajpurkar et al. (2016) released the SQuAD dataset. Different from cloze-style queries, answers include non-entities and longer phrases, and questions are more realistic. For SQuAD, Wang & Jiang (2016b) proposed an end-to-end neural network model that consists of a Match-LSTM encoder, originally introduced in Wang & Jiang (2016a), and a pointer network decoder (Vinyals et al., 2015) ; Yu et al. (2016) introduced a dynamic chunk reader, a neural reading comprehension model that extracts a set of answer candidates of variable lengths from the document and ranks them to answer the question. Lu et al. (2016) proposed a hierarchical co-attention model for visual question answering, which achieved state of the art result on the COCO-VQA dataset (Antol et al., 2015) . In (Lu et al., 2016) , the co-attention mechanism computes a conditional representation of the image given the question, as well as a conditional representation of the question given the image.

Inspired by the above works, we propose a dynamic coattention model (DCN) that consists of a novel coattentive encoder and dynamic decoder. In our model, instead of estimating the start and end positions of the answer span in a single pass (Wang & Jiang, 2016b) start and end positions in a similar fashion to the Iterative Conditional Modes algorithm (Besag, 1986) .

4. EXPERIMENTS:



4.1. IMPLEMENTATION DETAILS:

We train and evaluate our model on the SQuAD dataset. To preprocess the corpus, we use the tokenizer from Stanford CoreNLP (Manning et al., 2014) . We use as GloVe word vectors pretrained on the 840B Common Crawl corpus (Pennington et al., 2014) . We limit the vocabulary to words that are present in the Common Crawl corpus and set embeddings for out-of-vocabulary words to zero. Empirically, we found that training the embeddings consistently led to overfitting and subpar performance, and hence only report results with fixed word embeddings.

We use a max sequence length of 600 during training and a hidden state size of 200 for all recurrent units, maxout layers, and linear layers. All LSTMs have randomly initialized parameters and an initial state of zero. Sentinel vectors are randomly initialized and optimized during training. For the dynamic decoder, we set the maximum number of iterations to 4 and use a maxout pool size of 16. We use dropout to regularize our network during training (Srivastava et al., 2014) , and optimize the model using ADAM (Kingma & Ba, 2014) . All models are implemented and trained with Chainer (Tokui et al., 2015) .

4.2. RESULTS:

Evaluation on the SQuAD dataset consists of two metrics. The exact match score (EM) calculates the exact string match between the predicted answer and a ground truth answer. The F1 score calculates the overlap between words in the predicted answer and a ground truth answer. Because a document-question pair may have several ground truth answers, the EM and F1 for a documentquestion pair is taken to be the maximum value across all ground truth answers. The overall metric is then computed by averaging over all document-question pairs. The offical SQuAD evaluation is hosted on CodaLab2 . The training and development sets are publicly available while the test set is withheld.

The performance of the Dynamic Coattention Network on the SQuAD dataset, compared to other submitted models on the leaderboard3 , is shown in Table 1 . At the time of writing, our singlemodel DCN ranks first at 66.2% exact match and 75.9% F1 on the test data among single-model submissions. Our ensemble DCN ranks first overall at 71.6% exact match and 80.4% F1 on the test data.

The DCN has the capability to estimate the start and end points of the answer span multiple times, each time conditioned on its previous estimates. By doing so, the model is able to explore local maxima corresponding to multiple plausible answers, as is shown in Figure 5 . While the dynamic nature of the decoder allows the model to escape initial local maxima corresponding to incorrect answers, Question 3 demonstrates a case where the model is unable to decide between multiple local maxima despite several iterations. Namely, the model alternates between the answers "charged particle beam" and "particle beam weapons" indefinitely. Empirically, we observe that the model, trained with a maximum iteration of 4, takes 2.7 iterations to converge to an answer on average. 

Model:



Model Ablation:

The performance of our model and its ablations on the SQuAD development set is shown in Table 2. On the decoder side, we experiment with various pool sizes for the HMN maxout layers, using a 2-layer MLP instead of a HMN, and forcing the HMN decoder to a single iteration. Empirically, we achieve the best performance on the development set with an iterative HMN with pool size 16, and find that the model consistently benefits from a deeper, iterative decoder network. The performance improves as the number of maximum allowed iterations increases, with little improvement after 4 iterations. On the encoder side, replacing the coattention mechanism with an attention mechanism similar to Wang & Jiang (2016b) by setting C D to QA D in equation 3 results in a 1.9 point F1 drop. This suggests that, at an additional cost of a softmax computation and a dot product, the coattention mechanism provides a simple and effective means to better encode the document and question sequences. Further studies, such as performance without attention and performance on questions requiring different types of reasoning can be found in the appendix. Performance across length One point of interest is how the performance of the DCN varies with respect to the length of document. Intuitively, we expect the model performance to deteriorate with longer examples, as is the case with neural machine translation (Luong et al., 2015) . However, as in shown in Figure 6 , there is no notable performance degradation for longer documents and questions contrary to our expectations. This suggests that the coattentive encoder is largely agnostic to long documents, and is able to focus on small sections of relevant text while ignoring the rest of the (potentially very long) document. We do note a performance degradation with longer answers. However, this is intuitive given the nature of the evaluation metric. Namely, it becomes increas-ingly challenging to compute the correct word span as the number of words increases.

Performance across question type Another natural way to analyze the performance of the model is to examine its performance across question types. In Figure 7 , we note that the mean F1 of DCN exceeds those of previous systems (Wang & Jiang, 2016b; Yu et al., 2016) across all question types. The DCN, like other models, is adept at "when" questions and struggles with the more complex "why" questions.

Breakdown of F1 distribution Finally, we note that the DCN performance is highly bimodal. On the development set, the model perfectly predicts (100% F1) an answer for 62.2% of examples and predicts a completely wrong answer (0% F1) for 16.3% of examples. That is, the model picks out partial answers only 21.5% of the time. Upon qualitative inspections of the 0% F1 answers, some of which are shown in Appendix A.4, we observe that when the model is wrong, its mistakes tend to have the correct "answer type" (eg. person for a "who" question, method for a "how" question) and the answer boundaries encapsulate a well-defined phrase.

5. CONCLUSION:

We proposed the Dynamic Coattention Network, an end-to-end neural network architecture for question answering. The DCN consists of a coattention encoder which learns co-dependent representations of the question and of the document, and a dynamic decoder which iteratively estimates the answer span. We showed that the iterative nature of the model allows it to recover from initial local maxima corresponding to incorrect predictions. On the SQuAD dataset, the DCN achieves the state of the art results at 75.9% F1 with a single model and 80.4% F1 with an ensemble. The DCN significantly outperforms all other models.

Prediction UK's biggest national collection of material about live performance in the UK since Shakespeare's day

WHAT IS THE MAIN GOAL OF CRIMINAL PUNISHMENT OF CIVIL DISOBEDIENTS?:

Type of reasoning Ambiguous

Along with giving the offender his "just deserts", achieving crime control via incapacitation and deterrence is a major goal of crime punishment.

Ground truth achieving crime control via incapacitation and deterrence:

Prediction achieving crime control via incapacitation and deterrence 

ID 5730d473b7151e1900c0155b:

Elders are called by God, affirmed by the church, and ordained by a bishop to a ministry of Word, Sacrament, Order and Service within the church. They may be appointed to the local church, or to other valid extension ministries of the church. Elders are given the authority to preach the Word of God, administer the sacraments of the church, to provide care and counseling, and to order the life of the church for ministry and mission. Elders may also be assigned as District Superintendents, and they are eligible for election to the episcopacy. Elders serve a term of 23 years as provisional Elders prior to their ordination.

Ground truth bishop, the local church:

Prediction a bishop AN ALGORITHM FOR X WHICH REDUCES TO C WOULD ALLOW US TO DO WHAT?

ID 56e1ce08e3433e14004231a6:

This motivates the concept of a problem being hard for a complexity class. A problem X is hard for a class of problems C if every problem in C can be reduced to X. Thus no problem in C is harder than X, since an algorithm for X allows us to solve any problem in C. Of course, the notion of hard problems depends on the type of reduction being used. For complexity classes larger than P, polynomial-time reductions are commonly used. In particular, the set of problems that are hard for NP is the set of NP-hard problems.

Ground truth solve any problem in C:

Prediction solve any problem in C HOW MANY GENERAL QUESTIONS ARE AVAILABLE TO OPPOSITION LEADERS?

ID 572fd7b8947a6a140053cd3e:

Parliamentary time is also set aside for question periods in the debating chamber. A "General Question Time" takes place on a Thursday between 11:40 a.m. and 12 p.m. where members can direct questions to any member of the Scottish Government. At 2.30pm, a 40-minute long themed "Question Time" takes place, where members can ask questions of ministers in departments that are selected for questioning that sitting day, such as health and justice or education and transport. Between 12 p.m. and 12:30 p.m. on Thursdays, when Parliament is sitting, First Minister's Question Time takes place. This gives members an opportunity to question the First Minister directly on issues under their jurisdiction. Opposition leaders ask a general question of the First Minister and then supplementary questions. Such a practice enables a "lead-in" to the questioner, who then uses their supplementary question to ask the First Minister any issue. The four general questions available to opposition leaders are:

Ground truth four:

Prediction four WHAT ARE SOME OF THE ACCEPTED GENERAL PRINCIPLES OF EUROPEAN UNION LAW?

ID 5726a00cf1498d1400e8e551:

The principles of European Union law are rules of law which have been developed by the European Court of Justice that constitute unwritten rules which are not expressly provided for in the treaties but which affect how European Union law is interpreted and applies. In formulating these principles, the courts have drawn on a variety of sources, including: public international law and legal doctrines and principles present in the legal systems of European Union member states and in the jurisprudence of the European Court of Human Rights. Accepted general principles of European Union Law include fundamental rights (see human rights), proportionality, legal certainty, equality before the law and subsidiarity.

Ground truth fundamental rights (see human rights), proportionality, legal certainty, equality before the law and subsidiarity

Prediction fundamental rights (see human rights), proportionality, legal certainty, equality before the law and subsidiarity WHY WAS TESLA RETURNED TO GOSPIC?

ID 56dfaa047aa994140058dfbd:

On 24 March 1879, Tesla was returned to Gospi under police guard for not having a residence permit. On 17 April 1879, Milutin Tesla died at the age of 60 after contracting an unspecified illness (although some sources say that he died of a stroke). During that year, Tesla taught a large class of students in his old school, Higher Real Gymnasium, in Gospi.

Ground truth not having a residence permit:

Prediction not having a residence permit 

ID 56e0d6cf231d4119001ac424:

After leaving Edison's company Tesla partnered with two businessmen in 1886, Robert Lane and Benjamin Vail, who agreed to finance an electric lighting company in Tesla's name, Tesla Electric Light & Manufacturing. The company installed electrical arc light based illumination systems designed by Tesla and also had designs for dynamo electric machine commutators, the first patents issued to Tesla in the US.

Ground truth Tesla:

Prediction Robert Lane and Benjamin Vail

Comment The model produces an incorrect prediction that corresponds to people that funded Tesla, instead of Tesla who actually designed the illumination system. Empirically, we find that most mistakes made by the model have the correct type (eg. named entity type) despite not including types as prior knowledge to the model. In this case, the incorrect response has the correct type of person.

CYDIPPID ARE TYPICALLY WHAT SHAPE?

ID 57265746dd62a815002e821a:

Cydippid ctenophores have bodies that are more or less rounded, sometimes nearly spherical and other times more cylindrical or egg-shaped; the common coastal "sea gooseberry," Pleurobrachia, sometimes has an egg-shaped body with the mouth at the narrow end, although some individuals are more uniformly round. From opposite sides of the body extends a pair of long, slender tentacles, each housed in a sheath into which it can be withdrawn. Some species of cydippids have bodies that are flattened to various extents, so that they are wider in the plane of the tentacles.

Ground truth more or less rounded, egg-shaped

Prediction spherical:

Comment Although the mistake is subtle, the prediction is incorrect. The statement "are more or less rounded, sometimes nearly spherical" suggests that the entity is more often "rounded" than "spherical" or "cylindrical" or "egg-shaped" (an answer given by an annotator). This suggests that the model has trouble discerning among multiple intuitive answers due to a lack of understanding of the relative severity of "more or less" versus "sometimes" and "other times".

:

https://worksheets.codalab.org

https://rajpurkar.github.io/SQuAD-explorer

Figures and Tables:

Figure 1: Overview of the Dynamic Coattention Network.

Figure 3: Dynamic Decoder. Blue denotes the variables and functions related to estimating the start position whereas red denotes the variables and functions related to estimating the end position.

Figure 4: Highway Maxout Network. Dotted lines denote highway connections.

Figure 5: Examples of the start and end conditional distributions produced by the dynamic decoder. Odd (blue) rows denote the start distributions and even (red) rows denote the end distributions. i indicates the iteration number of the dynamic decoder. Higher probability mass is indicated by darker regions. The offset corresponding to the word with the highest probability mass is shown on the right hand side. The predicted span is underlined in red, and a ground truth answer span is underlined in green.

Figure 6: Performance of the DCN for various lengths of documents, questions, and answers. The blue dot indicates the mean F1 at given length. The vertical bar represents the standard deviation of F1s at a given length.

Figure 7: Performance of the DCN across question types. The height of each bar represents the mean F1 for the given question type. The lower number denotes how many instances in the dev set are of the corresponding question type.

, we iteratively update the Leaderboard performance at the time of writing (Nov 4 2016). * indicates that the model used for submission is unpublished.indicates that the development scores were not publicly available at the time of writing.

Table content: <table><tr><td>Model</td><td colspan="4">Dev EM Dev F1 Test EM Test F1</td></tr><tr><td>Ensemble</td><td/><td/><td/><td/></tr><tr><td>DCN (Ours)</td><td>70.3</td><td>79.4</td><td>71.2</td><td>80.4</td></tr><tr><td>Microsoft Research Asia  *</td><td>-</td><td>-</td><td>69.4</td><td>78.3</td></tr><tr><td>Allen Institute  *</td><td>69.2</td><td>77.8</td><td>69.9</td><td>78.1</td></tr><tr><td>Singapore Management University  *</td><td>67.6</td><td>76.8</td><td>67.9</td><td>77.0</td></tr><tr><td>Google NYC  *</td><td>68.2</td><td>76.7</td><td>-</td><td>-</td></tr><tr><td>Single model</td><td/><td/><td/><td/></tr><tr><td>DCN (Ours)</td><td>65.4</td><td>75.6</td><td>66.2</td><td>75.9</td></tr><tr><td>Microsoft Research Asia  *</td><td>65.9</td><td>75.2</td><td>65.5</td><td>75.0</td></tr><tr><td>Google NYC  *</td><td>66.4</td><td>74.9</td><td>-</td><td>-</td></tr><tr><td>Singapore Management University  *</td><td>-</td><td>-</td><td>64.7</td><td>73.7</td></tr><tr><td>Carnegie Mellon University  *</td><td>-</td><td>-</td><td>62.5</td><td>73.3</td></tr><tr><td>Dynamic Chunk Reader (Yu et al., 2016)</td><td>62.5</td><td>71.2</td><td>62.5</td><td>71.0</td></tr><tr><td>Match-LSTM (Wang &amp; Jiang, 2016b)</td><td>59.1</td><td>70.0</td><td>59.5</td><td>70.3</td></tr><tr><td>Baseline (Rajpurkar et al., 2016)</td><td>40.0</td><td>51.0</td><td>40.4</td><td>51.0</td></tr><tr><td>Human (Rajpurkar et al., 2016)</td><td>81.4</td><td>91.0</td><td>82.3</td><td>91.2</td></tr></table>

Single model ablations on the development set.

Table content: <table><tr><td>Dev EM Dev F1</td></tr></table>

A.3 SAMPLES OF CORRECT SQUAD PREDICTIONS BY THE DYNAMIC COATTENTION NETWORK HOW DID THE MONGOLS ACQUIRE CHINESE PRINTING TECHNOLOGY? ID: 572882242ca10214002da420 The Mongol rulers patronized the Yuan printing industry. Chinese printing technology was transferred to the Mongols through Kingdom of Qocho and Tibetan intermediaries. Some Yuan documents such as Wang Zhen's Nong Shu were printed with earthenware movable type, a technology invented in the 12th century. However, most published works were still produced through traditional block printing techniques. The publication of a Taoist text inscribed with the name of Tregene Khatun, gedei's wife, is one of the first printed works sponsored by the Mongols. In 1273, the Mongols created the Imperial Library Directorate, a government-sponsored printing office. The Yuan government established centers for printing throughout China. Local schools and government agencies were funded to support the publishing of books.

Table content: <table><tr><td>Ground truth through Kingdom of Qocho and Tibetan intermediaries</td></tr><tr><td>Prediction: through Kingdom of Qocho and Tibetan intermediaries</td></tr><tr><td>WHO APPOINTS ELDERS?</td></tr></table>

A.4 SAMPLES OF INCORRECT SQUAD PREDICTIONS BY THE DYNAMIC COATTENTION NETWORK WHAT IS ONE SUPPLEMENTARY SOURCE OF EUROPEAN UNION LAW? ID 5725c3a9ec44d21400f3d506 European Union law is applied by the courts of member states and the Court of Justice of the European Union. Where the laws of member states provide for lesser rights European Union law can be enforced by the courts of member states. In case of European Union law which should have been transposed into the laws of member states, such as Directives, the European Commission can take proceedings against the member state under the Treaty on the Functioning of the European Union. The European Court of Justice is the highest court able to interpret European Union law. Supplementary sources of European Union law include case law by the Court of Justice, international law and general principles of European Union law. Prediction case law by the Court of Justice Comment The prediction produced by the model is correct, however it was not selected by Mechanical Turk annotators. WHO DESIGNED THE ILLUMINATION SYSTEMS THAT TESLA ELECTRIC LIGHT & MANUFACTURING INSTALLED?

Table content: <table><tr><td>Ground truth international law</td></tr></table>

Peer review (iclr2017):
Summary: The paper proposes a novel deep neural network architecture for the task of question answering on the SQuAD dataset. The model consists of two main components -- coattention encoder and dynamic pointer decoder. The encoder produces attention over the question as well as over the document in parallel and thus learns co-dependent representations of the question and the document. The decoder predicts the starting and the end token of the answer iteratively, with the motivation that multiple iterations will help the model escape local maxima and thus will reduce the errors made by the model. The proposed model achieved state-of-art result on SQuAD dataset at the time of writing the paper. The paper reports some analyses of the results such as performance across question types, document, question, answer lengths, etc. The paper also performs some ablation studies such as performing only single round of iteration on decoder, etc. Strengths: 1. The paper is well-motivated with two main motivations -- co-attending to the document and the question, and iteratively producing the answer. 2. The proposed model architecture is novel and the design choices made seem reasonable. 3. The experiments show that the proposed model outperforms the existing model (at the time of writing the paper) on the SQuAD dataset by significant margin. 4. The analyses of the results and the ablation studies performed (as per someone*s request) provide insights into the various modelling design choices made. Weaknesses/Questions/Suggestions: 1. In order to gain insights into how much each additional iteration in the decoder help, I would like to see the following -- for every iteration report the mean F1 for questions that converged in that iteration along with the number of questions that converged in that iteration. 2. Example of Question 3 in figure 5 is an interesting example where the model is unable to decide between multiple local maxima despite several iterations. Could authors please report how often this happens? 3. In order to estimate how much modelling of attention in the encoder helps, it would be good if authors could report the performance of the model when attention is not modeled at all in the encoder (neither over question, nor over document). 4. I would like to see the variation in the performance of the proposed model for questions that require different types of reasoning (table 3 in SQuAD paper). This would provide insights into what are the strengths and weaknesses of the proposed model w.r.t the type reasoning required. 5. In Wang and Jiang (2016), the attention is predicted over question for each word in the document. But in table 2, when performing ablation study to make the proposed model similar to Wang and Jiang, C^D is set to C^Q. But isn’t C^Q attention over document for each word in the question? So, how is this similar to Wang and Jiang’s attention? I think QA^D will be similar to Wang and Jiang*s attention since QA^D is attention over question for each word in the document. Please clarify. 6. In section 2.1, “n” and “m” are swapped when explaining the Document and Question encoding matrix. Please fix it. Review Summary: The paper presents a novel and interesting model for the task of question answering on SQuAD dataset and shows that the model outperforms existing models. However, to gain more insights into the functioning of the model, I would like see more analyses of the results and one more ablation study (see weaknesses section above).