Venue: EMNLP 2017

# Globally Normalized Reader

Jonathan Raiman and John Miller

Baidu Silicon Valley Artificial Intelligence Lab

{jonathanraiman,millerjohn}@baidu.com

###### Abstract

Rapid progress has been made towards question answering (QA) systems that can extract answers from text. Existing neural approaches make use of expensive bi-directional attention mechanisms or score all possible answer spans, limiting scalability. We propose instead to cast extractive QA as an iterative search problem: select the answer's sentence, start word, and end word. This representation reduces the space of each search step and allows computation to be conditionally allocated to promising search paths. We show that globally normalizing the decision process and back-propagating through beam search makes this representation viable and learning efficient. We empirically demonstrate the benefits of this approach using our model, Globally Normalized Reader (GNR), which achieves the second highest single model performance on the Stanford Question Answering Dataset (68.4 EM, 76.21 F1 dev) and is 24.7x faster than bi-attention-flow. We also introduce a data-augmentation method to produce semantically valid examples by aligning named entities to a knowledge base and swapping them with new entities of the same type. This method improves the performance of all models considered in this work and is of independent interest for a variety of NLP tasks.

## 1 Introduction

Question answering (QA) and information extraction systems have proven to be invaluable in wide variety of applications such as medical information collection on drugs and genes Quirk and Poon (2016), large scale health impact studies Althoff et al. (2016), or educational material development Koedinger et al. (2015). Recent progress in neural-network based extractive question answering models are quickly closing the gap with human performance on several benchmark QA tasks such as SQuAD Rajpurkar et al. (2016), MS MARCO Nguyen et al. (2016), or NewsQA Trischler et al. (2016a). However, current approaches to extractive question answering face several limitations:

Figure 1: GNR answering a question. It first picks a sentence, then start word, then end word. Probabilities are global and normalized over the beam. Model initially picks the wrong sentence, but global normalization lets it recover. Final prediction’s probability (0.64) exceeds sentence pick (0.49), whereas with local normalization each probability is upper bounded by the previous step.



1. Computation is allocated equally to the entire document, regardless of answer location, with no ability to ignore or focus computation on specific parts. This limits applicability to longer documents.
2. They rely extensively on expensive bi-directional attention mechanisms Seo et al. (2016) or must rank all possible answer spans Lee et al. (2016).
3. While data-augmentation for question answering have been proposed Zhou et al. (2017), current approaches still do not provide training data that can _improve_ the performance of existing systems.

In this paper we demonstrate a methodology for addressing these three limitations, and make the following claims:

1. Extractive Question Answering can be cast as a nested search process, where sentences provide a powerful document decomposition and an easy to learn search step. This factorization enables conditional computation to be allocated to sentences and spans likely to contain the right answer.
2. When cast as a search process, models without bi-directional attention mechanisms and without ranking all possible answer spans can achieve near state of the art results on extractive question answering.
3. Preserving narrative structure and explicitly incorporating type and question information into synthetic data generation is key to generating examples that actually improve the performance of question answering systems.

Our claims are supported by experiments on the SQuAD dataset where we show that the Globally Normalized Reader (GNR), a model that performs an iterative search process through a document (shown visually in Figure 1), and has computation conditionally allocated based on the search process, achieves near state of the art Exact Match (EM) and F1 scores without resorting to more expensive attention or ranking of all possible spans. Furthermore, we demonstrate that Type Swaps, a type-aware data augmentation strategy that aligns named entities with a knowledge base and swaps them out for new entities that share the same type, improves the performance of all models on extractive question answering.

We structure the paper as follows: in Section 2 we introduce the task and our model. Section 3 describes our data-augmentation strategy. Section 4 introduces our experiments and results. In Section 5 we discuss our findings. In Section 6 we relate our work to existing approaches. Conclusions and directions for future work are given in Section 7.

## 2 Model

Given a document \(d\) and a question \(q\), we pose extractive question answering as a search problem. First, we select the sentence, the first word of the span, and finally the last word of the span. A example of the output of the model is shown in Figure 1, and the network architecture is depicted in Figure 2.

More formally, let \(d_{1},\ldots,d_{n}\) denote each sentence in the document, and for each sentence \(d_{i}\), let \(d_{i,1},\ldots,d_{i,m_{i}}\) denote the word vectors corresponding to the words in the sentence. Similarly, let \(q_{1},\ldots,q_{\ell}\) denote the word vectors corresponding to words in the question. An answer is a tuple \(a=(i^{*},j^{*},k^{*})\) indicating the correct sentence \(i^{*}\), start word in the sentence \(j^{*}\) and end word in the sentence \(k^{*}\). Let \(\mathcal{A}(d)\) denote the set of valid answer tuples for document \(d\). We now describe each stage of the model in turn.

### Question Encoding

Each question is encoded by running a stack of bidirectional LSTM (Bi-LSTM) over each word in the question, producing hidden states \((h_{1}^{\text{fwd}},h_{1}^{\text{wwd}}),\dots,(h_{\ell}^{\text{fwd}},h_{\ell} ^{\text{bwd}})\)Graves and Schmidhuber (2005). Following Lee et al. (2016), these hidden states are used to compute a _passage-independent question embedding_, \(q^{\text{indep}}\). Formally,

\[s_{j} =w_{q}^{\top}\text{MLP}([h_{j}^{\text{bwd}};h_{j}^{\text{fwd}}]) \tag{1}\] \[\alpha_{j} =\frac{\exp(s_{j})}{\sum_{j^{\prime}=1}^{\ell}\exp(s_{j^{\prime} })}\] (2) \[q^{\text{indep}} =\sum_{j=1}^{\ell}\alpha_{j}[h_{j}^{\text{bwd}};h_{j}^{\text{fwd}}], \tag{3}\]

where \(w_{q}\) is a trainable embedding vector, and MLP is a two-layer neural network with a \(\operatorname{Relu}\) non-linearity. The question is represented by concatenating the final hidden states of the for 

ward and backward LSTMs and the passage-independent embedding, \(q=[h_{1}^{\text{bwd}};h_{\ell}^{\text{fwd}};q^{\text{indeplcp}}]\).

### Question-Aware Document Encoding
Conditioned on the question vector, we compute a representation of each document word that is sensitive to both the surrounding context and the question. Specifically, each word in the document is represented as the concatenation of its word vector \(d_{i,j}\), the question vector \(q\), boolean features indicating if a word appears in the question or is repeated, and a _question-aligned embedding_ from Lee et al. (2016). The question-aligned embedding \(q_{i,j}^{\text{align}}\) is given by

\[s_{i,j,k} =\text{MLP}(d_{i,j})^{\top}\text{MLP}(q_{k}) \tag{4}\] \[\alpha_{i,j,k} =\frac{\exp(s_{i,j,k})}{\sum_{k^{\prime}=1}^{\ell}\exp(s_{i,j,k^{ \prime}})}\] (5) \[q_{i,j}^{\text{align}} =\sum_{k=1}^{\ell}\alpha_{i,j,k}q_{k}. \tag{6}\]

The document is encoded by a separate stack of Bi-LSTMs, producing a sequence of hidden states \((h_{1,1}^{\text{fwd}},h_{1,1}^{\text{bwd}}),\ldots,(h_{n,m_{u}}^{\text{fwd}} ,h_{n,m_{u}}^{\text{bwd}})\). The search procedure then operates on these hidden states.

### Answer Selection

Sentence selection.The first phase of our search process picks the sentence that contains the answer span. Each sentence \(d_{i}\) is represented by the hidden state of the first and last word in the sentence for the backward and forward LSTM respectively, \([h_{i,1}^{\text{bwd}};h_{i,m_{i}}^{\text{fwd}}]\), and is scored by passing this representation through a fully connected layer that outputs the unnormalized sentence score for sentence \(d_{i}\), denoted \(\phi_{\mathrm{sent}}(d_{i})\).

Span start selection.After selecting a sentence \(d_{i}\), we pick the start of the answer span within the sentence. Each potential start word \(d_{i,j}\) is represented as its corresponding document encoding \([h_{i,j}^{\text{fwd}};h_{i,j}^{\text{bwd}}]\), and is scored by passing this encoding through a fully connected layer that outputs the unnormalized start word score for word \(j\) in sentence \(i\), denoted \(\phi_{\mathrm{sw}}(d_{i,j})\).

Span end selection.Conditioned on sentence \(d_{i}\) and starting word \(d_{i,j}\), we select the end word from the remaining words in the sentence \(d_{i,j},\dots,d_{i,m_{i}}\). To do this, we run a BiLSTM over the remaining document hidden states \((h_{i,j}^{\text{fwd}},h_{i,j}^{\text{bwd}}),\ldots,(h_{i,m_{i}}^{\text{fwd}} ,h_{i,m_{i}}^{\text{out}})\) to produce representations \((\tilde{h}_{i,j}^{\text{fwd}},\tilde{h}_{i,j}^{\text{bwd}}),\ldots,(\tilde{ h}_{i,m_{i}}^{\text{fwd}},\tilde{h}_{i,m_{i}}^{\text{bwd}})\). Each end word \(d_{i,k}\) is then scored by passing \([\tilde{h}_{i,k}^{\text{fwd}};\tilde{h}_{i,k}^{\text{bwd}}]\) through a fully connected layer that outputs the unnormalized end word score for word \(k\) in sentence \(i\), with start word \(j\), denoted \(\phi_{\mathrm{ew}}(d_{i,j:k})\).

### Global Normalization

The scores for each stage of our model can be normalized at the local or global level. Previous work demonstrated that locally-normalized models have a weak ability to correct mistakes made in previous decisions, while globally normalized models are strictly more expressive than locally normalized models Andor et al. (2016); Zhou et al. (2015); Collins and Roark (2004).

In a locally normalized model each decision is made conditional on the previous decision. The probability of some answer \(a=(i,j,k)\) is decomposed as

\[\begin{split}\mathbb{P}(a|d,q)=&\mathbb{P}_{ \mathrm{sent}}(i|d,q)\cdot\mathbb{P}_{\mathrm{sw}}(j|i,d,q)\cdot\\ &\mathbb{P}_{\mathrm{ew}}(k|j,i,d,q).\end{split} \tag{7}\]

Each sub-decision is locally normalized by applying a softmax to the relevant selection scores:

\[\mathbb{P}_{\mathrm{sent}}(i|d,q)=\frac{\exp(\phi_{\mathrm{sent}}(d_{i}))}{ \sum_{x=1}^{n}\exp(\phi_{\mathrm{sent}}(d_{x}))}, \tag{8}\]

\[\mathbb{P}_{\mathrm{sw}}(j|i,d,q)=\frac{\exp(\phi_{\mathrm{sw}}(d_{i,j}))}{ \sum_{x=1}^{m_{i}}\exp(\phi_{\mathrm{sw}}(d_{i,x}))}, \tag{9}\]

\[\mathbb{P}_{\mathrm{ew}}(k|j,i,d,q)=\frac{\exp(\phi_{\mathrm{ew}}(d_{i,j:k}))}{ \sum_{x=j}^{m_{i}}\exp(\phi_{\mathrm{ew}}(d_{i,j:x}))}. \tag{10}\]

To allow our model to recover from incorrect sentence or start word selections, we instead globally normalize the scores from each stage of our procedure. In a globally normalized model, we define

\[\text{score}(a,d,q)=\phi_{\mathrm{sent}}(d_{i})+\phi_{\mathrm{sw}}(d_{i,j})+ \phi_{\mathrm{ew}}(d_{i,j:k}). \tag{11}\]

Then, we model

\[\mathbb{P}(a~{}|~{}d,q)=\frac{\exp(\text{score}(a,d,q))}{Z}, \tag{12}\]

where \(Z\) is the partition function

\[Z=\sum_{a^{\prime}\in\mathcal{A}(d)}\exp(\text{score}(a^{\prime},d,q)).\] (13) 

In contrast to locally-normalized models, the model is normalized over all possible search paths instead of normalizing each step of search procedure. At inference time, the problem is to find

\[\arg\max_{a\in\mathcal{A}(d)}\mathbb{P}(a~{}|~{}d,q), \tag{14}\]

which can be approximately computed using beam search.

### Objective and Training

We minimize the negative log-likelihood on the training set using stochastic gradient descent. For a single example \((a,d,q)\), the negative log-likelihood

\[-\text{score}(a,d,q)+\log Z \tag{15}\]

requires an expensive summation to compute \(\log Z\). Instead, to ensure learning is efficient, we use beam search during training and early updates Andor et al. (2016); Zhou et al. (2015); Collins and Roark (2004). Concretely, we approximate \(Z\) by summing only over candidates on the final beam \(\mathcal{B}\):

\[Z\approx\sum_{a^{\prime}\in\mathcal{B}}\exp(\text{score}(a^{\prime},d,q)). \tag{16}\]

At training time, if the gold sequence falls off the beam at step \(t\) during decoding, a stochastic gradient step is performed on the partial objective computed through step \(t\) and normalized over the beam at step \(t\).

### Implementation

Our best performing model uses a stack of 3 Bi-LSTMs for the question and document encodings, and a single Bi-LSTM for the end of span prediction. The hidden dimension of all recurrent layers is 200.

We use the 300 dimensional 8.4B token Common Crawl GloVe vectors Pennington et al. (2014). Words missing from the Common Crawl vocabulary are set to zero. In our experiments, all architectures considered have sufficient capacity to overfit the training set. We regularize the models by fixing the word embeddings throughout training, dropping out the inputs of the Bi-LSTMs with probability 0.3 and the inputs to the fully-connected layers with probability 0.4 Srivastava et al. (2014), and adding gaussian noise to the recurrent weights with \(\sigma=10^{-6}\). Our models are trained using Adam with a learning rate of 0.0005, \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), \(\epsilon=10^{-8}\) and a batch size of 32 Kingma and Ba (2014).

All our experiments are implemented in Tensorflow Abadi et al. (2016), and we tokenize using Ciseau Raiman (2017). Despite performing beam-search during training, our model trains to convergence in under 4 hours through the use of efficient LSTM primitives in CuDNN Chetlur et al. (2014) and batching our computation over examples and search beams. We release our code and augmented dataset.1

Footnote 1: [https://github.com/baidu-research/GloballyNormalizedReader](https://github.com/baidu-research/GloballyNormalizedReader)

Figure 2: Globally Normalized Reader’s search process. Same color Bi-LSTMs share weights.

 

Our implementation of the GNR is 24.7 times faster at inference time than the official Bi-Directional Attention Flow implementation2. Specifically, on a machine running Ubuntu 14 with 40 Intel Xeon 2.6Ghz processors, 386GB of RAM, and a 12GB TitanX-Maxwell GPU, the GNR with beam size 32 and batch size 32 requires \(51.58\pm 0.266\) seconds (mean \(\pm\) std)3 to process the SQUAD validation set. By contrast, the Bi-Directional Attention Flow model with batch size 32 requires \(1260.23\pm 17.26\) seconds. We attribute this speedup to avoiding expensive bi-directional attention mechanisms and making computation conditional on the search beams.

Footnote 2: [https://github.com/allenai/bi-att-flow](https://github.com/allenai/bi-att-flow)

Footnote 3: All numbers are averaged over 5 runs.

## 3 Type Swaps

**Question**: Who said in April 25, 2011 that the fight would change from military to law enforcement?

**Answer**: Sheryl Sandberg

**Document (snippet)**: ...Basic objectives of the Cabinet of Japan "war on terror", such as Bush administration targeting al Qaeda and building international counterterrorism alliances, remain in place. In April 25, 2011; Sheryl Sandberg, the General Deb 

with documents where we can safely assume that the majority of the entities will be contained in a large knowledge base (KB) such as Wikidata Vrandecic and Krotzsch (2014) we find that simple string matching techniques are sufficiently accurate. Specifically, we use a part of speech tagger Honnibal (2017) to extract nominal groups in the training data and string-match them with entities in Wikidata. Using this technique, we are able to extract 47,598 entities in SQuAD that fall under 6,380 Wikidata instance of4 types. Additionally we assign "number types" (e.g. _year_, _day of the week_, _distance_, etc.) to nominal groups that contain dates, numbers, or quantities5. These extraction steps produce 84,632 unique surface variants (on average 16.93 per type) with the majority of the variation found in humans, numbers or organizations as visible in Figure 4.

Footnote 4: [https://www.wikidata.org/wiki/Property:P3l](https://www.wikidata.org/wiki/Property:P3l)

With this method, we can generate \(2.92\cdot 10^{369}\) unique documents (average of \(3.36\cdot 10^{364}\) new documents for each original document). To ensure there is sufficient variation in the generated documents, we sample from this set and only keep variations where the question or answer is mutated. At each training epoch, we train on \(T\) Type Swap examples and the full original training data. An example output of the method is shown in Figure 3.

## 4 Results

We evaluate our model on the 100,000 example SQuAD dataset Rajpurkar et al. (2016) and perform several ablations to evaluate the relative importance of the proposed methods.

### Learning to Search

In our first experiment, we aim to quantify the importance of global normalization on the learning and search process. We use \(T=10^{4}\) Type Swap samples and vary beam width \(B\) between 1 and 32 for a locally and globally normalized models and summarize the Exact-Match and F1 score of the model's predicted answer and ground truth computed using the evaluation scripts from Rajpurkar et al. (2016) (Table 3). We additionally report another metric, the _Sentence_ score, which is a measure for how often the predicted answer came from the correct sentence. This metric provides a measure for where mistakes are made during prediction.

### Type Swaps

In our second experiment, we evaluate the impact of the amount of augmented data on the performance of our model. In this experiment, we use

\begin{table}
\begin{tabular}{|r|r|r|r|r|r|r|} \hline Cluster & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ Size & 84789 & 42187 & 53061 & 130022 & 27549 & 16894 & 28377 \\ \hline Bigram & \multicolumn{6}{c|}{Bigram Occurences} \\ \hline _what is_ & 3339 & 520 & 87 & 3736 & 20 & 8 & 138 \\ _what did_ & 2463 & 3 & 3 & 112 & 1 & 0 & 1 \\ _how many_ & 2 & 5095 & 1 & 1 & 0 & 0 & 0 \\ _how much_ & 7 & 1102 & 0 & 12 & 0 & 0 & 0 \\ _who was_ & 2 & 0 & 1934 & 0 & 0 & 0 & 1 \\ _who did_ & 2 & 0 & 683 & 2 & 0 & 0 & 0 \\ _what was_ & 2177 & 508 & 105 & 2034 & 71 & 31 & 92 \\ _when did_ & 0 & 0 & 0 & 1 & 2772 & 0 & 0 \\ _when was_ & 0 & 0 & 1 & 1 & 1876 & 0 & 0 \\ _what year_ & 0 & 0 & 0 & 1 & 13 & 2690 & 0 \\ _in what_ & 52 & 3 & 9 & 727 & 110 & 1827 & 518 \\ _where did_ & 0 & 0 & 0 & 13 & 1 & 0 & 955 \\ _where is_ & 0 & 1 & 0 & 11 & 0 & 0 & 665 \\ \hline \end{tabular}
\end{table}
Table 1: Top bigrams in K-means (\(K=7\)) clusters of question after Bi-LSTM. We observe emergent clustering according to question type: e.g. _Where_\(\rightarrow\) Cluster 7, _Who_\(\rightarrow\) Cluster 3. “What” granularity only observable with more clusters.

 

the best beam sizes for each model (\(B=10\) for local and \(B=32\) for global) and vary the augmentation from \(T=0\) (no augmentation) to \(T=5\cdot 10^{4}\). The results of this experiment are summarized in (Table 4).

We observe that both models improve in performance with \(T>0\) and performance degrades past \(T=10^{4}\). Moreover, data augmentation and global normalization are complementary. Combined, we obtain 1.6 EM and 2.0 F1 improvement over the locally normalized baseline.

We also verify that the effects of Type Swaps are not limited to our specific model by observing the impact of augmented data on the DCN+ Xiong et al. (2016)6. We find that it strongly reduces generalization error, and helps improve F1, with potential further improvements coming by reducing other forms of regularization (Table 5).

Footnote 6: The DCN+ is the DCN with additional hyperparameter tuning by the same authors as submitted on the SQuAD leaderboard [https://rajpurkar.github.io/SQuAD-explorer/](https://rajpurkar.github.io/SQuAD-explorer/).

## 5 Discussion

In this section we will discuss the results presented in Section 4, and explain how they relate to our main claims.

### Extractive Question Answering as a Search Problem

Sentences provide a natural and powerful document decomposition for search that can be easily learnt as a search step: for all the models and configurations considered, the _Sentence_ score was 

above 88% correct (Table 3)7. Thus, sentence selection is the easy part of the problem, and the model can allocate more computation (such as the end-word selection Bi-LSTM) to spans likely to contain the answer. This approach avoids wasteful work on unpromising spans and is important for further scaling these methods to long documents.

Footnote 7: The objective function difference explains the lower performance of globally versus locally normalized models on the _Sentence_ score: local models must always assign the highest probability to the correct sentence, while global models only ensure the correct span has the highest probability. Thus global models do not need to enforce a high margin between the correct answer’s sentence score and others and are more likely to keep alternate sentences around.

### Global Normalization

The Globally Normalized Reader outperforms previous approaches and achieves the second highest EM behind Wang et al. (2017), without using bi-directional attention and only scoring spans in its final beam. Increasing the beam width improves the results for both locally and globally normalized models (Table 3), suggesting search errors account for a significant portion of the performance difference between models. Models such as Lee et al. (2016) and Wang and Jiang (2016) overcome this difficulty by ranking all possible spans and thus never skipping a possible answer. Even with large beam sizes, the locally normalized model underperforms these approaches. However, by increasing model flexibility and performing search during training, the globally normalized model is able to recover from search errors and achieve much of the benefits of scoring all possible spans.

### Type-Aware Data Augmentation

Type Swaps, our data augmentation strategy, offers a way to incorporate the nature of the question and the types of named entities in the answers into the learning process of our model and reduce sensitivity to surface variation. Existing neural-network approaches to extractive QA have so far ignored this information. Augmenting the dataset with additional type-sensitive synthetic examples improves performance by providing better coverage of different answer types. Growing the number of augmented samples used improves the performance of all models under study (Table 4-5). With \(T\in[10^{4},5\cdot 10^{4}]\), (EM, F1) improve from \((65.8\to 66.7,74.0\to 75.0)\) for locally normalized models, and \((66.6\to 68.4,75.0\to 76.21)\) for globally normalized models.

Past a certain amount of augmentation, we observe performance degradation. This suggests that despite efforts to closely mimic the original training set, there is a train-test mismatch or excess duplication in the generated examples.

Our experiments are conducted on two vastly different architectures and thus these benefits are expected to carry over to different models Weissenborn et al. (2017); Seo et al. (2016); Wang et al. (2017), and perhaps more broadly in other natural language tasks that contain named entities and have limited supervised data.

## 6 Related Work

Our work is closely related to existing approaches in learning to search, extractive question answering, and data augmentation for NLP tasks.

Learning to Search.Several approaches to learning to search have been proposed for various NLP tasks and conditional computation. Most recently, Andor et al. (2016) and Zhou et al. (2015) demonstrated the effectiveness of globally normalized networks and training with beam search for part of speech tagging and transition-based dependency parsing, while Wiseman and Rush (2016) showed that these techniques could also be applied to sequence-to-sequence models in several application areas including machine translation. These works focus on parsing and sequence prediction tasks and have a fixed computation regardless of the search path, while we show that the same techniques can also be straightforwardly applied to question answering and extended to allow for conditional computation based on the search path.

Learning to search has also been used in context of modular neural networks with conditional computation in the work of Andreas et al. (2016) for image captioning. In their work reinforcement learning was used to learn how to turn on and off computation, while we find that conditional computation can be easily learnt with maximum likelihood and the help of early updates Andor et al. (2016); Zhou et al. (2015); Collins and Roark (2004) to guide the training process.

Our framework for conditional computation whereby the search space is pruned by a sequence of increasingly complex models is broadly reminiscent of the structured prediction cascades of Weiss and Taskar (2010). Trischler et al. (2016b) 

also explored this approach in the context of question answering.

Extractive Question Answering.Since the introduction of the SQuAD dataset, numerous systems have achieved strong results. Seo et al. (2016); Wang et al. (2017) and Xiong et al. (2016) make use of a bi-directional attention mechanisms, whereas the GNR is more lightweight and achieves similar results without this type of attention mechanism. The document representation used by the GNR is very similar to Lee et al. (2016). However, both Lee et al. (2016) and Wang and Jiang (2016) must score all \(O(N^{2})\) possible answer spans, making training and inference expensive. The GNR avoids this complexity by learning to search during training and outperforms both systems while scoring only \(O(B)\) spans. Weissenborn et al. (2017) is a locally normalized model that first predicts start and then end words of each span. Our experiments lead us to believe that further factorizing the problem and using global normalization along with our data augmentation would yield corresponding improvements.

Data augmentation.Several works use data augmentation to control the generalization error of deep learning models. Zhang and LeCun (2015) use a thesaurus to generate new training examples based on synonyms. Vijayaraghavan et al. (2016) employs a similar method, but uses Word2vec and cosine similarity to find similar words. Jia and Liang (2016) use a high-precision synchronous context-free grammar to generate new semantic parsing examples. Our data augmentation technique, Type Swaps, is unique in that it leverages an external knowledge-base to provide new examples that have more variation and finer-grained changes than methods that use only a desaurus or Word2Vec, while also keeping the narrative and grammatical structure intact.

More recently Zhou et al. (2017) proposed a sequence-to-sequence model to generate diverse and realistic training question-answer pairs on SQuAD. Similar to their approach, our technique makes use of existing examples to produce new examples that are fluent, however we also are able to explicitly incorporate entity type information into the generation process and use the generated data to improve the performance of question answering models.

## 7 Conclusions and Future Work

In this work, we provide a methodology that overcomes several limitations of existing approaches to extractive question answering. In particular, our proposed model, the Globally Normalized Reader, reduces the computational complexity of previous models by casting the question answering as search and allocating more computation to promising answer spans. Empirically, we find that this approach, combined with global normalization and beam search during training, leads to near state of the art results. Furthermore, we find that a type-aware data augmentation strategy improves the performance of all models under study on the SQuAD dataset. The method is general, only requiring that the training data contains named entities from a large KB. We expect it to be applicable to other NLP tasks that would benefit from more training data.

As future work we plan to apply the GNR to other question answering datasets such as MS MARCO Nguyen et al. (2016) or NewsQA Trischler et al. (2016a), as well as investigate the applicability and benefits of Type Swaps to other tasks like named entity recognition, entity linking, machine translation, and summarization. Finally, we believe there a broad range of structured prediction problems (code generation, generative models for images, audio, or videos) where the size of original search space makes current techniques intractable, but if cast as learning-to-search problems with conditional computation, might be within reach.

We would like to thank the anonymous reviewers for their valuable feedback. In addition, we thank Adam Coates, Carl Case, Andrew Gibiansky, and Szymon Sidor for thoughtful comments and fruitful discussion. We also thank James Bradbury and Bryan McCann for running Type Swap experiments on the DCN+.

## References

* Abadi et al. (2016) Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. _arXiv preprint arXiv:1603.04467_ 

Tim Althoff, Ryen W White, and Eric Horvitz. 2016. Influence of pokemon go on physical activity: Study and implications. _Journal of Medical Internet Research_, 18(12).
* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. _arXiv preprint arXiv:1603.06042_.
* Andreas et al. (2016) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. _arXiv preprint arXiv:1601.01705_.
* Chetlur et al. (2014) Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cudnn: Efficient primitives for deep learning. _arXiv preprint arXiv:1410.0759_.
* Collins and Roark (2004) Michael Collins and Brian Roark. 2004. Incremental parsing with the perceptron algorithm. In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics_, page 111. Association for Computational Linguistics.
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. _Neural Networks_, 18(5):602-610.
* Honnibal (2017) Matthew Honnibal. 2017. Spacy. [https://github.com/explosion/spaCy](https://github.com/explosion/spaCy).
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. _arXiv preprint arXiv:1606.03622_.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Koedinger et al. (2015) Kenneth R Koedinger, Sidney D'Mello, Elizabeth A McLaughlin, Zachary A Pardos, and Carolyn P Rose. 2015. Data mining and education. _Wiley Interdisciplinary Reviews: Cognitive Science_, 6(4):333-353.
* Lee et al. (2016) Kenton Lee, Tom Kwiatkowski, Ankur Parikh, and Dipanjan Das. 2016. Learning recurrent span representations for extractive question answering. _arXiv preprint arXiv:1611.01436_.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. _arXiv preprint arXiv:1611.09268_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, volume 14, pages 1532-1543.
* Quirk and Poon (2016) Chris Quirk and Hoifung Poon. 2016. Distant supervision for relation extraction beyond the sentence boundary. _arXiv preprint arXiv:1609.04873_.
* Raiman (2017) Jonathan Raiman. 2017. Ciseau. [https://github.com/jonathanraiman/ciseau](https://github.com/jonathanraiman/ciseau).
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. _arXiv preprint arXiv:1606.05250_.
* Seo et al. (2016) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. _arXiv preprint arXiv:1611.01603_.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_, 15(1):1929-1958.
* Trischler et al. (2016a) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016a. Newsqa: A machine comprehension dataset. _arXiv preprint arXiv:1611.09830_.
* Trischler et al. (2016b) Adam Trischler, Zheng Ye, Xingdi Yuan, Jing He, Phillip Bachman, and Kaheer Suleman. 2016b. A parallel-hierarchical model for machine comprehension on sparse data. _arXiv preprint arXiv:1603.08884_.
* Vijayaraghavan et al. (2016) Prashanth Vijayaraghavan, Ivan Sysoev, Soroush Vosoughi, and Deb Roy. 2016. Deepstance at semeval-2016 task 6: Detecting stance in tweets using character and word-level cnns. _arXiv preprint arXiv:1606.05694_.
* Vrandecic and Krutzsch (2014) Denny Vrandecic and Markus Krutzsch. 2014. Wikidata: a free collaborative knowledgebase. _Communications of the ACM_, 57(10):78-85.
* Wang and Jiang (2016) Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. _arXiv preprint arXiv:1608.07905_.
* Wang et al. (2017) Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_.
* Weiss and Taskar (2010) David Weiss and Benjamin Taskar. 2010. Structured prediction cascades. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_, pages 916-923.
* Weissenborn et al. (2017) Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Fastqa: A simple and efficient neural architecture for question answering. _arXiv preprint arXiv:1703.04816_.
* Wiseman and Rush (2016) Sam Wiseman and Alexander M Rush. 2016. Sequence-to-sequence learning as beam-search optimization. _arXiv preprint arXiv:1606.02960_.

 

Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. _arXiv preprint arXiv:1611.01604_.
* Zhang and LeCun (2015) Xiang Zhang and Yann LeCun. 2015. Text understanding from scratch. _arXiv preprint arXiv:1502.01710_.
* Zhou et al. (2015) Hao Zhou, Yue Zhang, Shujian Huang, and Jiajun Chen. 2015. A neural probabilistic structured-prediction model for transition-based dependency parsing. In _ACL (1)_, pages 1213-1222.
* Zhou et al. (2017) Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. _arXiv preprint arXiv:1704.01792_.