SlowGuess/ACL_2020_OCR huggingface dataset:
Dayiheng Liu\*† Yeyun Gong† Jie Fu\* Yu Yan†
Jiusheng Chen $^{\text{♥}}$  Daxin Jiang $^{\dagger}$  Jiancheng Lv $^{\spadesuit}$  Nan Duan $^{\dagger}$
$\spadesuit$ College of Computer Science, Sichuan University †Microsoft Research Asia
$^{\diamond}$ Mila  ${}^{‡}$ Microsoft Search Technology Center Asia  $\text{♥}$ Microsoft AI and Research
losinuris@gmail.com
Reading long documents to answer open-domain questions remains challenging in natural language understanding. In this paper, we introduce a new model, called RikiNet, which reads Wikipedia pages for natural question answering. RikiNet contains a dynamic paragraph dual-attention reader and a multi-level cascaded answer predictor. The reader dynamically represents the document and question by utilizing a set of complementary attention mechanisms. The representations are then fed into the predictor to obtain the span of the short answer, the paragraph of the long answer, and the answer type in a cascaded manner. On the Natural Questions (NQ) dataset, a single RikiNet achieves 74.3 F1 and 57.9 F1 on long-answer and short-answer tasks. To our best knowledge, it is the first single model that outperforms the single human performance. Furthermore, an ensemble RikiNet obtains 76.1 F1 and 61.3 F1 on long-answer and short-answer tasks, achieving the best performance on the official NQ leaderboard<sup>1</sup>.
Machine reading comprehension (MRC) refers to the task of finding answers to given questions by reading and understanding some documents. It represents a challenging benchmark task in natural language understanding (NLU). With the progress of large-scale pre-trained language models (Devlin et al., 2018), state-of-the-art MRC models (Ju et al., 2019; Yang et al., 2019; Lan et al., 2019; Zhang et al., 2019; Liu et al., 2019) have already surpassed human-level performance on certain commonly used MRC benchmark datasets, such as SQuAD 1.1 (Rajpurkar et al., 2016), SQuAD 2.0 (Rajpurkar et al., 2018), and CoQA (Reddy et al., 2019).
Recently, a new benchmark MRC dataset called Natural Questions $^2$  (NQ) (Kwiatkowski et al., 2019) has presented a substantially greater challenge for the existing MRC models. Specifically, there are two main challenges in NQ compared to the previous MRC datasets like SQuAD 2.0. Firstly, instead of providing one relatively short paragraph for each question-answer (QA) pair, NQ gives an entire Wikipedia page which is significantly longer compared to other datasets. Secondly, NQ task not only requires the model to find an answer span (called short answer) to the question like previous MRC tasks but also asks the model to find a paragraph that contains the information required to answer the question (called long answer).
In this paper, we focus on the NQ task and propose a new MRC model called RikiNet tailored to its associated challenges, which Reads the Wikipedia pages for natural question answering. For the first challenge of the NQ task mentioned above, RikiNet employs the proposed Dynamic Paragraph Dual-Attention (DPDA) reader which contains multiple DPDA blocks. In each DPDA block, we iteratively perform dual-attention to represent documents and questions, and employ paragraph self-attention with dynamic attention mask to fuse key tokens in each paragraph. The resulting context-aware question representation, question-aware token-level, and paragraph-level representations are fed into the predictor to obtain the answer. The motivations of designing DPDA reader are: (a) Although the entire Wikipedia page contains a large amount of text, one key observation is that most answers are only related to a few words in one paragraph; (b) The final paragraph representation can be used naturally for predicting long answers.
$^{2}$ NQ provides some visual examples of the data at https://ai.google.com/research/NaturalQuestions/visualization.
1 Till our submission time, 29 Nov. 2019. We refer readers to https://ai.google.com/research/NaturalQuestions/leaderboard for the latest results.
6762
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6762-6771 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We describe the details of DPDA reader in § 3.1.
For the second challenge, unlike prior works on NQ dataset (Alberti et al., 2019b; Pan et al., 2019) that only predict the short answer and directly select its paragraph as long answer, RikiNet employs a multi-level cascaded answer predictor which jointly predict the short answer span, the long answer paragraph, and the answer type in a cascaded manner. Another key intuition motivating our design is that even if the relevant documents are not given, humans can easily judge that some questions have no short answers (Borschinger et al., 2019). Take this question as a motivating example: "What is the origin of the Nobel prize?" The answer should be based on a long story, which cannot be easily expressed in a short span of entities. Therefore we also feed the question representation into the predictor as an auxiliary prior to answer type prediction. The details will be given in § 3.2.
On the NQ test set, our single model obtains 74.3 F1 scores on the long-answer task (LA) and 57.9 F1 scores on the short-answer task (SA) compared to the published best single model (Alberti et al., 2019a) results of 66.8 F1 on LA and 53.9 F1 on SA. To the best of our knowledge, RikiNet is the first single model that outperforms the single human performance (Kwiatkowski et al., 2019) on both LA and SA. Besides, our ensemble model obtains 76.1 F1 on LA and 61.3 F1 on SA, which achieves the best performance of both LA and SA on the official NQ leaderboard.
Before we describe our model in detail, we first introduce the notations and problem formalization. Our paper considers the following NQ (Kwiatkowski et al., 2019) task: Given a natural question  $q$ , a related Wikipedia page  $p$  (in the top 5 search results returned by the Google search engine), the model outputs a paragraph within the Wikipedia page  $p$  as the long answer which contains enough information to infer the answer to the question, and an entity span within the long answer that answers the question as the short answer. Also, the short answer of the  $1\%$  Wikipedia page is "yes" or "no", instead of a short span. Both long answers and short answers can be NULL (i.e., no such answer could be found).
Given a natural question  $q$  and its paired Wikipedia page  $p$ , we tokenize them with the 30,522 wordpiece vocabulary as used in (Devlin
et al., 2018). Following (Alberti et al., 2019b; Pan et al., 2019), we generate multiple document spans by splitting the Wikipedia page with a sliding window. Then, we obtain multiple 6-tuple training instances  $(q,d,c,s,e,t)$  for each NQ data pair  $(q,p)$  where  $q$  and  $d$  are wordpiece IDs of question with length  $n$  and document span with length  $m$ ,  $c\in \mathbb{S}$  indicates the paragraph index of the long answer where  $\mathbb{S}$  is the set that includes all paragraph indexes (i.e., all long answer candidates) within  $d$ ,  $s,e\in \{0,1,\dots,m - 1\}$  are inclusive indices pointing to the start and end of the short answer span, and  $t\in \{0,1,2,3,4\}$  represents the five answer types, corresponding to the labels "NULL" (no answer), "SHORT" (has short answer), "LONG" (only has long answer), "YES", and "NO".
For each tuple  $(q, d, c, s, e, t)$  of the data pair  $(q, p)$ , RikiNet takes  $d$  and  $q$  as inputs, and jointly predicts  $c, s, e, t$ . Finally we merge the prediction results of every tuple to obtain the final predicted long answer, short answer, and their confidence scores of the data pair  $(q, p)$  for evaluation.
We propose the RikiNet which Reads the Wikipedia pages for natural question answering. As shown in Fig. 1, RikiNet consists of two modules: (a) the dynamic paragraph dual-attention reader as described in §3.1, and (b) the multi-level cascaded answer predictor as described in §3.2.
3.1 Dynamic Paragraph Dual-Attention Reader
Dynamic Paragraph Dual-Attention (DPDA) reader aims to represent the document span  $d$  and the question  $q$ . It outputs the context-aware question representation, question-aware token-level document representation, and paragraph-level document representation, which will be all fed into the predictor to obtain the long and short answers.
3.1.1 Encoding Question and Document Span
We firstly employ a pre-trained language model such as BERT (Devlin et al., 2018) to obtain the initial question representation  $Q_0 \in \mathbb{R}^{n \times h}$  and the initial document span representation  $D_0 \in \mathbb{R}^{m \times h}$ , where  $h$  is the hidden size. Similar to (Devlin et al., 2018), we concatenate a “[CLS)” token, the tokenized question  $q$  with length  $n$ , a “[SEP)” token, the tokenized document span  $d$  with length  $m$ , and a final “[SEP)” token. Then we feed the resulting sequence into the pre-trained language model.
6763
Figure 1: Overview of RikiNet framework.
3.1.2 Dynamic Paragraph Dual-Attention Block
As shown on the left in Fig. 1, DPDA reader contains multiple Dynamic Paragraph Dual-Attention (DPDA) blocks. The first block takes  $Q_{0}$  and  $D_{0}$  as the inputs. The outputs  $Q_{(t)}$  and  $D_{(t)}$  of the  $t$ -th block are then fed into the next block. Each block contains three types of layers: the dual-attention layer, the paragraph dynamic self-attention layer, and the question self-attention layer. The last DPDA block outputs the final question and document representations. We describe them in detail now.
Dual-Attention Layer To strengthen the information fusion from the question to the paragraphs as well as from the paragraphs to the question, we adapt a dual-attention mechanism, which has been shown effective in other MRC models (Xiong et al., 2018; Seo et al., 2017; Xiong et al., 2017). We further tweak it by increasing the depth of attention followed by a residual connection (He et al., 2016) and layer normalization (Ba et al., 2016).
In particular, the  $t$ -th block first calculates a similarity metric  $L_{(t)} \in \mathbb{R}^{m \times n}$  which is then normalized row-wise and column-wise to produce two attention weights:  $A_{(t)}^Q \in \mathbb{R}^{m \times n}$ , across the document for each token in the question; and  $A_{(t)}^D \in \mathbb{R}^{n \times m}$ , across the question for each token
in the document,
$$
L _ {(t)} = D _ {(t - 1)} Q _ {(t - 1)} ^ {\top} \in \mathbb {R} ^ {m \times n},
$$
$$
A _ {(t)} ^ {Q} = \operatorname {S o f t m a x} \left(L _ {(t)}\right) \in \mathbb {R} ^ {m \times n},
$$
$$
A _ {(t)} ^ {D} = \mathrm {S o f t m a x} \left(L _ {(t)} ^ {\top}\right) \in \mathbb {R} ^ {n \times m}.
$$
Similar to (Xiong et al., 2017; Seo et al., 2017), we obtain the question-aware representation of the document by
$$
\bar {\bar {Q}} _ {(t)} ^ {C} = \left(D _ {(t - 1)} ^ {\top} A _ {(t)} ^ {Q}\right) ^ {\top} \in \mathbb {R} ^ {n \times h},
$$
$$
\bar {D} _ {(t)} ^ {C} = \left(A _ {(t)} ^ {D}\right) ^ {\top} \left[ Q _ {(t - 1)}; \bar {\bar {Q}} _ {(t)} ^ {C} \right] \in \mathbb {R} ^ {m \times 2 h},
$$
where  $[\cdot ;\cdot ]$  denotes concatenation. We also obtain the context-aware question representation in a dual way:
$$
\bar {\bar {D}} _ {(t)} ^ {C} = \left(Q _ {(t - 1)} ^ {\top} A _ {(t)} ^ {D}\right) ^ {\top} \in \mathbb {R} ^ {m \times h},
$$
$$
\bar {Q} _ {(t)} ^ {C} = \left(A _ {(t)} ^ {Q}\right) ^ {\top} \left[ D _ {(t - 1)}; \bar {\bar {D}} _ {(t)} ^ {C} \right] \in \mathbb {R} ^ {n \times 2 h}.
$$
We finally apply the residual connection and layer normalization to both the question and the document representations with the linear transformations.
$$
D _ {(t)} ^ {C} = \mathrm {L a y e r N o r m} \left(D _ {(t - 1)} + \bar {D} _ {(t)} ^ {C} W _ {(t)} ^ {D}\right) \in \mathbb {R} ^ {m \times h},
$$
$$
Q _ {(t)} ^ {C} = \mathrm {L a y e r N o r m} \left(Q _ {(t - 1)} + \bar {Q} _ {(t)} ^ {C} W _ {(t)} ^ {Q}\right) \in \mathbb {R} ^ {n \times h},
$$
6764
where  $W_{(t)}^{D}\in \mathbb{R}^{2h\times h}$  and  $W_{(t)}^{Q}\in \mathbb{R}^{2h\times h}$  are trainable parameters in the dual-attention layer of the  $t$ -th block. The document representation  $D_{(t)}^{C}$  will be fed into the paragraph dynamic self-attention layer to obtain the paragraph representation. The question representation  $Q_{(t)}^{C}$  will be fed into the question self-attention layer to get the question embedding.
Question Self-Attention Layer This layer uses a transformer self-attention block (Vaswani et al., 2017) to further enrich the question representation:
$$
Q _ {(t)} = \mathrm {T r a n s f o r m e r} \left(Q _ {(t)} ^ {C}\right) \in \mathbb {R} ^ {n \times h},
$$
where the transformer block consists of two sublayers: a multi-head self-attention layer and a position-wise fully connected feed-forward layer. Each sub-layer is placed inside a residual connection with layer normalization. After the last DPDA block, we obtain the final question embedding  $\pmb{q} \in \mathbb{R}^h$  by applying the mean pooling,
$$
\pmb {q} = \mathrm {M e a n P o o l i n g} \left(Q _ {(T)} ^ {C}\right) \in \mathbb {R} ^ {h},
$$
where  $T$  denotes the number of the DPDA blocks. This question embedding  $\pmb{q}$  will be further fed into the predictor for answer type prediction.
Paragraph Dynamic Self-Attention Layer This layer is responsible for gathering information on the key tokens in each paragraph. The token-level representation  $D_{(t)}$  is first given by:
$$
D _ {(t)} = \operatorname {T r a n s f o r m e r} \left(D _ {(t)} ^ {C}\right) \in \mathbb {R} ^ {m \times h}. \tag {1}
$$
The difference from the original multi-head selfattention in (Vaswani et al., 2017) is that we incorporate two extra attention masks, which will be introduced later in Eq. (3) and (4). The last DPDA block applies a mean pooling to the tokens within the same paragraph to obtain the paragraph representation  $L \in \mathbb{R}^{l \times h}$  as
$$
L [ i,: ] = \underset {\mathbb {L} _ {j} = i} {\text {M e a n P o o l i n g}} \left(\left\{D _ {(T)} [ j,: ] \right\}\right) \in \mathbb {R} ^ {h}, \tag {2}
$$
where  $l$  denotes the number of paragraph within the document span  $d$  (i.e., the number of long answer candidates within the document span  $d$ ),  $L[i,:]$  is the representation of the  $i$ -th paragraph,  $D_{(T)}[j,:]$  is the representation of the  $j$ -th token at last DPDA
block, and  $\mathbb{L}_j$  indicates the index number of the paragraph where the  $j$ -th token is located.
Tokens in the original multi-head attention layer of the transformer self-attention block attend to all tokens. We introduce two attention masks to the self-attention sub-layer in Eq. (1) based on two key motivations: 1) Each paragraph representation should focus on the question-aware token information inside the paragraph; 2) Most of the answers are only related to a few words in a paragraph. For the first motivation, we introduce the paragraph attention mask  $\mathcal{M}^L\in \mathbb{R}^{m\times m}$  which is defined as:
$$
\mathcal {M} ^ {L} [ i, j ] = \left\{ \begin{array}{l l} 0, & \text {i f} \mathbb {L} _ {i} = \mathbb {L} _ {j}, \\ - \infty , & \text {o t h e r w i s e .} \end{array} \right. \tag {3}
$$
It forces each token to only attend to the tokens within the same paragraph. Therefore, each paragraph representation focuses on its internal token information after the mean pooling of Eq. (2).
Based on the second motivation, we dynamically generate another attention mask to select key tokens before self-attention. We use a neural network  $\mathcal{F}_{(t)}^{\Phi}$  called scorer with the Sigmoid activation function to calculate the importance score for each token:
$$
\Phi_ {(t)} = \mathcal {F} _ {(t)} ^ {\Phi} \left(D _ {(t)} ^ {C}\right) \in \mathbb {R} ^ {m \times 1},
$$
Then we obtain the dynamic attention mask  $\mathcal{M}_{(t)}^{\Phi} \in \mathbb{R}^{m \times m}$  by selecting top- $K$  tokens
$$
\mathcal {M} _ {(t)} ^ {\Phi} [ i, j ] = \left\{ \begin{array}{l l} 0, & \text {i f} i \in \mathbb {S} _ {(t)} ^ {\Phi} \text {a n d} j \in \mathbb {S} _ {(t)} ^ {\Phi} \\ - \infty , & \text {o t h e r w i s e}, \end{array} \right. \tag {4}
$$
where  $\mathbb{S}_{(t)}^{\Phi} = \operatorname*{argmax - K}\left(\left\{\Phi_{(t)}[k]\right\}\right)$ . Here  $\Phi_{(t)}[k]$  denotes the score of the  $k$ -th token at  $t$ -th block,  $K$  is a hyperparameter, and  $\mathbb{S}_{(t)}^{\Phi}$  is the set that includes the index of the selected top- $K$  tokens. This attention mask lets the paragraph representation concentrate on the selected key tokens.
The final scaled dot-product attention weight  $A_{(t)}\in \mathbb{R}^{m\times m}$  of the multi-head self-attention sublayer (Vaswani et al., 2017) in Eq. (1) with two proposed attention masks can be written as:
$$
\underline {{A _ {(t)}}} = \operatorname {S o f t m a x} \left(\mathcal {M} _ {(t)} ^ {\Phi} + \mathcal {M} ^ {L} + \frac {\left(D _ {(t)} ^ {C} D _ {(t)} ^ {C} {} ^ {\top}\right)}{\sqrt {h}}\right).
$$
$^3$ Following Zhuang and Wang (2019), our implementation pads the unselected token representations with zero embeddings and adds the scorer representation with the linear transformation to  $D_{(t)}$  to avoid gradient vanishing for scorer training.
3.2 Multi-level Cascaded Answer Predictor
Due to the nature of the NQ tasks, a short answer is always contained within a long answer, and thus it makes sense to use the prediction of long answers to facilitate the process of obtaining short answers. As shown on the right in Fig. 1, we design a cascaded structure to exploit this dependency. This predictor takes the token representation  $D_{(T)}$ , the paragraph representation  $L$ , and the question embedding  $q$  as inputs to predict four outputs in a cascaded manner: (1) long answer  $\rightarrow$  (2) the start position of the short answer span  $\rightarrow$  (3) the end position of the short answer span  $\rightarrow$  (4) the answer type. That is, the previous results are used for the next tasks as indicated by the notation  $\rightarrow$ .
Long Answer Prediction We employ a dense layer  $\mathcal{F}^L$  with Tanh activation function as long answer prediction layer, which takes the paragraph representation  $L\in \mathbb{R}^{l\times h}$  as input to obtain the long-answer prediction representation  $H^{L}\in \mathbb{R}^{l\times h}$ . Then the long-answer logits  $\pmb{o}^{L}$  are computed with a linear layer
$$
H ^ {L} = \mathcal {F} ^ {L} (L) \in \mathbb {R} ^ {l \times h},
$$
$$
\boldsymbol {o} ^ {L} = H ^ {L} W ^ {L} \in \mathbb {R} ^ {l},
$$
where  $W^{L}\in \mathbb{R}^{h\times 1}$  is a trainable parameter.
Short Answer Prediction Firstly, we use the long-answer prediction representation  $H^L$  and the token representation  $D_{(T)}$  as the inputs to predict the start position of the short answer. Then the prediction representation of the start position of the short answer will be re-used to predict the end position.
Since the row-dimension of  $D_{(T)}\in \mathbb{R}^{m\times h}$  is different from that of  $H^L\in \mathbb{R}^{l\times h}$ , we cannot directly concatenate the  $H^{L}$  to  $D_{(T)}$ . We tile the  $H^{L}\in \mathbb{R}^{l\times h}$  with  $\bar{H}^L\in \mathbb{R}^{m\times h}$  along the row-dimension:  $\bar{H}^L [i,:] = H^L [\mathbb{L}_i,:]\in \mathbb{R}^h$ . Note that  $\mathbb{L}_i$  indicates the index number of the paragraph where the  $i$ -th token is located. Thus, the model can consider the prediction information of the long answer when predicting the short answer. Similarly, the start and end position logits of the short answer are predicted by,
$$
H ^ {S} = \mathcal {F} ^ {S} \left(\left[ \bar {H} ^ {L}; D _ {(T)} \right]\right) \in \mathbb {R} ^ {m \times h},
$$
$$
\boldsymbol {o} ^ {S} = H ^ {S} W ^ {S} \in \mathbb {R} ^ {m},
$$
$$
H ^ {E} = \mathcal {F} ^ {E} \left(\left[ H ^ {S}; D _ {(T)} \right]\right) \in \mathbb {R} ^ {m \times h},
$$
$$
\boldsymbol {o} ^ {E} = H ^ {E} W ^ {E} \in \mathbb {R} ^ {m},
$$
where  $o^S$  and  $o^E$  are the output logit vectors of the start positions and the end positions of the short answer,  $\mathcal{F}^S$  and  $\mathcal{F}^E$  are two dense layers with Tanh activation function, and  $W^{S}\in \mathbb{R}^{h\times 1}$ $W^{E}\in \mathbb{R}^{h\times 1}$  are trainable parameters.
Answer Type Prediction Finally, the predictor outputs the answer type. There are five answer types as discussed in § 2. With the observation that humans can easily judge that some questions have no short answers even without seeing the document, we treat the question embedding  $\pmb{q} \in \mathbb{R}^h$  as an auxiliary input for the answer type prediction. Besides, the token representation  $D_{(T)}$  and the short-answer prediction representation  $H^{E}$  are also used for that prediction:
$$
\boldsymbol {d} = \operatorname {M e a n P o o l i n g} \left(D _ {(T)}\right) \in \mathbb {R} ^ {h},
$$
$$
\boldsymbol {e} = \operatorname {M a x P o o l i n g} \left(H ^ {E}\right) \in \mathbb {R} ^ {h},
$$
$$
\boldsymbol {h} ^ {T} = \mathcal {F} ^ {T} \left(\left[ \boldsymbol {d}; \boldsymbol {q}; \boldsymbol {e} \right]\right) \in \mathbb {R} ^ {h},
$$
$$
\boldsymbol {o} ^ {T} = \operatorname {S o f t m a x} \left(\boldsymbol {h} ^ {T} \boldsymbol {W} ^ {T}\right) \in \mathbb {R} ^ {5},
$$
where  $\pmb{o}^T$  is the logits of the five answer types,  $\mathcal{F}^T$  is a dense layer with Tanh activation function, and  $W^{T}\in \mathbb{R}^{h\times 5}$  is a trainable parameter.
Training Loss and Inference For training, we compute cross-entropy loss over the above mentioned output logits, and jointly minimize these four cross-entropy losses as:
$$
\mathcal {L} = \mathcal {L} ^ {L} + \mathcal {L} ^ {S} + \mathcal {L} ^ {E} + \mathcal {L} ^ {T}.
$$
During inference, we calculate the final long-answer score  $\Psi^L$  for all the paragraphs within the Wikipedia page based on the long-answer logits  $o^L$  and the answer type logits  $o^T$ . The long-answer score of paragraph  $c$  can be written as
$$
\Psi^ {L} (c) = \boldsymbol {o} ^ {L} [ c ] + \underbrace {\left(\sum_ {t = 1} ^ {4} \boldsymbol {o} ^ {T} [ t ] - \boldsymbol {o} ^ {T} [ 0 ]\right)} _ {\text {a n s w e r t y p e s c o r e}},
$$
where  $\boldsymbol{o}^T[0]$  denotes the logits where the answer type is "NULL" (no answer),  $\sum_{t=1}^{4} \boldsymbol{o}^T[t]$  denotes the sum of the logits where the answer type is not "NULL". The answer type score can be seen as a bias of each document span in the Wikipedia page. Then we select the paragraph of the highest long-answer score  $\Psi^L$  over the entire Wikipedia page as the long answer.
6766
Similarly, the short-answer score of the corresponding span  $(s,e)$  is calculated by
$$
\Psi^ {S} (s, e) = \underbrace {\left(\boldsymbol {o} ^ {S} [ s ] + \boldsymbol {o} ^ {E} [ e ]\right)} _ {\text {a n s w e r s p a n s c o r e}} + \underbrace {\left(\boldsymbol {o} ^ {T} [ 1 ] - \boldsymbol {o} ^ {T} [ 0 ]\right)} _ {\text {a n s w e r t y p e s c o r e}},
$$
where  $\pmb{o}^T[1]$  denotes the score where the answer type is "SHORT" (has short answer). We select the short answer span which has the highest short-answer score  $\Psi^S$  within the long answer as the final short answer. We use the official NQ evaluation script to set two separate thresholds for predicting whether the two types of answers are answerable.
We focus on the Natural Questions (NQ) (Kwiatkowski et al., 2019) dataset in this work. The public release of the NQ dataset consists of 307,373 training examples and 7,830 examples for development data (dev set). NQ provides a blind test set contains 7,842 examples, which can only be accessed through a public leaderboard submission.