SlowGuess/ACL_2020_OCR huggingface dataset:
Bo Zheng $^{1*}$ , Haoyang Wen $^{1}$ , Yaobo Liang $^{2}$ , Nan Duan $^{2}$ , Wanxiang Che $^{1\dagger}$ , Daxin Jiang $^{3}$ , Ming Zhou $^{2}$ , Ting Liu $^{1}$ $^{1}$ Harbin Institute of Technology, Harbin, China  
 $^{2}$ Microsoft Research Asia, Beijing, China  
 $^{3}$ STCA NLP Group, Microsoft, Beijing, China  
{bzheng, hywen, car, tliu} @ir.hit.edu.cn  
{yalia, NANDuan, djiang, mingzhou} @microsoft.com
Natural Questions is a new challenging machine reading comprehension benchmark with two-grained answers, which are a long answer (typically a paragraph) and a short answer (one or more entities inside the long answer). Despite the effectiveness of existing methods on this benchmark, they treat these two sub-tasks individually during training while ignoring their dependencies. To address this issue, we present a novel multi-grained machine reading comprehension framework that focuses on modeling documents at their hierarchical nature, which are different levels of granularity: documents, paragraphs, sentences, and tokens. We utilize graph attention networks to obtain different levels of representations so that they can be learned simultaneously. The long and short answers can be extracted from paragraph-level representation and token-level representation, respectively. In this way, we can model the dependencies between the two-grained answers to provide evidence for each other. We jointly train the two sub-tasks, and our experiments show that our approach significantly outperforms previous systems at both long and short answer criteria.
Machine reading comprehension (MRC), a task that aims to answer questions based on a given document, has been substantially advanced by recently released datasets and models (Rajpurkar et al., 2016; Seo et al., 2017; Xiong et al., 2017; Joshi et al., 2017; Cui et al., 2017; Devlin et al., 2019; Clark and Gardner, 2018). Natural Questions (NQ, Kwiatkowski et al., 2019), a newly released benchmark, makes it more challenging by introducing much longer documents than existing datasets
Question: where is the bowling hall of fame located  
Wikipedia page: International Bowling Hall of Fame  
Long answer: The World Bowling Writers (WBW)  
International Bowling Hall of Fame was established in 1993 and is located in the International Bowling Museum and Hall of Fame, on the International Bowling Campus in Arlington, Texas.  
Short answer: Arlington, Texas
Figure 1: An example from NQ dataset.
and questions that are from real user queries. Besides, unlike conventional MRC tasks (e.g. Rajpurkar et al.,2016), in NQ, answers are provided in a two-grained format: long answer, which is typically a paragraph, and short answers, which are typically one or more entities inside the long answer. Figure 1 shows an example from NQ dataset.
Existing approaches on NQ have obtained promising results. For example, Kwiatkowski et al. (2019) builds a pipeline model using two separate models: the Decomposable Attention model (Parikh et al., 2016) to select a long answer, and the Document Reader model (Chen et al., 2017) to extract the short answer from the selected long answer. Despite the effectiveness of these approaches, they treat the long and short answer extraction as two individual sub-tasks during training and fail to model this multi-grained characteristic of this benchmark, while we argue that the two sub-tasks of NQ should be considered simultaneously to obtain accurate results.
According to Kwiatkowski et al. (2019), a valid long answer must contain all of the information required to answer the question. Besides, an accurate short answer should be helpful to confirm the long answer. For instance, when humans try to find the two-grained answers in the given Wikipedia page in Figure 1, they will first try to retrieve paragraphs
*Work was done while this author was an intern at Microsoft Research Asia.
† Email corresponding.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6708-6718 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(long answer) describing the entity bowling hall of fame, then try to confirm if the location (short answer) of the asked entity exists in the paragraph, which helps to finally decide which paragraph is the long answer. In this way, the two-grained answers can provide evidence for each other.
To address the two sub-tasks together, instead of using conventional documents modeling methods like hierarchical RNNs (Cheng and Lapata, 2016; Yang et al., 2016; Nallapati et al., 2017; Narayan et al., 2018), we propose to use graph attention networks (Velickovic et al., 2018) and BERT (Devlin et al., 2019), directly model representations at tokens, sentences, paragraphs, and documents, the four different levels of granularity to capture hierarchical nature of documents. In this way, we directly derive scores of long answers from its paragraph-level representations and obtain scores of short answers from the start and end positions on the token-level representations. Thus the long and short answer selection tasks can be trained jointly to promote each other. At inference time, we use a pipeline strategy similar to Kwiatkowski et al. (2019), where we first select long answers and then extract short answers from the selected long answers.
Experiments on NQ dataset show that our model significantly outperforms previous models at both long and short answer criteria. We also analyze the benefits of multi-granularity representations derived from the graph module in experiments.
To summarize, the main contributions of this work are as follows:
- We propose a multi-grained MRC model based on graph attention networks and BERT.
- We apply a joint training strategy where long and short answers can be considered simultaneously, which is beneficial for modeling the dependencies of the two-grained answers.
- We achieve state-of-the-art performance on both long and short answer leaderboard of NQ at the time of submission (Jun. 25th, 2019), and our model surpasses single human performance on the development dataset at both long and short answer criteria.
We will release our code and models at https://github.com/DancingSoul/NQ_BERT-DM.
Figure 2: System overview. The document fragments of one document are fed into our model independently. The outputs of graph encoders are merged and sent into the answer selection module, which generates a long answer and a short answer.
2.1 Natural Questions Dataset
Each example in NQ dataset contains a question together with an entire Wikipedia page. The models are expected to predict two types of outputs: 1) long answer, which is an HTML span containing enough information for a reader to completely infer the answer to the question. It can be a paragraph, a table, a list item, or a whole list. A long answer is selected in a list of candidates, or a “no answer” should be given if no candidate answers the question; 2) short answer, which can be “yes”, “no” or a list of entities within the long answer. Also, a “no answer” should be given if there is no suitable short answer.
2.2 Data Preprocessing
Since the average length of the documents in NQ is too long to be considered as one training instance, we first split each document into a list of document fragments with overlapping windows of tokens, like in the original BERT model for the MRC tasks (Alberti et al., 2019b; Devlin et al., 2019). Then we generate an instance from a document fragment by concatenating a "[CLS]" token, tokenized question, a "[SEP]" token, tokens from the content of the doc
6709
Figure 3: Inner structure of our graph encoder.
ument fragment and a final "[SEP]" token. "[CLS]" and "[SEP]" follow the definitions from Devlin et al. (2019). We tag each document fragment with an answer type as one of the five labels to construct a training instance: "short" for instances that contain all annotated short spans, "yes" and "no" for yes/no annotations where the instances contain the long answer span, "long" when the instances contain the long answer span, but there is no short or yes/no answer. In addition to the above situations, we tag a "no-answer" to those instances.
We will explain more details of the data preprocessing in the experiment section.
In this section, we will explain our model. The main idea of our model lies in multi-granularity document modeling with graph attention networks. The overall architecture of our model is shown in Figure 2.
3.1 Input & Output Definition
Formally, we define an instance in the training set as a six-tuple
$$
(\boldsymbol {c}, S, l, s, e, t).
$$
Suppose the instance is generated from the  $i$ -th document fragment  $D_{i}$  of the corresponding example, then  $\pmb{c} = ([\mathrm{CLS}], Q_{1}, \dots, Q_{|Q|}, [\mathrm{SEP}], D_{i,1}, \dots, D_{i,|D_{i}|}, [\mathrm{SEP}])$  defines the document fragment  $D_{i}$  along with a question  $Q$  of the instance,  $|Q| + |D_{i}| + 3 = 512$  corresponding to the data preprocessing method.  $S$  denotes the set of long answer candidates inside the document fragment.  $l \in S$
Figure 4: The graph on the left is an illustration of the graph integration layer. The graph on the right shows the incoming information when updating a paragraph node. The solid lines represent the edges in the hierarchical tree structure of a document while the dash lines stand for the edges we additionally add.
is the target long answer candidate among the candidate set  $S$  of this instance.  $s, e \in \{0, 1, \dots, 511\}$  are inclusive indices pointing to the start and end of the target answer span.  $t \in \{0, 1, 2, 3, 4\}$  is the annotated answer type, corresponding to the five labels. For instances containing multiple short answers, we set  $s$  and  $e$  to point to the leftmost position of the first short answer and the rightmost position of the last short answer, respectively.
Our goal is to learn a model that identifies a long answer candidate  $l$  and a short answer span  $(s, e)$  in  $l$  and predicting their scores for evaluation.
3.2 Multi-granularity Document $^{1}$  Modeling
The intuition of representing documents in multi-granularity is derived from the natural hierarchical structure of a document. Generally speaking, a document can be decomposed to a list of paragraphs, which can be further decomposed to lists of sentences and lists of tokens. Therefore, it is straightforward to treat the document structure as a tree, which has four types of nodes, namely token nodes, sentence nodes, paragraph nodes, and a document node. Different kinds of nodes represent information at different levels of granularity. Since long answer candidates are paragraphs, tables, or lists, information at paragraph nodes also represents the information for long answer candidates.
The hierarchical tree structure for a document contains edges that are between tokens and sentences, between sentences and paragraphs, and between paragraphs and documents. Besides, we further add edges between tokens and paragraphs, between tokens and documents, between sentences and the document to construct a graph. All these
For brevity, the word "document" refers to document fragment in the rest of our paper.
edges above are bidirectional in our graph representation. Hence information between every two nodes can be passed through no more than two edges in the graph. In the rest of this section, we will present how we utilize this graph structure to pass information between nodes with graph attention networks so that the two-grained answers can promote each other.
Figure 3 shows the inner structure of our graph encoder. Each layer in our graph encoder consists of three self-attention layers, a graph integration layer, and a feed-forward layer. The self-attention layers are used for interactions among nodes with the same granularity, while the graph integration layer aims at gathering information from other levels of granularity with graph attention networks. Figure 4 is an illustration for the graph integration layer. Since self-attention is a special case of graph attention networks, where the graph is fully connected, we only introduce the general form of graph attention networks, which can be generalized to the self-attention mechanism.
3.3.1 Graph Attention Networks
We apply graph attention networks (Velickovic et al., 2018) to model the information flow between nodes, which can further improve the representations of nodes by attention mechanism over features from its neighbors. In this way, the interaction between the two-grained answers can be enhanced. Instead of other graph-based models, we use graph attention networks to keep consistency with the multi-head attention module in the BERT model. We will describe a single layer of our graph attention networks in the following.
We define a graph  $\mathcal{G} = (\mathcal{V},\mathcal{E},X)$  that is composed of a set of nodes  $\nu$  node features  $X = (h_{1},\dots,h_{|\mathcal{V}|})$  and a list of directed edge set  $\mathcal{E} = (\mathcal{E}_1,\dots,\mathcal{E}_K)$  where  $K$  is the number of edges. Each  $i\in \mathcal{V}$  has its own representation  $h_i\in \mathbb{R}^{d_h}$  where  $d_{\mathrm{h}}$  is the hidden size of our model.
We use the multi-head attention mechanism in our graph attention networks following Vaswani et al. (2017). We describe one of the  $m$  attention heads. All the parameters are unique to each attention head and layer. If there is an edge from node  $j$  to node  $i$ , the attention coefficient  $e_{ij}$  is calculated as follows:
$$
e _ {i j} = \frac {\left(\boldsymbol {h} _ {i} \mathbf {W} ^ {\mathrm {Q}}\right) \left(\boldsymbol {h} _ {j} \mathbf {W} ^ {\mathrm {K}}\right) ^ {\mathrm {T}}}{\sqrt {d _ {\mathrm {z}}}}. \tag {1}
$$
We normalize the attention coefficients of node  $i$  by using the softmax function across all the neighbor nodes  $j \in \mathcal{N}_i$ . Especially, there is a self-loop for each node (i.e.  $i \in \mathcal{N}_i$ ) to allow it update itself. This process can be expressed as:
$$
\alpha_ {i j} = \operatorname {s o f t m a x} _ {j} \left(e _ {i j}\right) = \frac {\exp \left(e _ {i j}\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(e _ {i k}\right)}.
$$
Then the output of this attention head  $\mathbf{z}_i$  is computed as a weighted sum of linear transformed input elements:
$$
\boldsymbol {z} _ {i} = \sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \boldsymbol {h} _ {j} \mathbf {W} ^ {\mathrm {V}}. \tag {2}
$$
In the above equations,  $\mathbf{W}^{\mathrm{Q}},\mathbf{W}^{\mathrm{K}}$  and  $\mathbf{W}^{\mathrm{V}}\in$ $\mathbb{R}^{d_{\mathrm{h}}\times d_{\mathrm{z}}}$  are parameter matrices,  $d_{\mathrm{z}}$  is the output size of one attention head, we use  $d_{\mathrm{z}}\times m = d_{\mathrm{h}}$
Finally we get the multi-head attention result  $z_{i}^{\prime}\in \mathbb{R}^{d_{\mathrm{h}}}$  by concatenating the outputs of  $m$  individual attention heads:
$$
z_{i}^{\prime} = \prod_{k = 1}^{m}z_{i}^{k}.
$$
3.3.2 Self-Attention Layer
The self-attention mechanism is equivalent to the fully-connected version of graph attention networks. To make interactions among nodes with the same granularity, we utilize three self-attention layers, which are token-level self-attention, sentence-level self-attention, and paragraph-level self-attention. Since the four types of nodes are essentially heterogeneous, we separate the self-attention layer from the graph integration layer to distinguish information from nodes with the same granularity or different ones.
3.3.3 Graph Integration Layer
We use graph attention networks on the graph presented in Figure 4, this layer allows information to be passed to nodes with different levels of granularity. Instead of integrating information only once after the graph encoder, we put this layer right after every self-attention layer inside the graph encoder, which means the update brought by the self-attention layer will also be utilized by the nodes with other levels of granularity. This layer helps to model the dependencies of the two-grained answers. We concatenate the input and output of the graph integration layer and pass it to the feedforward layer.
6711
3.3.4 Feed-Forward Layer
Following the inner structure of the transformer (Vaswani et al., 2017), we also utilize an additional fully connected feed-forward network at the end of our graph encoder. It consists of two linear transformations with a GELU activation in between. GELU is Gaussian Error Linear Unit activation (Hendrycks and Gimpel, 2016), and we use GELU as the non-linear activation, which is consistent with BERT.
3.3.5 Relational Embedding
Inspired by positional encoding in Vaswani et al. (2017) and relative position representations in Shaw et al. (2018), we introduce a novel relational embedding on our constructed graph, which aims at modeling the relative position information between nodes on the multi-granularity document structure. We make the edges in our document modeling graph to embed relative positional information. We modify equation 1 and 2 for  $e_{ij}$  and  $z_i$  to introduce our relational embedding as follows:
$$
\begin{array}{l} e _ {i j} = \frac {\left(\boldsymbol {h} _ {i} \mathbf {W} ^ {\mathrm {Q}}\right) \left(\boldsymbol {h} _ {j} \mathbf {W} ^ {\mathrm {K}}\right) ^ {\mathrm {T}} + \boldsymbol {h} _ {i} \mathbf {W} ^ {\mathrm {Q}} \left(\boldsymbol {a} _ {i j} ^ {\mathrm {K}}\right) ^ {\mathrm {T}}}{\sqrt {d _ {\mathrm {z}}}}, \\ \boldsymbol {z} _ {i} = \sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \left(\boldsymbol {h} _ {j} \mathbf {W} ^ {\mathrm {V}} + \boldsymbol {a} _ {i j} ^ {\mathrm {V}}\right). \\ \end{array}
$$
In above equations, the edge between node  $i$  and node  $j$  is represented by learnable embedding  $\pmb{a}_{ij}^{K},\pmb{a}_{ij}^{V}\in \mathbb{R}^{d_{z}}$ . The representation can be shared across attention heads. Compared to previous work which encodes positional information in the embedding layer, our proposed relational embedding is more flexible, and the positional information can be taken into consideration in each graph layer. For example, relational embedding between two nodes of the same type represents the relative distance between them in the self-attention layer. In the graph integration layer, relational embedding between a sentence and its paragraph represents the relative position of the sentence in the paragraph, and it is the same for other types of edges.
3.3.6 Graph Initialization
Since the BERT model can only provide token-level representation, we use a bottom-up average-pooling strategy to initialize the nodes other than token-level nodes. We use  $o_i \in \{0, 1, 2, 3\}$  to indicate the type of node  $i$ , representing token node, sentence node, paragraph node and document node
respectively. The initialized representation is calculated as follows:
$$
\boldsymbol{h}_{i}^{0} = \operatorname *{average}_{j\in \mathcal{N}_{i},o_{j} + 1 = o_{i}}\bigl\{ \boldsymbol{h}_{j}^{0} + \boldsymbol{a}_{ij}\bigr \} +\boldsymbol{b}_{o_{i}},
$$
where  $\pmb{a}_{ij}, \pmb{b}_{o_i} \in \mathbb{R}^{d_h}$  represent the relational embedding and node type embedding in the graph initializer.
3.4 Output Layer
The objective function is defined as the negative sum of the log probabilities of the predicted distributions, averaged over all the training instances. The log probabilities of predicted distributions are indexed by the true start and end indices, true long answer candidate index, and the type of this instance:
$$
\begin{array}{l} L (\theta) = - \frac {1}{N} \sum_ {i} ^ {N} [ \log p (s, e, t, l \mid \mathbf {c}, S) ] \\ = - \frac {1}{N} \sum_ {i} ^ {N} [ \log p _ {s} (s \mid \mathbf {c}, S) \\ + \log p _ {e} (e \mid \mathbf {c}, S) + \log p _ {t} (t \mid \mathbf {c}, S) \\ + \log p _ {l} (l \mid \mathbf {c}, S) ], \\ \end{array}
$$
where  $p_s(s \mid \mathbf{c}, S)$ ,  $p_e(e \mid \mathbf{c}, S)$ ,  $p_l(l \mid \mathbf{c}, S)$  and  $p_t(t \mid \mathbf{c}, S)$  are the probabilities for the start and end position of the short answer, probabilities for the long answer candidate, and probabilities for the answer type of this instance, respectively. One of the probability,  $p_s(s \mid \mathbf{c}, S)$ , is computed as follow, and the others are similar to it:
$$
p _ {s} (s \mid \mathbf {c}, S) = \operatorname {s o f t m a x} \left(f _ {s} (s, \mathbf {c}, S; \theta)\right),
$$
where  $f_{s}$  is a scoring function, derived from the last layer of graph encoder. Similarly, we derive score functions at the other three levels of granularity. For instances without short answers, we set the target start and end indices to the “[CLS)” token. We also make “[CLS)” markup as the first sentence and paragraph, and the paragraph-level “[CLS)” will be classified as long answers for the instances without long answers. At inference time, we get the score of a document fragment  $g(\mathbf{c}, S)$ , long answer score  $g(\mathbf{c}, S, l)$  and short answer score  $g(\mathbf{c}, S, s, e)$
6712
Long Answer Dev	Long Answer Test	Short Answer Dev	Short Answer Test
P	R	F1	P	R	F1	P	R	F1	P	R	F1
DocumentQA	47.5	44.7	46.1	48.9	43.3	45.7	38.6	33.2	35.7	40.6	31.0	35.1
DecAtt + DocReader	52.7	57.0	54.8	54.3	55.7	55.0	34.3	28.9	31.4	31.9	31.1	31.5
BERTjoint	61.3	68.4	64.7	64.1	68.3	66.2	59.5	47.3	52.7	63.8	44.0	52.1
+ 4M synthetic data	62.3	70.0	65.9	65.2	68.4	66.8	60.7	50.4	55.1	62.1	47.7	53.9
BERT-syn+Model-III	72.4	73.0	72.7	-	-	-	60.1	54.1	56.9	-	-	-
+ ensemble 3 models	74.2	73.6	73.9	73.7	75.3	74.5	64.0	54.9	59.1	62.6	55.3	58.7
Single Human	80.4	67.6	73.4	-	-	-	63.4	52.6	57.5	-	-	-
Super-annotator	90.0	84.6	87.2	-	-	-	79.1	72.6	75.7	-	-	-
Table 1: Results of our best model on NQ compared to the previous systems and to the performance of a single human annotator and of an ensemble of human annotators. The previous systems include DocumentQA (Clark and Gardner, 2018), DecAtt + DocReader (Parikh et al., 2016; Chen et al., 2017), BERT $_{\text{joint}}$  and BERT $_{\text{joint}} + 4\text{M}$  synthetic data (Alberti et al., 2019a).
$$
\begin{array}{l} g (\mathbf {c}, S) = f _ {t} (t > 0, \mathbf {c}, S; \theta) \\ - f _ {t} (t = 0, \mathbf {c}, S; \theta); \\ \end{array}
$$
$$
\begin{array}{l} g (\mathbf {c}, S, l) = f _ {l} (l, \mathbf {c}, S; \theta) \\ - f _ {l} (l = [ \mathrm {C L S} ], \mathbf {c}, S; \theta); \\ \end{array}
$$
$$
\begin{array}{l} g (\mathbf {c}, S, s, e) = f _ {s} (s, \mathbf {c}, S; \theta) + f _ {e} (e, \mathbf {c}, s; \theta) \\ - f _ {s} (s = [ \mathrm {C L S} ], \mathbf {c}, S; \theta) \\ - f _ {e} (e = [ \mathrm {C L S} ], \mathbf {c}, S; \theta). \\ \end{array}
$$
We use the sum of  $g(\mathbf{c}, S, l)$  and  $g(\mathbf{c}, S)$  to select a long answer candidate with highest score.  $g(\mathbf{c}, S)$  is considered as a bias term for document fragments. Then we use  $g(\mathbf{c}, S, s, e)$  to select the final short answer within the selected long answer span. We rely on the official NQ evaluation script to set thresholds to separate the predictions to positive and negative on both long and short answer.
In this section, we will first describe the data preprocessing details, then give the experimental results and analysis. We also conduct an error analysis and two case studies in the appendix.
4.1 Data Preprocessing Details
We ignore all the HTML tags as well as tokens not belonging to any long answer candidates. The average length of documents is approximately 4,500 tokens after this process. Following Devlin et al. (2019) and Alberti et al. (2019b), we first tokenize questions and documents using a 30,522 wordpiece vocabulary. Then we slide a window of a certain length over the entire length of the document with a stride of 128 tokens, generating a list of document fragments. There are about 7 paragraphs and 18 sentences on average per document
fragment. We add special markup tokens at the beginning of each long answer candidate according to the content of the candidate. The special tokens we introduced are of the form “[Paragraph=N]”, “[Table=N]” and “[List=N]”. According to Alberti et al. (2019b), this decision was based on the observation that the first few paragraphs and tables in the document are more likely to contain the annotated answer. We generate 30 instances on average per NQ example, and each instance will be processed independently during the training phase.