SlowGuess/NAACL_2021_OCR huggingface dataset:
Yongjie Huang<sup>1</sup> and Meng Yang<sup>1,2,*</sup>
$^{1}$ School of Computer Science and Engineering, Sun Yat-sen University / Guangzhou, China
$^{2}$ Key Laboratory of Machine Intelligence and Advanced Computing,
Sun Yat-sen University, Ministry of Education / China
huangyj229@mail2.sysu.edu.cn, yangm6@mail.sysu.edu.cn
Recently Graph Neural Network (GNN) has been used as a promising tool in multi-hop question answering task. However, the unnecessary updations and simple edge constructions prevent an accurate answer span extraction in a more direct and interpretable way. In this paper, we propose a novel model of Breadth First Reasoning Graph (BFR-Graph), which presents a new message passing way that better conforms to the reasoning process. In BFR-Graph, the reasoning message is required to start from the question node and pass to the next sentences node hop by hop until all the edges have been passed, which can effectively prevent each node from over-smoothing or being updated multiple times unnecessarily. To introduce more semantics, we also define the reasoning graph as a weighted graph with considering the number of co-occurrence entities and the distance between sentences. Then we present a more direct and interpretable way to aggregate scores from different levels of granularity based on the GNN. On HotpotQA leaderboard, the proposed BFR-Graph achieves state-of-the-art on answer span prediction.
Typical Question Answering (QA) or Reading Comprehension (RC) task aims at exploring a desired answer through a single evidence document or paragraph. Recently, a more challenging multi-hop QA task, where we need to reason over multiple paragraphs to find the answer, is gradually catching attention. One example from HotpotQA dataset (Yang et al., 2018) is shown in Fig. 1.
One method for achieving multi-hop QA is to concatenate all the paragraphs together and treat it as a typical single-hop QA task (Yang et al., 2018), then existing QA techniques can be applied. Although multi-hop QA can be solved to some extent,
Question: The southern terminus of State Route 102 is located in a town on the southwestern side of an island known to locals as what?
Paragraph 1: Maine State Route 102
s1 State Route 102 is a north-south state highway.
s2 As of 2007, its southern terminus is located at an intersection with State Route 102A in Tremont on the southern part of the island.
s3 Its northern terminus is at intersection with State Route 3 in Bar Harbor, a terminus it shares with State Route 198.
Paragraph 2. Tremont, Maine
s4 Tremont is a town in Hancock County, Maine, United States, located on the southwestern side of Mount Desert Island, known to locals as "the backside."
s5 ...
Answer: the backside
Supporting sentences: s1, s2, s4
Figure 1: One example from HotpotQA dataset. "s1", "s2", ... denote the sentences in paragraphs. The model needs to find the answer and supporting sentences by reasoning over multiple sentences and paragraphs. It's obvious that the reasoning is in a ordered process from the question to "s1", "s2" and finally to "s4".
this method lacks interpretation of the reasoning process from one hop to the next hop.
Graph Neural Networks (GNN) is a natural way to represent the solving procedure of multi-hop QA. For instance, nodes in GNN represent sentences/entities in the paragraphs, and from the update through edges we can get interactive message between them, which is similar to the process of reasoning. Thus, a more reasonable method is to construct GNN to simulate the reasoning process among multiple paragraphs (Ding et al., 2019; Qiu et al., 2019; Tu et al., 2020). Promising performance has been reported in methods that designed different type of nodes or edges for GNN(De Cao et al., 2019; Tu et al., 2019, 2020; Fang et al., 2020) and the features generated from GNN has also been combined with those from the context encoder in a latent way (Qiu et al., 2019; Fang et al., 2020).
Association for Computational Linguistics: Human Language Technologies, pages 5810-5821
Despite of the success that GNN achieves in multi-hop QA, new problems associated to GNN arise. Firstly, current approaches update all the nodes, including some unnecessary ones, together within each layer, which may lead the nodes to converge to similar values and lose the discriminating ability for GNN with more layers (Kipf and Welling, 2017). Secondly, although different types of edges have been designed for GNN, there is no more fine-grained distinction between edges of the same type, without considering the other relational information between sentences. Thirdly, existing methods only latently fuse the hidden representations of GNN and context encoder, without contributing to the answer span extraction in a direct and interpretable way.
To solve the aforementioned issues, we proposed a novel model of Breadth First Reasoning Graph (BFR-Graph) to effectively adapt GNN to multihop QA. The proposed BFR-Graph is a weighted graph in which the weight of an edge is computed based on other relational information (e.g., co-occurrence entities and distance) of the connected sentences. Inspired by the Human reasoning mechanism and the Breadth First Search algorithm, in BFR-Graph the reasoning message starts from the question and passes to the next sentence nodes hop by hop until all the edges have been passed, effectively preventing each node from updating multiple times or being updated unnecessarily. Then the reasoning result from BFR-Graph is converted to the sentence scores and paragraph scores, contributing to the answer span extraction. Specifically, the final answer span probability is the sum of the score of answer span, the sentence and the paragraph, in both of which the answer is located. Experiment results show that our methods make GNN more powerful in multi-hop QA and achieves state-of-the-art on answer span prediction of HotpotQA.
- We propose BFR-Graph for multi-hop QA, which is more in line with reasoning process than existing GNNs. The reasoning message starts at the question and then reasons to the next sentences hop by hop.
- Our BFR-Graph is a weighted graph, considering the number of co-occurrence entities and the distance between sentences.
- To take advantage of the reasoning result from
BFR-Graph, multi-score mechanism is used for answer span extraction in a more direct and interpretable way.
2.1 Multi-hop QA
Serval multi-hop QA datasets have been proposed such as WikiHop (Welbl et al., 2018) and HotpotQA (Yang et al., 2018). WikiHop provides candidate answers for selection while HotpotQA needs to find an answer span over all paragraphs. Based on these datasets, several categories of multi-hop QA approaches were proposed.
Yang et al. (2018) proposed a baseline method based on RNNs and Min et al. (2019) decomposed the multi-hop question into simpler single-hop subquestion that can be answered by existing single-hop RC models. To better utilize multiple paragraphs, Nishida et al. (2019) proposed Query Focused Extractor to sequentially summarize the context and Asai et al. (2020) used a recurrent retrieval approach that learns to sequentially retrieve evidence paragraphs. Moreover, reasoning has also been conducted in multi-hop QA. Jiang and Bansal (2019) designed a neural modular network to perform unique types of reasoning; Chen et al. (2020) presented extra hop attention that can naturally hops across the connected text sequences. Qiu et al. (2019) regards the task as a two-stage task including paragraph selection and downstream model and Tu et al. (2020) further proposed a pairwise learning-to-rank loss for better interaction between paragraphs. Although the aforementioned methods are specifically designed for multi-hop QA with different structures, they lack an explicit scheme to show the reasoning process.
2.2 GNNs for Multi-hop QA
Recently GNNs such as Graph Convolution Networks (Kipf and Welling, 2017) and Graph Attention Networks (Velicković et al., 2018) show enhancement in multi-hop QA because the GNN-based methods are more intuitive and explicit.
Entity-GCN (De Cao et al., 2019) considered different type of edges and Tu et al. (2019) further built a heterogeneous graph with multiple types of nodes and edges for different granularity levels of information. Besides, Ding et al. (2019) coordinated implicit extraction and explicit reasoning through a GNN inspired by the dual process theory in cognitive science, and Tu et al. (2020) built a
Figure 2: Diagram of our system. Each node in the graph represents a sentence. The dotted line in light blue means the sentences are from the same paragraph.
GNN model for reasoning over sentence, which is summarized over token representations based on a mixed attentive pooling mechanism. Furthermore, more complex graphs is also designed. Qiu et al. (2019) proposed a Dynamically Fused Graph Network to explore along the entity graph dynamically and finds supporting entities from the context. Fang et al. (2020) created a hierarchical graph for different levels of granularity to aggregate clues from scattered texts across multiple paragraphs. However, GNNs in these methods update all the nodes together, including some unnecessary ones.
To solve the aforementioned issues, we propose a novel model of Breadth First Reasoning Graph (BFR-Graph) for multi-hop QA. Different from existing GNN-based methods, BFR-Graph introduces new restrictions on the message passing: the message only starts from the question and then passes to the latter sentence nodes hop by hop. Besides, our graph is constructed as a weighted graph considering the co-occurrence entities and distance between sentences. Moreover, multi-score answer prediction is designed to take advantage of the reasoning result from BFR-Graph. In short, we propose breadth first reasoning on the weighted graph and then combine multi-level scores for answer prediction in the framework of multi-task joint training.
The diagram of our system is shown in Fig. 2. Given multiple paragraphs, we first filter out irrelevant paragraph with paragraph selection (Sec. 3.1) and then use a BERT for context encoding (Sec. 3.2). A weighted graph is constructed
(Sec. 3.3) to reason over sentences (Sec. 3.4) and calculate the sentence score and paragraph score. Finally, we use multi-score mechanism to predict the answer span (Sec. 3.5).
3.1 Paragraph Selection
Although multiple candidate paragraphs are given for answering the question, not all of them are useful (i.e., relevant to the question). Following Qiu et al. (2019), we retrieve  $N$  useful paragraphs for each question through a straightforward way. Each candidate paragraph is concatenated with the question ("[CLS]" + question + "[SEP]" + paragraph + "[SEP]") and fed into a BERT (Devlin et al., 2019) for binary classification. After training procedure, we select paragraphs with top- $N$  score as the useful paragraphs, which are then concatenated together as context  $C$ .
3.2 Context Encoding
Following Qiu et al. (2019), we concatenate each question  $Q$  and its corresponding context  $C$ , and feed them into a BERT followed by a bi-attention layer (Seo et al., 2017) to obtain the encoded representations of question and context. The output is denoted as:
$$
\mathbf {H} = \left\{\mathbf {h} _ {0}, \dots , \mathbf {h} _ {L - 1} \right\} \in \mathbb {R} ^ {L \times d}, \tag {1}
$$
where  $L$  is the length of the input sequence (concatenating question and context), and  $d$  is the output dimension of bi-attention layer (also the dimension of BERT).
To achieve sentence-level representations, we first obtain token-level representation of each sen
Figure 3: Message passing procedure of BFR-Graph and typical GNN. Active node is the node that is reachable for its neighbors while the quiet one is on the contrary. Active edge is the passable edge while the quiet one is on the contrary.
tence:
$$
\mathbf {S} _ {i} ^ {\text {s e q}} = \mathbf {H} \left[ s _ {i} ^ {\text {s t a r t}}: s _ {i} ^ {\text {e n d}}, : \right] \in \mathbb {R} ^ {L _ {s _ {i}} \times d}, \tag {2}
$$
where  $s_i^{start}$ ,  $s_i^{end}$  are the start and end position of the sentence  $i$  respectively,  $L_{s_i}$  is the length of sentence  $i$ . Note that the question is also a sentence. Then using the method in Rei and Søgaard (2019), we get sentence representation:
$$
\mathbf {s} _ {i} = \sum_ {k = 0} ^ {L _ {s}} \alpha_ {k} ^ {i} \mathbf {S} _ {i} ^ {\text {s e q}} [ k,: ] \in \mathbb {R} ^ {d}, \tag {3}
$$
where  $\alpha_{k}^{i}$  is the weight on the  $k$ -th token of sentence  $i$ , obtained from a two-layer MLP (MultiLayer Perceptron) with output size  $= 1$ .
3.3 Weighted Graph Construction
The nodes in our weighted graph represent question  $Q$  and sentences in context  $C$ . To better exploit complex relational information between sentences, two types of correlation are defined: positive correlation and negative correlation. Although they can be designed in many ways, now we illustrate our design:
(1) Positive correlation: an edge is added if the nodes representing the sentences  $i$  and  $j$  have  $n(n \geq 1)$  of the same named entities, and the weight of the edge is:
$$
w _ {i j} = \frac {1}{1 + e ^ {- n + K _ {1}}}. \tag {4}
$$
(2) Negative correlation: otherwise, an edge is added if the two nodes are originally from the same paragraph, and the weight of the edge is:
$$
w _ {i j} = \frac {1}{1 + e ^ {d + K _ {2}}}, \tag {5}
$$
where  $d$  is the distance of the two sentences (e.g.,  $d = 1$  if the sentence is immediately followed by the other sentence in a paragraph,  $d = 2$  if there is a sentence between them, etc.).  $K_{1}$  and  $K_{2}$  are hyperparameters.
To simplify our design, we treat our graph as a homogeneous graph, which contains single type of nodes and edges.
3.4 Breadth First Reasoning
When we reason over paragraphs to answer a question, we start from the question and find the next sentence hop by hop. For a GNN where nodes represent sentences, the following message passing is unnecessary and may suppress the disturbance from useless nodes: (1) from the latter node to the former node, (2) a node haven't received the message from question but it updates other nodes.
To prevent each node from being updated multiple times unnecessarily, the reasoning message in our BFR-Graph starts from the question node and passes to the next nodes hop by hop until all the edges have been passed. Note that a node is allowed to update multiple times, depending on whether the connected edges have all been passed.
Specifically, a node  $i$  is updated by node  $j$  when the following conditions are met simultaneously: (1) node  $i$  and node  $j$  are neighbors, (2) node  $j$  is active, i.e., it is updated last layer, (3) the edge between node  $i$  and node  $j$  haven't been passed previously. The overall message passing procedure of BFR-Graph is illustrated in Algorithm 1.
Inspired by Graph Attention Networks (Velicković et al., 2018), the updating function (or message passing function) is defined as:
$$
\mathbf {s} _ {i} ^ {\prime} = \operatorname {L e a k y R e l u} \left(\sum_ {j \in \mathcal {N} _ {i} ^ {\prime}} \beta_ {i j} \mathbf {s} _ {j} \mathbf {W}\right), \tag {6}
$$
$$
\beta_ {i j} = \frac {\exp \left(f \left(\mathbf {s} _ {i} , \mathbf {s} _ {j}\right)\right) \cdot w _ {i j}}{\sum_ {k \in \mathcal {N} _ {i} ^ {\prime}} \exp \left(f \left(\mathbf {s} _ {i} , \mathbf {s} _ {k}\right)\right) \cdot w _ {i k}}, \tag {7}
$$
Fig. 3 visually shows the difference between BFR-Graph and typical GNN.
Figure 4: Multi-score answer prediction. The example is calculating the score for an answer span located in paragraph 1 ("p1") and sentence 5 ("s5").
where  $\mathcal{N}_i^\prime$  is the set of reachable neighbors for node  $i$ , calculated with Algorithm 1.  $f(\mathbf{s}_i,\mathbf{s}_j) = \mathbf{s}_i\mathbf{W}_1\mathbf{W}_2\mathbf{s}_j$  is for calculating the attention score between node  $i$  and  $j$ .  $\mathbf{W}$ ,  $\mathbf{W}_1$  and  $\mathbf{W}_2$  are learnable parameters.  $w_{ij}$  is the weight of the edge  $(i,j)$ , described in the Sec. 3.3. For clarity,  $\mathbf{s}'$  is written as  $\mathbf{s}$  in following contents.
3.5 Multi-score Answer Prediction
The answer in HotpotQA dataset is a span from the context. Existing works only calculate the span probability on the output of encoder (e.g., BERT) or additionally concatenate the GNN's hidden output. Differently, we use a more interpretable method by calculating the sentence score and paragraph score obtained from the GNN. An example is shown in Fig. 4.
Conventionally, the score of  $y$ -th word in context being the start / end of the answer span is calculated by:
$$
\phi_ {s t a r t} (y) = \operatorname {M L P} _ {1} (\mathbf {H} [ y,: ]), \tag {8}
$$
$$
\phi_ {e n d} (y) = \operatorname {M L P} _ {2} (\mathbf {H} [ y,: ]), \tag {9}
$$
where MLP is a two-layer MLP with output size  $= 1$  to obtain the score value.
Then, we calculate the sentence score corresponding to each node in GNN:
$$
\phi_ {s e n t} \left(s _ {i}\right) = \mathrm {M L P} _ {3} \left(\mathbf {s} _ {i}\right). \tag {10}
$$
Similarly, we calculate the paragraph score through a global-max-pooling:
$$
\phi_ {p a r a} (p _ {j}) = \operatorname {M L P} _ {4} (\operatorname {M a x} \left(\left\{\mathbf {s} _ {0} ^ {p _ {j}}, \dots , \mathbf {s} _ {L _ {p _ {j}} - 1} ^ {p _ {j}} \right\}\right)), \tag {11}
$$
where  $\mathbf{s}_i^{p_j}$  is the representation of the  $i$ -th sentence in paragraph  $p_j$ ,  $L_{p_j}$  is the number of sentences in paragraph  $p_j$ .  $\operatorname{Max}(\cdot)$  is a max-pooling layer with pooling size  $= L_{p_j} \times 1$ , which can also be done by taking the maximum hidden value on each dimension over all the sentence nodes.
Finally, the probability of  $y$ -th word in context being the start of the answer span is determined by:
$$
p _ {s t a r t} (y) = \mathrm {s o f t m a x} (\phi_ {s t a r t} ^ {\prime} (y)), \qquad (1 2)
$$
$$
\phi_ {s t a r t} ^ {\prime} (y) = \phi_ {s t a r t} (y) + \phi_ {s e n t} \left(s _ {i}\right) + \phi_ {p a r a} \left(p _ {j}\right), \tag {13}
$$
where the  $y$ -th word is located in sentence  $s_i$  and paragraph  $p_j$ . And the probability of  $y$ -th word in context being the end of the answer span can be calculated similarly.
In other words, if a sentence or paragraph has a higher score, the words located in it are more likely to be the answer.
3.6 Multi-task Joint Training
In addition to the answer span prediction, there are other two training tasks in HotpotQA. One is the answer type prediction task: some answers cannot be retrieved from the context, but are "Yes" or "No", so finally there are three types of answers (e.g., span, "Yes" and "No"). We use a global-max-pooling similar with Eq.(11) to compress all the nodes in the GNN and predict the answer type through a two-layer MLP.
The other task is to predict whether a sentence in the context is a support sentence (or called supporting fact in some papers) that is an evidence to the answer. Following previous works (Tu et al., 2020), we use the output of the GNN to predict the supporting sentences with a two-layer MLP.
The tasks in HotpotQA are jointly performed through multi-task learning, and the loss function is:
$$
\begin{array}{l} \mathcal {L} = \mathcal {L} _ {C E} (\hat {y} ^ {\text {s t a r t}}, y ^ {\text {s t a r t}}) + \mathcal {L} _ {C E} (\hat {y} ^ {\text {e n d}}, y ^ {\text {e n d}}) + \\ \lambda_ {1} \cdot \mathcal {L} _ {C E} (\hat {y} ^ {\text {t y p e}}, y ^ {\text {t y p e}}) + \lambda_ {2} \cdot \mathcal {L} _ {B C E} (\hat {y} ^ {\text {s p}}, y ^ {\text {s p}}), \tag {14} \\ \end{array}
$$
where  $\mathcal{L}_{CE}$  and  $\mathcal{L}_{BCE}$  denote the cross entropy and binary cross entropy loss respectively.  $\hat{y}^{start}$  denotes the logits of start position from Eq.(12) and  $y^{start}$  is the label. Similarly,  $\hat{y}^{type}$  and  $\hat{y}^{sp}$  are the logits of answer type prediction and supporting sentence prediction respectively.
The HotpotQA dataset (Yang et al., 2018) is the first explainable multi-hop QA dataset with sentence-level evidence supervision. Each sample in the dataset contains 2 gold paragraphs and 8 distracting paragraphs. Three tasks are included for evaluation: (1) answer span prediction (denoted as "Ans") that extracts a span in the paragraphs or generate "Yes"/"No"; (2) supporting sentences prediction (denoted as "Sup") that determines which sentences are evidences to the answer; (3) joint prediction (denoted as "Joint"). We submit our model to HotpotQA official leaderboard<sup>1</sup> and carry out ablation studies on the dev-set.
We also apply the main idea of BFR-Graph to the WikiHop dataset (Welbl et al., 2018), which provides candidate answers for selection while HotpotQA dataset needs to find an answer span over all paragraphs.
Implementation details can be found in Appendix A.
The experimental result on HotpotQA dataset is shown in Table 1. As a reading comprehension task, the performance of answer prediction should be emphasized. Our model improves  $0.84\%$  Ans-EM (Exact Match) than HGN-large, becoming the first model to break through  $70\%$  and achieving state-of-the-art on answer span prediction. On supporting sentence prediction and joint prediction, our model shows a close performances to HGN-large, possibly because this paper is based on the standard GNN (homogeneous graph) for simple clarification, and we just plan to prove that our algorithm can improve the performance of GNN. Existing GNN methods mostly constructed elaborate graphs for more granular expression of nodes, while our BFR-Graph solve the problem from another novel perspective. Thus, BFR-Graph is universal and can be easily applied to existing promising models (e.g., HGN) to get better results, which provides a promising direction for future research.
We also compare our model with two state-of-the-art GNN models (i.e., SAE and HGN), shown in Table 2. Both of them need to set the number of GNN layers manually while BFR-Graph can pass through all the connected nodes automatically with
<sup>1</sup>https://hotpotqa.github.io/
Model	Ans	Sup	Joint
EM	F1	EM	F1	EM	F1
Official baseline (Yang et al., 2018)	45.60	59.02	20.32	64.49	10.83	40.16
QFE (Nishida et al., 2019)	53.86	68.06	57.75	84.49	34.63	59.61
DFGN (Qiu et al., 2019)	56.31	69.69	51.50	81.62	33.62	59.82
LQR-Net (Grail et al., 2020)	60.20	73.78	56.21	84.09	36.56	63.68
SAE-large (Tu et al., 2020)	66.92	79.62	61.53	86.86	45.36	71.45
C2F-reader (Shao et al., 2020)	67.98	81.24	60.81	87.63	44.67	72.73
HGN-large (Fang et al., 2020)	69.22	82.19	62.76	88.47	47.11	74.21
BFR-Graph	70.06	82.20	61.33	88.41	45.92	74.13
Table 1: Results on HotpotQA leaderboard. "Ans", "Sup" and "Joint" denote answer span prediction, supporting sentence prediction and joint prediction, respectively.
Layers	Edges	Intuitive
SAE	manual	3 types	false
NGN	manual	7 types	false
BFR-Graph	adaptive	fine-grained	true
Table 2: Comparison with state-of-the-art GNN models. "Layers" denotes the number of GNN layers, "Edges" denotes how fine-grained the edges are, and "Intuitive" denotes whether the output of GNN can be intuitively observed.
Model	Accuracy
HDE (Tu et al., 2019)	68.1
DynSAN (Zhuang and Wang, 2019)	70.1
Path-based GCN (Tang et al., 2020)	70.8
ChainEx (Chen et al., 2019)	72.2
Longformer*	73.8
Longformer+BFR	74.4
Table 3: Results on WikiHop dev-set. Model annotated with “*” is our re-implementation.
an extremely low risk of over-smoothing (Kipf and Welling, 2017). SAE and HGN set a fixed types of edges, which is still not fine-grained enough, while BFR-Graph define different weights (can up to  $\infty$  different weights depends on the dataset) to distinguish nodes in a finer granularity. Furthermore, we can easily observe scores from GNN in an intuitive way in BFR-Graph.
Besides, Table 3 shows the results on WikiHop dev-set. When we add the breadth first reasoning graph and weights to Longformer (Beltagy et al., 2020), the performance is slightly improved, showing that our method have the ability for better reasoning.
5 Ablations and Analysis
In this section, we carry out ablation studies on HotpotQA dev-set. Table 4 shows the results of our full model and that without breadth first reasoning, weights, and multi-score. It indicates that our methods obviously improve the performance of GNN.