SlowGuess/NAACL_2021_OCR huggingface dataset:
5.1 Evaluation on Breadth First Reasoning
Table 5 shows the result by gradually replace the BFR-Graph layers with standard GNN layers. In detail, "r/p 1 layer" denotes replacing the first layer with a standard GNN layer, "r/p 2 layers" denotes the same operation for the first and second layers, etc.. We observe that the more layers to be replaced, the more severely the result drops. And when we replace 4 layers, the joint F1 drops at about  $6\%$ , meaning that it causes over-smoothing. It also reflects the severe problem of typical GNN: if it have more layers, over-smoothing is caused; if it has less layers, it cannot achieve long-path reasoning.
To further analyze why this particular approach of message passing in a breadth first reasoning fashion should result in better reasoning, we propose to calculate how many useful messages the answer sentence node received from supporting sentences: precision  $= \frac{N_{sp\&rcv}}{N_{rcv}}$ , recall  $= \frac{N_{sp\&rcv}}{N_{sp}}$ , where  $N_{rcv}$  denotes how many nodes' massages the answer sentence node received,  $N_{sp}$  denotes the number of supporting sentence (containing the question sentence here), and  $N_{sp\&rcv}$  denotes how many supporting nodes' massages the answer sentence node received.
The above-mentioned precision, recall and corresponding F1 on dev-set is shown in Table 6, where the typical GNN is a 2-layer GNN following previous works. With breadth first reasoning, the answer
Ans F1	Sup F1	Joint F1
full model	81.82	88.80	73.98
- bfr&ws&ms	80.72	87.77	72.20
Table 4: General ablation study for our full model. “-bfr” denotes a typical GNN without breadth first reasoning; “ws” and “ms” denote the weights and multi-score respectively.
Ans F1	Sup F1	Joint F1
full model	81.82	88.80	73.98
r/p 1 layer	81.86	88.49	73.80
r/p 2 layers	81.57	88.50	73.62
r/p 3 layers	80.66	87.63	72.04
r/p 4 layers	77.08	86.50	67.97
sentence could receive messages from supporting sentences with a higher precision, meaning that it can focus on useful sentences and eliminate invalid distractions. Since the restrictions on message passing in breadth first reasoning, it leads to a decrease in recall. However, it is hard to draw a PR curve or get different precision-recall results because this is not a binary classification task as we generally understand. But fortunately, BFR-Graph shows a higher F1 than the typical GNN.
5.2 Evaluation on Weights and Multi-score
Table 7 (top) presents the results with and without the weights in the GNN. “-ent” denotes removing the weights (we set the weights  $= 0.5$  rather than simply remove them) and “-dist” denotes removing the distance weights. When we remove the weights, although the answer F1 rises slightly, the supporting F1 falls to a greater extent. This shows that the proposed weights is beneficial to the supporting sentences prediction, which is directly predicted from the GNN nodes.
To our understanding, our model enhances the discrimination of edges by setting weights for them, and inevitably reduces the robustness of model. Fortunately, by designing Eqs.(4) and (5), the quantitative error will not cause the weight to increase or decrease sharply, and is still able to distinguish
Table 5: Ablations on breadth first reasoning.
Precision	Recall	F1
typical GNN	37.62	95.61	52.89
BFR-Graph	59.44	83.49	63.08
Table 6: Message passing in different style.
Ans F1	Sup F1	Joint F1
full model	81.82	88.80	73.98
- ent	81.91	88.53	73.90
- dist	81.98	88.55	73.91
- ent&dist	81.90	88.51	73.75
full model	81.82	88.80	73.98
- sent	81.73	88.75	73.97
- para	81.81	88.64	73.95
- sent&para	81.73	88.56	73.68
Table 7: Ablations on weights and multi-score.
Complexity
typical GNN	K*N*M*d
BFR-Graph	K*Nupdate*Mreach*d
Table 8: Complexities for different message passing ways on  $K$ -layer GNN with  $N$  nodes and representation dimension  $d$ , and  $M$  is the average number of neighbors for each node. For BFR-Graph,  $N_{update}$  is the number of nodes to be updated in current layer and  $M_{reach}$  is the number of neighbors for current node in current layer. For clarity, we ignore the difference between different layers and different nodes.
the difference between sentences.
For multi-score, we evaluate how the result changes if this particular way of exploiting GNN's output is replaced by traditional way. In Table 7 (bottom), “-sent” and “-para” denote removing multi-score for sentence and paragraph respectively. It indicates that both the addition of sentence scores and paragraph scores are beneficial to the performance.
5.3 Complexity Analysis
We also analyze the complexities of BFR-Graph and typical GNN, which is simply shown in Table 8. Firstly, in each layer of our BFR-Graph, only several nodes are updated by active nodes, so the number of nodes to be updated in a BFR-Graph layer is less than or equal to that in a typical GNN  $(N_{update} \leq N)$ . Secondly, for a node in a layer of BFR-Graph, it is only updated by its reachable nodes (i.e., active neighbors), so the number of reachable nodes for a node in a BFR-Graph layer is also less than or equal to that in typical GNN  $(M_{reach} \leq M)$ . Therefore, breadth first reasoning leads to lower complexity.
For GPU parallel training, we also show the actual cost of time per epoch. BFR-Graph cost 158.6 minutes per epoch, while a 2-layer and 3-layer typical GNN costs 157.5 and 165.6 minutes respec
tively. We find that BFR-Graph is always 4 layers in HotpotQA dataset, and it can even cost less time than a 3-layer typical GNN and is close to a 2-layer typical GNN.
In this paper, we proposed a novel GNN model of BFR-Graph. Specifically, the reasoning message starts from the question node and passes to the next sentences node hop by hop until all the edges have been passed. We also construct the reasoning graph as a weighted graph and present a more interpretable way to aggregate scores of different levels from GNN. On HotpotQA leaderboard, BFR-Graph achieved state-of-the-art on answer span prediction.
This work is partially supported by National Natural Science Foundation of China (Grants no. 61772568), Guangdong Basic and Applied Basic Research Foundation (Grant no. 2019A1515012029), and Youth science and technology innovation talent of Guangdong Special Support Program.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over wikipedia graph for question answering. In Proceedings of International Conference on Learning Representations.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Jifan Chen, Shih-ting Lin, and Greg Durrett. 2019. Multi-hop question answering via reasoning chains. arXiv preprint arXiv:1910.02610.
Zhao Chen, Xiong Chenyan, Rosset Corby, Song Xia, Bennett Paul, and Tiwary Saurabh. 2020. Transformer-xh: Multi-evidence reasoning with extra hop attention. In Proceedings of International Conference on Learning Representations.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 2306-2317.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 4171-4186.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2694-2703.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8823-8838.
Quentin Grail, Julien Perez, and Eric Gaussier. 2020. Latent question reformulation and information accumulation for multi-hop machine reading. https://openreview.net/forum?id=S1x63TEYvr.
Dirk Groeneveld, Tushar Khot, Mausam, and Ashish Sabharwal. 2020. A simple yet strong pipeline for HotpotQA. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8839-8845.
Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4474-4484.
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of International Conference on Learning Representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Multi-hop reading comprehension through question decomposition and rescoring. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097-6109.
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. 2019. Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2335-2345.
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6140-6150.
Marek Rei and Anders Søgaard. 2019. Jointly learning to label sentences and tokens. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6916-6923.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of International Conference on Learning Representations.
Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Is Graph Structure Necessary for Multi-hop Question Answering? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7187-7192.
Zeyun Tang, Yongliang Shen, Xinyin Ma, Wei Xu, Jiale Yu, and Weiming Lu. 2020. Multi-hop reading comprehension across documents with path-based graph convolutional network. pages 3905-3911.
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. 2020. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9073-9080.
Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, and Bowen Zhou. 2019. Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2704-2713.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of International Conference on Learning Representations.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380.
Yimeng Zhuang and Huadong Wang. 2019. Token-level dynamic self-attention network for multipassage reading comprehension. In Proceedings of
the 57th Annual Meeting of the Association for Computational Linguistics, pages 2252-2262.
We select  $N = 3$  useful paragraphs in paragraph selection, which achieves  $98.7\%$  recall in dev-set. We use RoBerta-large (Liu et al., 2019) for context encoding, with a maximum length of 512 tokens. We also fine-tune the model on SQuAD dataset similar as Groeneveld et al. (2020). We use  $\mathrm{spaCy}^2$  for named entity recognition and we found the balance factor  $K_{1} = 0$ ,  $K_{2} = -2$  lead to better result. The manual weights of the loss function are  $\lambda_1 = 1$ ,  $\lambda_{2} = 5$  in this work. The sentences number is limited to 30 and the max sentence length is set to 512 (same with BERT). We use Adam with learning rate of 1e-5, L2 weight decay of 0.01, learning rate warm-up over the first 1,000 steps and linear decay to 0. Other hyperparameters mainly follow previous works (Fang et al., 2020). We implement our model using PyTorch<sup>3</sup> and train it on RTX 2080ti GPUs.
The whole task consists of two stage training: the first stage is the paragraph selection and the second stage is the following. For the second stage, we train the model using annotated gold paragraphs, and take the predicted paragraphs from the first stage during evaluation.
More details of the dataset and metrics can be found in Yang et al. (2018). For WikiHop dataset, we migrate the breadth first reasoning and weights to a baseline model (we reimplement Longformer-base (Beltagy et al., 2020) as the baseline) and evaluate the models on the dev-set.
B Case Study and Error Analysis
In Fig.5, we provide an example for case study. The reasoning chain in this case should be divided into two parts:  $\mathrm{Q} \rightarrow \mathrm{s1} \rightarrow \mathrm{s2} \rightarrow \mathrm{s5}$  and  $\mathrm{Q} \rightarrow \mathrm{s6} \rightarrow \mathrm{s5}$ , and finally the two parts of the chain is combined together and contribute to the final answer. The complex and long reasoning chain make the question hard to answer.
As reported in Fang et al. (2020), HGN retrieved another incorrect answer span. But fortunately, our BFR-Graph can effectively deal with complex reasoning and extract a better answer through the long reasoning chain.
Figure 5: Case study.
Reference Answer: Republic of Chad
BFR-Graph: Chad (correct)
HGN: Sudan (incorrect)
Category	Percentage(%)
Annotation	10
Multiple Answers	22
Discrete Reasoning	16
External Knowledge	20
Multi-hop	16
MRC	16
Table 9: Error Analysis of our BFR-Graph.
To provide in-depth understanding of the weaknesses of our model, we carry out error analysis. Following Fang et al. (2020), we randomly sample 100 examples in the dev-set with the answer F1 as 0. Then we group the error cases into 6 categories: (1) Annotation: the reference answer is incorrect; (2) Multiple Answers: multiple correct answers can answer the question, but only one is provided in the dataset; (3) Discrete Reasoning: this type of error often appears in "comparison" questions, where discrete reasoning is required to answer the question; (4) External Knowledge: commonsense, external knowledge or mathematical operation is required; (5) Multi-hop: the model fails to perform multi-hop reasoning, and finds the final answer from wrong paragraphs; (6) MRC: the model extracts the wrong answer span but correctly finds the supporting paragraphs and sentences.
Table 9 shows the percentages of the 6 error cate
gories of our BFR-Graph. We find that many errors are due to the wrong reference answer (10%) or multiple answers (22%), which actually should not be considered as the error cases. Among other error cases, the major category of errors comes from the questions that need external knowledge (20%, including commonsense and mathematical operation), which is hard to handle without a knowledge base.
C A Case for Multi-score Prediction
Fig. 6 shows an example with specific scores when calculating multi-scores. The RoBerta-style tokens have already been converted to the BERT-style tokens for better reading.
“Token-idx” denotes the index for each token. “Para-score” and “Sent-score” denote paragraph scores and sentences scores respectively. “Start-score” and “End-score” are the scores that be the start and end of the answer span.
There are 3 paragraphs in this case (token index: 0-54, 55-108, 109-204), and the second paragraph achieve the highest paragraph score. Similarly, we can find the highest sentence score (token index: 55-79). Both (token index: 66-67) and (token index: 88-89) lead to the correct answer, with high span scores.
Question: Which German project recorded a song that featured vocals by a duo from Silverdale, England?
Target answer: Enigma
Token-idx	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
Paragraphs	A	#/#qu	#/#lo	is	an	alternative	musical	duo	from	Silver	#/#dale	,	Lanc	#/#ash	#/#ire	,	England	,	consisting
Para-score	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1,47	-1.47
Sent-score	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89
Start-score	-5.71	-9.40	-10.38	-14.01	-13.32	-8.07	-7.30	-11.96	-14.11	-12.13	-13.39	-14.19	-9.55	-10.54	-10.87	-13.10	-9.81	-12.81	-9.94
End-score	-10.92	-11.60	-6.38	-11.54	-11.80	-10.78	-9.09	-9.68	-12.01	-12.50	-11.40	-11.07	-11.55	-11.48	-11.48	-11.19	-8.66	-10.07	-10.68

19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38
of	Tom	High	#/#am	and	Ben	Fletcher	.	They	began	gaining	recognition	in	2013	for	their	single	such	as	"
-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1 .17	-1.47	-1.47	-1.47
0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42
-9.48	-7.82	-8.35	-9.29	-8.83	-7.75	-8.69	-6.75	-6.03	-8.32	-7.87	-7.90	-7.76	-7.09	-8.24	-7.70	-8.03	-8.43	-8.96	-7.16
-10.67	-10.02	-10.45	-9.05	-11.00	-10.11	-8.91	-2.39	-7.80	-7.87	-9.18	-8.63	-9.41	-7.69	-9.27	-9.65	-8.61	-8.15	-8.64	-9.95

39	40	41	42	43	44	45	46	47	48	49	50	51	52	53	54	55	56	57	58
of	Tom	High	#/#am	and	Ben	Fletcher	.	They	began	gaining	recognition	in	2013	for	their	single	such	as	"
-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1 .17	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.47	-1.19	-1.19	-1.19	-1.19
0.89	0.89	0.89	0.89	0.89	0.89	0.89	0.89	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5.42	-5 .42	-5.42	-5.42
-9.48	-7.82	-8.35	-9.29	-8.83	-7.75	-8.69	-6.75	-6.03	-8.32	-7.87	-7.90	-7.76	-7.0 9	-8.24	-7.70	-8.03	-8.43	-8.96	-7.16
-10.67	-10.02	-10.45	-9.05	-11.00	-10.11	-8.91	-2.39	-7.80	7.87	-9.18	-8.63	-9.41	-7.69	-9.27	-9.65	-8.61	-8.15	-8.64	-9.95

39	40	41	42 and 63	63	64	65	66	67	68	69	70	71	72	73	74	75	76	77	78
is	a	song	by German musical project	en	#/#gma	,	adding to	en	#/#gma	, featuring vocals	vocals by	English	dream	-	pop	duo	Aquao	#/#do
-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1.19	-1 .19	-1.19	-1.19
1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49	1.49
-7.08	-6.43	-8.43	-7.22	0.79	-4.55	-8.23	7.74	-4.57	-13.01	-9.64	-9.92	-6.94	-8.39	-10.40	-9.45	-9.59	-8.63	-3.14	-9.65
-11.40	-11.34	-10.64	-13.69	-8.36	-9.88	-9.32	-7.24	7.12	-6.61	-12.59	-10.05	-11.03	-8.69	-9.75	-11.18	-9.59	-8.45	-10.13	-2.71

79	80	81	82	83	84	85	86	87	88	89	90	91	92	93	94	95	96	97	98

It	It was released as
199	200	201	202	203	204
and	owner	of	Drone	Records	.
-1.97	-1.97	-1.97	-1.97	-1.97	-1.97
-8.64	-8.64	-8.64	-8.64	-8.64	-8.64
-7.59	-6.92	-7.59	-5.69	-7.84	-6.75
-9.36	-9.34	-9.18	-8.01	-6.69	-2.39
Figure 6: A Case for Multi-score Prediction.