SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 4: Performance comparison between supervised models and unsupervised models on HotpotQA.
Setting	Components	Reasoning Types	Performance
Text	Table	Fusion	Filtration	Table→Text	Text→Table	In-Table	In-Passage	Total
EM / F1	EM / F1	EM / F1
A1	✓	12.4 / 14.9	2.7 / 4.3	6.4 / 8.3
A2	✓	19.4 / 23.3	3.4 / 5.5	9.6 / 12.3
A3	✓	✓	14.8 / 19.2	5.6 / 7.8	9.1 / 12.1
A4	✓	✓	✓	✓	11.1 / 15.2	17.3 / 21.9	14.9 / 19.4
A5	✓	✓	✓	✓	41.5 / 47.9	0.2 / 1.9	16.2 / 19.8
A6	✓	✓	✓	✓	✓	33.0 / 37.1	18.6 / 23.4	23.8 / 28.2
A7	✓	✓	✓	✓	✓	✓	36.2 / 40.6	19.8 / 25.0	25.7 / 30.5
Table 5: Ablations on the HybridQA development set. Text/Table: whether we utilize the information in the text/table. Fusion: whether we fuse the information from table and text. Filtration: whether we perform question filtration. Reasoning Types: which types of multi-hop questions are generated.
To understand the impact of different components in MQA-QG, we perform an ablation study on the HybridQA development set. In Table 5, we compare our full model (A7) with six ablation settings by removing certain the model components (A1-A4) or by restricting the reasoning types (A5 and A6). We make three key observations.
Single-hop questions vs. multi-hop questions. A1 to A3 generates single-hop questions using the reasoning graph of Text-Only (A1), Table-Only (A2), or a union of them (A3). Afterwards, we use them to train the HYBRIDER model and test the multi-hop QA performance. In these cases, the model is trained to answer questions based on either table or text but lacking the ability to rea
son between table and text. As shown in Table 5, A1-A3 achieves a low performance of EM and F1, especially for In-Passage questions, showing that single-hop questions alone are insufficient to train a good multi-hop QA system. This reveals that learning to reason between different contexts is essential for multi-hop QA and justifies the necessity of generating multi-hop questions. However, for HotpotQA, we observe that the benefit of multi-hop questions is not as evident as in HybridQA: the SQuAD-Transfer (U6) achieves a relatively good F1 of 62.8. A potential reason is that the examples of HotpotQA contain reasoning shortcuts through which models can directly locate the answer by word-matching, without the need of multi-hop reasoning, as observed by Jiang and Bansal (2019).
(a) HybridQA
(b) HotpotQA
Effect of reasoning types. When we train the model with only the Text-to-Table questions (A5), the model achieves 47.9 F1 for In-Table questions and nearly zero performance for In-Passage questions. However, training with only the Table-to-Text questions (A4) also benefits the In-Table questions (15.2 F1). We believe the reason is that the information in the text can also answer some In-Table questions. Using both reasoning types (A6), the model improves on average by 8.6 F1 compared with the models using a single reasoning type (A4, A5). This shows that it is beneficial to train the multi-hop QA model with diverse reasoning chains.
Effect of question filtration. Question filtration also helps to train a better QA model, leading to a +2.3 F1 for HybridQA and +1.1 F1 for HotpotQA. We find that the GPT-2 based model can filter out most ungrammatical questions but would keep valid yet unnatural questions such as "Where was the event that is held in 2016 held?"
4.3 Few-shot Multi-hop QA
We then explore MQA-QG's effectiveness in the few-shot learning setting where only a few human-labeled  $(q,a)$  pairs are available. We first train the unsupervised QA model based on the training data generated by our best model. Then we fine-tune the model with limited human-labeled data. The blue line in Figure 5(a) and Figure 5(b) shows the F1 scores with different numbers of labeled training data for HybridQA and HotpotQA, respectively. We compare this with training the QA model directly on the human-labeled data without unsupervised QA pretraining (grey lines in Figure 5).
With progressively larger training dataset sizes, our model performs consistently better than the
Figure 5: The few-shot learning experiment. The figure shows the F1 score on the HybridQA (a) / HotpotQA (b) development set for progressively larger training dataset sizes. Note the difference in scales for the Y-axes.
Figure 6: Question type distribution for our generated dataset and the human-labeled dataset for HybridQA.
model without unsupervised pretraining for both two datasets. The performance improvement is especially prominent in very data-poor regimes; for example, our approach achieves 69.3 F1 with only 100 labeled examples in HotpotQA, compared with 21.4 F1 without unsupervised pretraining (47.9 absolute gain). The results show pretraining QA with MQA-QG greatly reduce the demand for human-annotated data. It can be used to provide a "warm start" for online learning QA system in which training data are quite limited for a new domain.
4.4 Analysis of Generated Questions
Although the generated questions are used to optimize for downstream QA performance, it is still instructive to examine the output QA pairs to better understand our system's advantages and limitations. In Figure 6, we plot the question type distribution for both the human-labeled dataset and the generated data for HybridQA. We find that the two datasets have a similar question type distribution, where "What" questions constitute the major type.
Type	#	Generated Question	Answer
Table-to-Text	1	When did the one that won the Eurovision Song Contest in 1966 join Gals and Pals?	1963
2	How many students attend the teams that played in the Dryden Township Conference?	1900
Text-to-Table	3	What album did the Oak Ridge Boys release in 1989?	American Dreams
4	When was the name that is the name of the bridge that crosses Youngs Bay completed?	1921
Text-to-Text	5	Which Canadian cinematographer is best known for his work on Fargo?	Craig Wrobleski
6	What is illegal in the country that is Bashar Hafez al - Assad ’s father?	Cannabis
Comp.	7	Who was born first, Terry Southern or Neal Town Stephenson?	Terry Southern
8	Are Beth Ditto and Mary Beth Patterson of the same nationality?	Yes
Table 6: Examples of multi-hop question-answers generated by MQA-QG, categorized by reasoning graphs. The two major error types are highlighted: red for inaccurate reference and blue for redundancy.
HybridQA	In-Table	In-Passage	Total
EM / F1	EM / F1	EM / F1
MQA-QG + Paraphrasing	36.2 / 40.6	19.8 / 25.0	25.7 / 30.5
37.7 / 43.5	12.1 / 15.8	21.8 / 26.2
HotpotQA	Bridge	Comparison	Total
EM / F1	EM / F1	EM / F1
MQA-QG + Paraphrasing	56.5 / 72.2	48.8 / 54.4	54.9 / 68.6
51.7 / 67.0	45.7 / 51.1	50.5 / 63.8
Table 7: Unsupervised multi-hop QA performance with/without question paraphrasing.
However, our model generates more "When" and "Where" questions but fewer "Which" questions. This is because the two reasoning graphs we apply for HybridQA are bridge-type questions while "Which" questions mostly compare.
Table 6 shows representative examples generated by our model. Most questions are fluent and exhibit encouraging language variety, such as Examples 2, 3, 5. Our model also shows almost no sign of semantic drift, meaning most of the questions are valid despite sometimes being unnatural. The two major deficiencies are inaccurate references (in red) and redundancies (in blue), shown in Examples 1, 4, 6. This can be addressed by incorporating minimal supervision to guide the fusion process; i.e., more flexible paraphrasing in fusion.
4.5 Effects of Question Paraphrasing
As discussed in Section 3.3, to generate more natural-looking questions, we attempted to train a BART-based question paraphrasing model to paraphrase each generated question. We finetune the pretrained BART model on the Quora Question Paraphrasing dataset<sup>1</sup>, which contains over 100,000 question pairs with equivalent semantic meaning.
The evaluation results are shown in Table 7. Surprisingly, we observe a performance drop for both
the HybridQA and the HotpotQA dataset, with a 4.3 and 4.8 decrease in F1, respectively. We observe that paraphrasing indeed produces more fluent questions by rewriting the redundancy parts of the original questions into more concise expression. However, paraphrasing introduces the "semantic drift" problem, i.e., the paraphrased question changes the semantic meaning of the original question. We believe this severally hurts the QA performance because it produces noisy samples with inconsistent question and answer. Therefore, we argue that in unsupervised multi-hop QA, semantic faithfulness is more important than fluency for the generated questions. This explains why we design hand-crafted reasoning graphs to ensure the semantic faithfulness. However, how to generate fluent human-like questions while keeping semantic faithfulness is an important future direction.
In this work, we study unsupervised multi-hop QA and propose a novel framework MQA-QG to generate multi-hop questions via composing reasoning graphs built upon basic operators. The experiments show that our model can generate human-like questions that help to train a well-performing multi-hop QA model in both the unsupervised and the few-shot learning setting. Further work is required to include more flexible paraphrasing at the fusion stage. We can also design more reasoning graphs and operators to generate more complex questions and support more input modalities.
This research is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative. UCSB authors are not supported by any of the above projects.
<sup>1</sup>https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Yllias Chali and Sadid A. Hasan. 2012. Towards automatic topical question generation. In International Conference on Computational Linguistics (COLING), pages 475-492.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 7929-7942.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2020b. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. CoRR, abs/1809.02922.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 4171-4186.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 13042-13054.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1342-1352.
Alexander R. Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. Template-based question generation from retrieved sentences for improved unsupervised question answering. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4508-4513.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA
2019 shared task: Evaluating generalization in reading comprehension. In The 2nd Workshop on Machine Reading for Question Answering (MRQA@EMNLP), pages 1-13.
Michael Heilman. 2011. Automatic factual question generation from text. Language Technologies Institute School of Computer Science Carnegie Mellon University, 195.
Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2726-2736.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics (TACL), 8:64-77.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. In AAAI Conference on Artificial Intelligence (AAAI), pages 6602-6609.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics (TACL), 7:452-466.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT@ACL), pages 228-231.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 7871-7880.
Patrick S. H. Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4896-4910.
Zhongli Li, Wenhui Wang, Li Dong, Furu Wei, and Ke Xu. 2020. Harvesting and refining question-answer pairs for unsupervised QA. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6719-6728.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
Bang Liu, Haojie Wei, Di Niu, Haolan Chen, and Yancheng He. 2020. Asking questions the human way: Scalable question-answer generation from text corpus. In International World Wide Web Conference (WWW).
Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent advances in neural question generation. CoRR, abs/1905.08949.
Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yen Kan. 2020. Semantic graphs for generating deep questions. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1463-1475.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 311-318.
Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. Totto: A controlled table-to-text generation dataset. CoRR, abs/2004.14373.
Ethan Perez, Patrick S. H. Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864-8880.
Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)\.
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6140-6150.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 784-789.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In *Conference on Empirical Methods in Natural Language Processing* (EMNLP), pages 2383-2392.
Luu Anh Tuan, Darsh J. Shah, and Regina Barzilay 2020. Capturing greater context for question generation. In AAAI Conference on Artificial Intelligence (AAAI).
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics (TACL), 6:287-302.
Tomer Wolfson, Mor Geva, Ankit Gupta, Yoav Goldberg, Matt Gardner, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics (TACL), 8:183-198.
Yuxi Xie, Liangming Pan, Dongzhe Wang, Min-Yen Kan, and Yansong Feng. 2020. Exploring question-specific rewards for generating deep questions. In International Conference on Computational Linguistics (COLING), pages 2534-2546.
Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Hong Wang, Shiyu Chang, Murray Campbell, and William Yang Wang. 2019. Simple yet effective bridge reasoning for open-domain multi-hop question answering. In The 2nd Workshop on Machine Reading for Question Answering (MRQA@EMNLP), pages 48-52.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2369-2380.
Jianxing Yu, Xiaojun Quan, Qinliang Su, and Jian Yin 2020. Generating multi-hop reasoning questions to improve machine reading comprehension. In International World Wide Web Conference (WWW), pages 281-291.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3901-3910.
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In CCF International Conference of Natural Language Processing and Chinese Computing (NLPCC), pages 662-671.
A Implementation Details of Operators
In this section, we give the detailed implementation of four key operators, including QGwithAns, QGwithEnt, DescribeEnt, and CompBlend. We also separately evaluate their performance.
A.1 The QGwithAns, QGwithEnt, and DescribeEnt Operators
In summary, QGwithAns, QGwithEnt are T5-based question generation model trained on the SQuAD dataset, and DescribeEnt is a GPT-2 based model trained on the ToTTo dataset.
Implementation Details For the question generation model (the QGwithAns and QGwith-Ent operators), we use the SQuAD data split from Zhou et al. (2017) to fine-tune the Google T5 model (Radford et al., 2019). We implement this based on the pretrained T5 model provided by https://github.com/patil-suraj/question_generation.
For the table-to-text generation model (the DescribeEnt operator), we adopt the GPT-TabGen model proposed in Chen et al. (2020b). The model first uses a template to flatten the input table  $T$  into a document  $P_T$  and then feed  $P_T$  to the pre-trained GPT-2 model to generate the output sentence  $Y$ . We fine-tune the model on the ToTTo dataset (Parikh et al., 2020), a large-scale dataset for controlled table-to-text generation. In ToTTo, given a Wikipedia table and a set of highlighted table cells, the objective is to produce a one-sentence description that best describes the highlighted cells. The original dataset contains 120,761 human-labeled training samples and 7,700 testing samples. To implement the DescribeEnt operator, we select the ToTTo samples that focuses on describing a given target entity  $e$  rather than the entire table, based on the following criteria: 1) the highlighted cells are in the same row and contains the target entity, 2) the description starts with the target entity. This gives us 15,135 training  $(T, e, s)$  triples and 1,194 testing triples, where  $T$  is the table,  $e$  is the target entity, and  $s$  is the target description.
Evaluation Setup We employ BLEU-4 (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), and ROUGE-L (Lin, 2004) to evaluate the performance of our implementation. For question generation, we compare the T5-based model with several state-of-the-art QG models, using their re
ported performance on the Zhou split of SQuAD. For the table-to-text generation, we compare GPTTabGen with the Seq2Seq baseline with attention.
Evaluation Results Table 8 shows the evaluation results comparing against all baseline methods. For question generation, the Google-T5 model achieves a BLEU-4 of 21.32, outperforming NQG++, S2ga-mp-gsa, and CGC-QG by large margins. This is as expected since these three baselines are based on Seq2Seq and do not apply language model pretraining. Compared with the current state-of-the-art model UniLM, the Google-T5 model achieves comparable results, with slightly lower BLEU-4 but higher METEOR. For the table-to-text generation model, we find that GPT2-TabGen outperforms Seq2Seq with attention by 5.61 in BLEU-4. When switching to GPT-2-Medium as the pretraining model, the BLEU-4 further improves by 2.04. In our final model MQA-QG, we use the Google-T5 and the GPT2-Medium in the operators.
A.2 The CompBlend Operator
The inputs of the CompBlend operator are two single-hop questions  $Q_{1}$  and  $Q_{2}$  that ask about the same comparative property  $p$ ; for example,  $Q_{1} =$  "What is the nationality of Edward Wood?",  $Q_{2} =$  "What is the nationality of Scott Derrickson", and  $p =$  "Nationality". We then identify the entity appearing in  $Q_{1}$  and  $Q_{2}$ , denoted as  $e_{1}$  and  $e_{2}$ , respectively. To form the multi-hop question, we fill in the comparing entities  $e_{1}$  and  $e_{2}$  into the corresponding templates that we define for the comparative property  $p$ . One of the resulting comparison question for the above example is "Are Edward Wood and Scott Derrickson of the same nationality?" This paper considers four comparative properties and defined a total number of 11 templates for them, summarized in Table 9.
In this section, we give further details for both the HotpotQA and the HybridQA dataset, as well as the generated datasets by our model MQA-QG.
B.1 HotpotQA and HybridQA Examples
Figure 7 gives data examples for the HotpotQA and the HybridQA dataset. The evidence used to compose the multi-hop question is highlighted, with different colors denoting information from different input contexts.
Operator	Model	BLEU-4	METEOR	ROUGE-L
QGwithAns & QGwithEnt	NQG++ (Zhou et al., 2017)	13.51	18.18	41.60
S2ga-mp-gsa (Zhao et al., 2018)	15.82	19.67	44.24
CGC-QG (Liu et al., 2020)	17.55	21.24	44.53
Google-T5 (Radford et al., 2019)	21.32	27.09	43.60
UniLM (Dong et al., 2019)	23.75	25.61	52.04
DescribeEnt	Seq2Seq Attention (Bahdanau et al., 2014)	28.31	27.61	56.63
GPT2-TabGen (Chen et al., 2020b)	33.92	32.46	55.61
GPT2-Medium (Chen et al., 2020b)	35.94	33.74	57.44