SlowGuess/NAACL_2021_OCR huggingface dataset:
Liangming Pan<sup>1</sup> Wenhu Chen<sup>2</sup> Wenhan Xiong<sup>2</sup> Min-Yen Kan<sup>1</sup> William Yang Wang<sup>2</sup>
<sup>1</sup>School of Computing, National University of Singapore, Singapore
$^{2}$ University of California, Santa Barbara, CA, USA
e0272310@u.nus.edu
{wenhuchen, xwhan, william}@cs.ucsb.edu
kanmy@comp.nus.edu.sg
Obtaining training data for multi-hop question answering (QA) is time-consuming and resource-intensive. We explore the possibility to train a well-performed multi-hop QA model without referencing any human-labeled multi-hop question-answer pairs, i.e., unsupervised multi-hop QA. We propose MQA-QG, an unsupervised framework that can generate human-like multi-hop training data from both homogeneous and heterogeneous data sources. MQA-QG generates questions by first selecting/generating relevant information from each data source and then integrating the multiple information to form a multi-hop question. Using only generated training data, we can train a competent multi-hop QA which achieves  $61\%$  and  $83\%$  of the supervised learning performance for the HybridQA and the HotpotQA dataset, respectively. We also show that pretraining the QA system with the generated data would greatly reduce the demand for human-annotated training data. Our codes are publicly available at https://github.com/teacherpeterpan/Unsupervised-Multi-hop-QA.
Extractive Question Answering (EQA) is the task of answering questions by selecting a span from the given context document. Works on EQA can be divided into the single-hop (Rajpurkar et al., 2016, 2018; Kwiatkowski et al., 2019) and multi-hop cases (Yang et al., 2018; Welbl et al., 2018; Perez et al., 2020). Unlike single-hop QA, which assumes the question can be answered with a single sentence or document, multi-hop QA requires combining disjoint pieces of evidence to answer a question. Though different well-designed neural models (Qiu et al., 2019; Fang et al., 2020) have achieved near-human performance on the multi-hop QA datasets (Welbl et al., 2018; Yang et al., 2018), these approaches rely heavily on the
Figure 1: An overview of our approach for generating bridge-type multi-hop questions from table and text. The full set of supported input types and question types are described in Section 3.2.
availability of large-scale human annotation. Compared with single-hop QA datasets (Rajpurkar et al., 2016), annotating multi-hop QA datasets is significantly more costly and time-consuming because a human worker needs to read multiple data sources in order to propose a reasonable question.
To address the above problem, we pursue a more realistic setting, i.e., unsupervised multi-hop QA, in which we assume no human-labeled multi-hop question is available for training, and we explore the possibility of generating human-like multi-hop question-answer pairs to train the QA model. We study multi-hop QA for both the homogeneous case where relevant evidence is in the textual forms (Yang et al., 2018) and the heterogeneous case where evidence is manifest in both tabular and textual forms (Chen et al., 2020b). Though successful attempts have been made to generate single-hop question-answer pairs by style transfer (Lewis et al., 2019) or linguistic rules (Li et al.,
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5866-5880 June 6-11, 2021. ©2021 Association for Computational Linguistics
2020), these methods are not directly applicable to the multi-hop setting as: 1) they cannot integrate information from multiple data sources, and 2) they only handle free-form text but not heterogeneous sources as input contexts.
We propose Multi-Hop Question Generator (MQA-QG), a simple yet general framework that decomposes the generation of a multi-hop question into two steps: 1) selecting relevant information from each data source, 2) integrating the multiple information to form a question. Specifically, the model first defines a set of basic operators to retrieve / generate relevant information from each input source or to aggregate different information. Afterwards, we define six reasoning graphs. Each corresponds to one type of multi-hop question and is formulated as a computation graph built upon the operators. We generate multi-hop question-answer pairs by executing the reasoning graph. Figure 1 shows an example of generating a table-to-text question: a) Given the inputs of (table, text), the FindBridge operator locates a bridge entity that connects the contents between table and text. b) We generate a simple, single-hop question for the bridge entity from the text (QGwithEnt operator) and generate a sentence describing the bridge entity from the table (DescribeEnt operator). c) The BridgeBlend operator blends the two generated contents to obtain the multi-hop question.
We evaluate our method on two multi-hop QA datasets: HotpotQA (Yang et al., 2018) and HybridQA (Chen et al., 2020b). Questions in HotpotQA reason over multiple texts (homogeneous data), while questions in HybridQA reason over both table and text (heterogeneous data). The experiments show that MQA-QG can generate high-quality multi-hop questions for both datasets. Without using any human-labeled examples, the generated questions alone can be used to train a surprisingly well QA model, reaching  $61\%$  and  $83\%$  of the F1 score achieved by the fully-supervised setting on the HybridQA and HotpotQA dataset, respectively. We also find that our method can be used in a few-shot learning setting. For example, after pretraining the QA model with our generated data, we can obtain 64.6 F1 with only 50 labeled examples in HotpotQA, compared with 21.6 F1 without the warm-up training.
- To the best of our knowledge, this is the first work to investigate unsupervised multi-hop QA.
- We propose MQA-QG, a novel framework to generate high-quality training data without the need to see any human-annotated multi-hop question.
- We show that the generated training data can greatly benefit the multi-hop QA system in both unsupervised and few-shot learning settings.
Unsupervised Question Answering. To reduce the reliance on expensive data annotation, Unsupervised/Zero-Shot QA has been proposed to train question answering models without any human-labeled training data. Lewis et al. (2019) proposed the first unsupervised QA model which generates synthetic (context, question, answer) triples to train the QA model using unsupervised machine translation. However, the generated questions are unlike human-written questions and tend to have a lot of lexical overlaps with the context. To address this, followup works utilized the Wikipedia cited documents (Li et al., 2020), predefined templates (Fabbri et al., 2020), or pretrained language model (Puri et al., 2020) to produce more natural questions resembling the human-annotated ones.
However, all the existing studies are focused on the SQuAD (Rajpurkar et al., 2016) dataset to answer single-hop and text-only questions. These methods do not generalize to multi-hop QA because they lack integrating and reasoning over disjoint pieces of evidence. Furthermore, they are restricted to text-based QA without considering structured or semi-structured data sources such as KB and Table. In contrast, we propose the first framework for unsupervised multi-hop QA, which can reason over disjoint structured or unstructured data to answer complex questions.
Multi-hop Question Generation. Question Generation (QG) aims to automatically generate questions from textual inputs (Pan et al., 2019). Early work of Question Generation (QG) relied on syntax rules or templates to transform a piece of given text to questions (Heilman, 2011; Chali and Hasan, 2012). With the proliferation of deep learning, QG evolved to use supervised neural models, where most systems were trained to generate questions from (passage, answer) pairs in the SQuAD dataset (Du et al., 2017; Zhao et al., 2018; Kim et al., 2019).
With the advent of pretraining language models (Dong et al., 2019), the challenge of generating single-hop questions similar to SQuAD
Group	Operator	Inputs → Outputs	Description
Selection	FindBridge	(Table T, Text D) or Texts (D1, D2) → Bridge Entities EB	Select an entity EB that links the two input texts D1 and D2 (or links the table T and the text D)
FindComEnt	Text D → Comparative Entities EC	Extract potential comparative entities from the input text (location, datetime, number, etc.).
Generation	QGwithAns	(Text D, Answer A) → Question Q	Generate a single-hop question Q with answer A from the input text D
QGwithEnt	(Text D, Entity E) → Question Q	Generate a single-hop question Q that contains the given entity E from the input text D
DescribeEnt	(Table T, Entity E) → Sentence S	Generate a sentence S that describes the given entity E based on the information of the table T
QuestToSent	Question Q → Sentence S	Convert a question Q into its declarative form S
Fusion	BridgeBlend	(Question Q, Sentence S, Bridge EB) → Bridge-type multi-hop question QB	Generate a bridge-type multi-hop question QB by fusing the single-hop question Q and the sentence S given the entity EB as the bridge
CompBlend	(Question Q1, Question Q2) → Comparative multi-hop question QC	Generate a comparison-type multi-hop question QC by fusing two single-hop questions
Table 1: The 8 basic operators for MQA-QG, categorized into 3 groups. Selection: retrieve relevant information from contexts. Generation: generate information from a single context. Fusion: fuse retrieved/generated information to construct multi-hop questions. Each operator is defined as a function mapping  $f(X) \to Y$ .
have largely been addressed. QG research has started to generate more complex questions that require deep comprehension and multi-hop reasoning (Tuan et al., 2020; Pan et al., 2020; Xie et al., 2020; Yu et al., 2020). For example, Tuan et al. (2020) proposed a multi-state attention mechanism to mimic the multi-hop reasoning process. Pan et al. (2020) parsed the input passage as a semantic graph to facilitate the reasoning over different entities. However, these supervised methods require large amounts of human-written multi-hop questions as training data. Instead, we propose the first unsupervised QG system to generate multi-hop questions without the need to access those annotated data.
The setup of Multi-hop QA is as follows. Given a question  $q$  and a set of input contexts  $\mathcal{C} = \{C_1, \dots, C_n\}$ , where each context  $C_i$  can be a passage, table, image, etc., the QA model  $p_{\theta}(a|q, \mathcal{C})$  predicts the answer  $a$  for the question  $q$  by integrating and reasoning over information from  $\mathcal{C}$ .
In this paper, we consider two-hop questions and denote the required contexts as  $C_i$  and  $C_j$ . Formally, each time our model takes as inputs  $\langle C_i, C_j \rangle$  to generate a set of  $(q, a)$  pairs. We focus on two modalities: the heterogeneous case where  $C_i, C_j$  are table and text and the homogeneous case where  $C_i, C_j$  are both texts. However, the design of our framework is flexible enough to generalize to multi-hop QA for other modalities.
Our model MQA-QG consists of three compo
nents: operators, reasoning graphs, and question filtration. Operators are atomic operations implemented by rules or off-the-shelf pretrained models to retrieve, generate, or fuse relevant information from input contexts  $(C_i,C_j)$ . Different reasoning graphs define different types of reasoning chains for multi-hop QA with the operators as building blocks. Training  $(q,a)$  pairs are generated by executing the reasoning graphs. Question filtration removes irrelevant and unnatural  $(q,a)$  pairs to give the final training set  $\mathcal{D}$  for multi-hop QA.
3.1 Operators
In Table 1, we define eight basic operators and divide them into three types: 1) selection: retrieve relevant information from a single context, 2) generation: generate information from a single context, and 3) fusion: fuse multiple retrieved/generated information to construct multi-hop questions.
- FindBridge: Most multi-hop questions rely on the entities that connect different input contexts, i.e., bridge entities, to integrate multiple pieces of information (Xiong et al., 2019). FindBridge takes two contexts  $(C_i, C_j)$  as inputs, and extracts the entities that appear in both  $C_i$  and  $C_j$  as bridge entities. For example, in Figure 1, we extract "Jenson Button" as the bridge entity.
- FindComEnt: When generating comparative-type multi-hop questions, we need to decide what property to compare for the bridge entity. FindComEnt extracts potential comparative properties
Figure 2: The implementation of DescribeEnt operator.
from the input text. We extract entities with NER types Nationality, Location, DateTime, and Number from the input text as comparative properties (cf, "Comparison" in Figure 4).
- QGwithAns, QGwithEnt: These two operators generate simple, single-hop questions from a single context, which are subsequently used to compose multi-hop questions. We use the pretrained Google T5 model (Raffel et al., 2019) fine-tuned on SQuAD to implement these two operators. Given the SQuAD training set of context-question-answer triples  $\mathcal{D} = \{(c,q,a)\}$ , we jointly fine-tune the model on two tasks. 1) QGwithAns aims to generate a question  $q$  with  $a$  as the answer, given  $(c,a)$  as inputs. 2) QGwithEnt aims to generate a question  $q$  that contains a specific entity  $e$ , given  $(c,e)$  as inputs. The evaluation of this T5-based model can be found in Appendix A.1.
- DescribeEnt: Given a table  $T$  and a target entity  $e$  in the table, the DescribeEnt operator generates a sentence that describes the entity  $e$  based on the information in the table  $T$ . We implement this using the GPT-TabGen model (Chen et al., 2020a) shown in Figure 2. The model first uses template to flatten the table  $T$  into a document  $P_T$  and then feed  $P_T$  to the pre-trained GPT-2 model (Radford et al., 2019) to generate the output sentence  $Y$ . To avoid irrelevant information in  $P_T$ , we apply a template that only describes the row where the target entity locates. We then finetune the model on the ToTTo dataset (Parikh et al., 2020), a large-scale dataset of controlled table-to-text generation, by maximizing the likelihood of  $p(Y|P_T; \beta)$ , with  $\beta$  denoting the model parameters. The implementation details and the model evaluation are in Appendix A.1.
- QuesToSent: This operator converts a question  $q$  into its declarative form  $s$  by applying the linguistic rules defined in Demszky et al. (2018).
Figure 3: An example of the BridgeBlend operator.
- BridgeBlend: The operator composes a bridgetype multi-hop question based on: 1) a bridge entity  $e$ , 2) a single-hop question  $q$  that contains  $e$ , and 3) a sentence  $s$  that describes  $e$ . As exemplified in Figure 3, we implement this by applying a simple yet effective rule that replaces the bridge entity  $e$  in  $q$  with "the [MASK] that  $s$ " and employ the pretrained BERT-Large (Devlin et al., 2019) to fill in the [MASK] word.
- CompBlend: This operator composes a comparison-type multi-hop question based on two single-hop questions  $q_{1}$  and  $q_{2}$ . The two questions ask about the same comparative property  $p$  for two different entities  $e_{1}$  and  $e_{2}$ . We form the multi-hop question by filling  $p$ ,  $e_{1}$ , and  $e_{2}$  into pre-defined templates (Further details in Appendix A.2).
3.2 Reasoning Graphs
Based on the basic operators, we define six types of reasoning graphs to generate questions with different types. Each reasoning graph is represented as a directed acyclic graph (DAG)  $\mathcal{G}$ , where each node in  $\mathcal{G}$  corresponds to an operator. A node  $s_i$  is connected by an incoming edge  $\langle s_j, s_i \rangle$  if the output of  $s_j$  is given as an input to  $s_i$ .
As shown in Figure 4, Table-Only and Text-Only represent single-hop questions from table and text, respectively. The remaining reasoning graphs define four types of multi-hop questions. 1) Table-to-Text: bridge-type question between table and text, where the answer comes from the text. 2) Text-to-Table: bridge-type question between table and text, where the answer comes from the table. 3) Text-to-Text: bridge-type question between two texts. 4) Comparison: comparison-type question based on two passages. These four reasoning chains can cover a large portion of questions in existing multi-hop QA datasets, such as HotpotQA and HybridQA. We generate QA pairs by executing each reasoning graph. Our framework can easily extend to other modalities and reasoning chains by defining new operators and reasoning graphs.
Table-Only
Table-to-Text
Text-Only
Text-to-Table
Text-to-Text
Figure 4: The 6 types of reasoning graphs for MQA-QG. Each graph is represented as a DAG of operators.
3.3 Question Filtration
Finally, we employ two methods to refine the quality of generated QA pairs. 1) Filtration. We use a pretrained GPT-2 model to filter out those questions that are disfluent or unnatural. The top  $N$  samples with the lowest perplexity scores are selected as the generated dataset to train the multi-hop QA model. 2) Paraphrasing. We train a question paraphrasing model based on the BART model (Lewis et al., 2020) to paraphrase each generated question. Our experiments show that filtration brings noticeable improvements to the QA model. However, we show in Section 4.5 that paraphrasing produces more human-like questions but introduces the semantic drift problem that harms the QA performance.
We evaluate our framework on two multi-hop QA datasets: HotpotQA (Yang et al., 2018) and HybridQA (Chen et al., 2020b). HotpotQA focuses on multi-hop QA over homogeneous inputs, while HybridQA deals with multi-hop QA over heterogeneous information. HotpotQA contains  $\sim 100\mathrm{K}$  crowd-sourced multi-hop questions, where each question requires reasoning over two supporting Wikipedia documents to infer the answer. HybridQA contains  $\sim 70\mathrm{K}$  human-labeled multi-hop questions, where each question is aligned with a structured Wikipedia table and multiple passages linked with the entities in the table. The questions
Split	Train	Dev	Test	Total
HotpotQA
Bridge	72,991	5,918	-	78,909 (81 %)
Comparison	17,456	1,487	-	18,943 (19 %)
Total	90,447	7,405	-	97,852
HybridQA
In-Passage	35,215	2,025	2,045	39,285 (56 %)
In-Table	26,803	1,349	1,346	29,498 (43 %)
Compute	664	92	72	828 (1.1 %)
Total	62,682	3,466	3,463	69,611
Table 2: Basic statistics of HotpotQA and HybridQA.
are designed to aggregate both tabular information and text information, i.e., lack of either form renders the question unanswerable.
Table 2 shows the statistics of these two datasets and Appendix B.1 gives their data examples. There are two types of multi-hop questions in HotpotQA: bridge-type (81%) and comparison-type (19%). For HybridQA, questions are divided by whether their answers come from the table (In-Table question, 56%) or from the passage (In-Passage question, 44%). Around 80% HybridQA questions require bridge-type reasoning.
4.1 Unsupervised QA Results
Question Generation. In HybridQA, we extract its table-text corpus consisting of  $(T,D)$  input pairs, where  $T$  denotes the table and set of its linked passages  $D$ . We generate two multi-hop QA datasets  $\mathcal{Q}_{tbl\rightarrow txt}$  and  $\mathcal{Q}_{txt\rightarrow tbl}$  with MQA-QG by executing the "Table-to-Text" and "Text-to-Table" reasoning graphs for each  $(T,D)$ , resulting in a total of 170K QA pairs. We then apply question filtration to obtain the training set  $\mathcal{Q}_{hybrid}$  with 100K QA pairs. Similarly, for HotpotQA, we first generate  $\mathcal{Q}_{bge}$  and  $\mathcal{Q}_{com}$ , which contains only the bridge-type questions and only the comparison-type questions, respectively. Afterward, we merge them and filter the questions to obtain the final training set  $\mathcal{Q}_{hotpot}$  with 100K QA pairs. In Appendix B.2, we gives the statistics of all the generated datasets.
Question Answering For HybridQA, we use the HYBRIDER (Chen et al., 2020b) as the QA model, which breaks the QA into linking and reasoning to cope with heterogeneous information, achieving the best result in HybridQA. For HotpotQA, we use the SpanBERT (Joshi et al., 2020) since it achieved promising results on HotpotQA with reproducible codes. We use the standard Exact Match (EM) and  $F_{1}$  metrics to measure the QA performance.
Baselines. We compare MQA-QG with both supervised and unsupervised baselines. For HybridQA, we first include the two supervised baselines Table-Only and Passage-Only in Chen et al. (2020b), which only rely on the tabular information or the textual information to find the answer. As we are the first to target unsupervised QA on HybridQA, there is no existing unsupervised baseline for direct comparison. Therefore, we construct a strong baseline QDMR-to-Question that generate questions from Question Decomposition Meaning Representation (QDMR) (Wolfson et al., 2020), a logical representation specially designed for multi-hop questions. We first generate QDMR expressions from the input (table, text) using pre-defined templates and then train a Seq2Seq model (Bahdanau et al., 2014) to translate QDMR into question. Details of this baseline are introduced in Appendix C. For HotpotQA, we introduce three unsupervised baselines. SQuAD-Transfer trains SpanBERT on SQuAD and then transfers it for multi-hop QA. Bridge-Only / Comparison-Only use only the bridge-type / comparison-type questions by MQA-QG to train the QA model.
Performance Comparison. Table 3 and Table 4 summarizes the QA performance on HybridQA and HotpotQA, respectively. For HybridQA, we use the reported performance of HYBRIDER as the supervised benchmark (S3) and apply the same model setting of HYBRIDER to train the unsupervised version, i.e., using our generated QA pairs as the training data (U2 and U3). For HotpotQA, the original paper of SpanBERT only reported the results for the MRQA-2019 shared task (Fisch et al., 2019), which only includes the bridge-type questions in HotpotQA. Therefore, we retrain the SpanBERT on the full HotpotQA dataset to get the supervised benchmark (S4) and using the same model setting to train the unsupervised versions (U7 and U8).
Our unsupervised model MQA-QG attains 30.5  $F_{1}$  on the HybridQA test set and 68.6  $F_{1}$  on the HotpotQA dev set, outperforming all the unsupervised baselines (U1, U4, U5, U6) by large margins. Without using their human-annotated training data, the  $F_{1}$  gap to the fully-supervised version is only 19.5 and 14.2 for HybridQA and HotpotQA, respectively. In particular, the results of U2 and U3 even outperform the two weak supervised baselines (S1 and S2) in HybridQA. This demonstrates the effectiveness of MQA-QG in generating good multi-hop questions for training the QA model.
Model	In-Table	In-Passage	Total
EM / F1	EM / F1	EM / F1
Supervised	S1. Table-Only (Chen et al., 2020b)	14.7 / 19.1	2.4 / 4.5	8.4 / 7.1
S2. Passage-Only (Chen et al., 2020b)	9.2 / 13.5	26.1 / 32.4	19.5 / 25.1
S3. HYBRIDER (Chen et al., 2020b)	51.2 / 58.6	39.6 / 46.4	42.9 / 50.0
Unsupervised	U1. QDMR-to-Question	25.7 / 29.7	12.8 / 16.5	17.7 / 21.4
U2. MQA-QG -w/o Filtration	33.0 / 37.1	18.6 / 23.4	23.8 / 28.2
U3. MQA-QG	36.2 / 40.6	19.8 / 25.0	25.7 / 30.5
Table 3: Performance comparison between supervised models and unsupervised models on HybridQA.
Model	Bridge	Comparison	Total
EM / F1	EM / F1	EM / F1
Supervised	S4. SpanBERT (Joshi et al., 2020)	68.2 / 83.5	74.2 / 80.3	69.4 / 82.8
Unsupervised	U4. Bridge-Only	55.4 / 71.4	12.4 / 19.1	46.7 / 60.9
U5. Comparison-Only	9.8 / 14.5	38.2 / 45.0	15.5 / 20.6
U6. SQuAD-Transfer	54.6 / 69.7	25.3 / 35.2	48.7 / 62.8
U7. MQA-QG -w/o Filtration	55.2 / 71.2	44.8 / 52.9	53.1 / 67.5
U8. MQA-QG	56.5 / 72.2	48.8 / 54.4	54.9 / 68.6