Introduction
Few-shot learning for open domain multi-hop question answering seeks to answer complex questions by iteratively retrieving relevant information with a handful of human-annotated question answer pairs. It has become increasingly popular for evaluating the abilities of grounding to factual and up-to-date information (Lazaridou et al., 2022 ) and the reasoning capabilities (Press et al., 2022 ) of large language models (LLMs). Recent approaches in this area typically rely on in-context learning (Brown et al., 2020 ) where LLMs are prompted to retrieve relevant information using external search tools (Lazaridou et al., 2022 ; Press et al., 2022 ) . While powerful, the in-context learning capability usually emerges when LLMs have billions of parameters and improves as LLMs become larger in size (Wei et al., 2022 ) . This property makes LLMs expensive to experiment with even for inference.

In this work, we propose a data synthesis framework for multi-hop question answering (MQA) that allows for improving smaller language models with less than 10 human-annotated QA pairs (see Figure 1 for an overall pipeline of our approach). The framework seeks to generate MQA data using documents that are related in different aspects, e.g., sharing similar topics, providing extra information about entities, or talking about events occurred in sequence. This framework is general in that (1) the relationships among documents are naturally-occurring, covering a diverse set of reasoning types; and (2) the data generation pipeline depends on few hand-crafted, task-dependent features.

Specifically, we choose to use Wikipedia as our data sources due to its comprehensive coverage of knowledge and use hyperlinks to capture rich document relationships beyond topic similarity. We start from document pairs that are either topically similar or connected by hyperlinks, then we prompt LLMs to perform three generation tasks: question generation, question answering, and query generation.
We do so by simply changing the format of prompts while re-using the same set of QA pairs. Finally, we verify the quality of queries against retrieval corpora using a neural retriever. We also show that this framework can be easily adapted to other tasks, e.g., fact verification, as demonstrated in our experiments.

Unlike prior work on data synthesis for MQA (Pan et al., 2021 ) , which often depends on carefully designed templates to facilitate complex question generation, limiting the diversity of types of reasoning in their generation questions, our approach requires minimal hand-crafted features as it is built upon LLMs through prompting. In contrast to most work on data synthesis with LLMs (Schick and Schütze, 2021 ; Wang et al., 2021 , inter alia ) that primarily uses a single data generation function per task, our data generation process involves multiple generation functions because of the complexity of multi-hop question answering.

In experiments, we use a frozen LLaMA 65B (Touvron et al., 2023 ) to synthesize approximately 1.5 million multi-hop questions and 1.9 million claims, each of which comes with with queries and answers. To validate the effectiveness of the synthetic data, we finetune 7B- and 65B-parameter LLaMA models on it and then evaluate the finetuned models on three popular multi-hop question answering benchmarks and one fact verification dataset. Empirically, we observe that finetuning on the synthetic data drastically improves model performance, allowing our finetuned LLaMA 7B to achieve better performance than vanilla LLaMA 65B. Crucially, since the data is synthesized by LLaMA 65B, the improvement from LLaMA 65B essentially comes from the effect similar to self-training.
When comparing to prior work on question and query generation, we show that our approach achieve better performance while requiring less hand-crafted features. Analysis reveals that finetuning on the synthetic data helps models of different sizes, particularly showcasing greater benefits for smaller models. Moreover, we find that automatic filtering steps and having diverse relationships among documents are crucial in improving model performance.

To summarize, our contributions are:

We propose a novel data synthesis framework that requires less than 10 human-annotated QA pairs and minimal hand-crafted features;

We show that finetuning LLaMA models on the synthetic data can improve 19.9 points (+63.6%) and 13.2 points (+33.0%) on average for the 7B and 65B models respectively. The finetuned LLaMA 7B outperforms the prompting-based LLaMA 65B and finetuned LLaMA 65B achieves results competitive to prior work based on GPT-3.5;

We compare to prior work on MQA data generation, demonstrating that our approach achieves better performance while requiring less hand-crafted features.