RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Dataset Synthesis using Language Models.
There have been several attempts in using LLMs to synthesize data for text classification (Ye et al., 2022 ; Meng et al., 2022 ) , semantic similarity predictions (Schick and Schütze, 2021 ; Wang et al., 2021 ) , question answering (Wang et al., 2021 ; Agrawal et al., 2022 ; Ye et al., 2022 ) , summarization (Wang et al., 2021 ) , and instruction tuning (Honovich et al., 2022 ; Wang et al., 2022c ) among others. Unlike these works where they primarily employ one data generation function for a task, our data generation process is built upon a combination of several generation functions due to the complexity of multi-hop question answering. Since our work involves finetuning models on intermediate queries, it is also related to work that finetune models on model-generated intermediate reasoning steps (Zelikman et al., 2022 ; Huang et al., 2022 ; Chung et al., 2022 ; Yao et al., 2023 ) . Different from these works, which typically assume the availability of a sizable amount of initial labeled data (e.g., question answer pairs for question answering tasks), our approach requires only a few human annotations.
Question/Query Generation.
Most prior work on automatic multi-hop question generation is cast as a generation task (Pan et al., 2020 ; Su et al., 2020 ; Sachan et al., 2020 ; Fei et al., 2022 ) , where models are trained in a supervised fashion and designed to maximize the generation metrics, such as BLEU scores (Papineni et al., 2002 ) . Before prompting LLMs becomes popular, most work attempted to generate queries for information retrieval tasks (Nogueira et al., 2019 ; Ma et al., 2021 ; Wang et al., 2022b , inter alia ) . In this line of research, Pan et al. ( 2021 ) and Qi et al. ( 2019 ) are the closest to our work. Pan et al. ( 2021 ) try to improve model performance in downstream question answering tasks by augmenting question answer pairs in the training data. Qi et al. ( 2019 ) use rule-based algorithms to find overlapping strings between sources and targets to use as queries for multi-hop questions. Although both of these works avoid directly using human supervision, they require heavily hand-crafted data generation functions, and our approach does not. There also are works that automatically generate questions for single-hop question answering (Lewis et al., 2021 ) , language model pretraining (Jia et al., 2022 ) , and passage reranking (Sachan et al., 2022 ) .
Prompting for Multi-Hop Question Answering.
Lazaridou et al. ( 2022 ) propose to condition on retrieved information through prompting LLMs.
More recent work prompts LLMs to decompose complex questions into simpler ones through either explicit queries (Press et al., 2022 ; Yao et al., 2023 ; Khattab et al., 2022 ; Khot et al., 2023 ) , integrating retrieval into the chain of thought process (Trivedi et al., 2022a ; Jiang et al., 2023 ) , or sub-questions that can be answered by dedicated question answering models (Dua et al., 2022 ) . Wang et al. ( 2022a ) and Zhou et al. ( 2023 ) iteratively prompt LLMs to elicit their parametric knowledge. Yoran et al. ( 2023 ) propose to meta-reason over multiple chains of thought instead of using a voting mechanism over the final answers.
Knowledge Distillation.
A large amount of effort has been devoted to distilling smaller models (Buciluundefined et al., 2006 ; Ba and Caruana, 2014 ; Hinton et al., 2015 ; Kim and Rush, 2016 , inter alia ) . Most recent ones seek to generate datasets (Wang et al., 2021 ) or rationals (Wang et al., 2023a ; Hsieh et al., 2023 ; Chen et al., 2023 ) from LLMs. However, unlike our work, they either focus on tasks solvable by LLMs’ parametric knowledge or assume the availability of amounts of human labeled data. Relatedly, Izacard and Grave ( 2021 ) seek to achieve better performance by distilling knowledge from LLMs to retrievers, whereas in this work, we aim to learn smaller language models and we do not finetune retrievers.
Approach
We seek to synthesize training data for multi-hop question answering using a handful of human annotations.
Our data synthesis pipeline leverages naturally-occurring relationships among documents and the powerful reasoning abilities of LLMs.
Each generated data instance contains a question, up to two queries, and an answer. We then finetune models on the generated data.

The data generation process consists of four main steps: question generation, question answering, query generation, and query verification. To achieve this, we use a frozen LLaMA 65B and parameterize the underlying data generation functions with different prompts. 1 1 1 We will leave the research on further improving model performance by iteratively finetuning on synthetic data and then synthesizing for future work.

As shown in Figure 1 Right, our approach can be broken into following steps:

Prepare document pairs
and then randomly choose answers either from context or a pre-defined list of candidates. ( Section 3.1 ) 2 2 2 While our approach can generalize to multiple documents to generate questions with more then two hops, we focus on single- and two-hop questions as prior work found that questions with more than two hops can be difficult to understand even for human readers (Press et al., 2022 ) .

Use LLMs to generate questions based on the given documents and answers. ( Section 3.2 )

Use LLMs to answer the generated questions and only keep those that are answerable. ( Section 3.3 )

Use LLMs to generate queries given the Wikipedia documents, questions, and answers. ( Section 3.4 )

Use retrievers to verify the correctness of generated queries against retrieval corpora. ( Section 3.5 )

We note that this entire process uses the same set of examples, consisting of up to 10 human-annotated data instances. We use these examples to create prompts for the tasks specified in steps 2, 3, and 4. We describe each step in detail below.
Data Preparation
During this step, our objective is to construct data tuples comprising of a pair of documents and an associated answer. To accomplish this, we employ Wikipedia pages as our primary data source, given their comprehensive coverage of knowledge.
We leverage the hyperlinks present within Wikipedia pages, along with the topics of the pages themselves, in order to generate appropriate document pairs.

To extract topics, we finetune a RoBERTa large model (Liu et al., 2019 ) on the DBPedia ontology classification dataset (Zhang et al., 2015 ) and apply the model to predict the topics of all the Wikipedia pages. 3 3 3 We use the predicted topics here as opposed to human-annotated category information associated with Wikipedia pages as this approach is more general and can be applied to other data sources without naturally-annotated category information. However, we assume there are abundant data sources for hyperlinks. We then cluster documents using the topics. Given a Wikipedia document, we create four document pairs by sampling other documents that either (1) are directly connected by hyperlinks; or (2) belong to the same topic cluster. We will refer to the first setting as “hyper” and the second as “topic”.

We select potential answers in different ways for “hyper” and “topic”. For the “hyper” setting, the candidates are from the named entities predicted by the spaCy toolkit and the anchor texts from hyperlinks. For the “topic” setting, since generated questions are mostly related to comparing the two documents, we consider the titles of both documents, “yes”, and “no” as candidate answers. We then randomly pick one from the candidate set to use in the final data tuples.
Question Generation
As shown in Figure 2 , we prompt LLMs to generate questions by providing the prepared document pairs and the associated answer. The examples in the prompt are either from prior work or randomly picked from the training set of HotpotQA, consisting of single- and two-hop questions.

Questions generated from the “topic” setting are typically related to comparison of two concepts whereas the ones from the “hyper” setting tend to be more nested in nature. In light of the different fashions, we use a separate set of examples in the prompts for the “hyper” and “topic” settings for all of our data generation functions.
We observe LLMs sometimes reference the provided context to ask questions (e.g., What is the birthplace of the man?), which is undesirable since the context will be stripped away when we finetune models on the data. So, we finetune a RoBERTa large model on the CoNLL-2003 training set (Tjong Kim Sang and
De Meulder, 2003 ) to identify named entities in the generated questions. We then drop the questions that have less than one entity in the “hyper” setting or less than two entities in the “topic” setting. We set the maximum generation step to be 64.
Question Answering
To verify the correctness of generated questions, we reformat the prompts to ask LLMs to predict answers given the generated questions and the Wikipedia document pairs (see Figure 3 for an example). We define that a question is “answerable” if its LLMs’ prediction achieve over 70 F_{1} scores 4 4 4 We compute F_{1} scores by comparing the string of predicted answers to that of ground truth answers after normalization, following Rajpurkar et al. ( 2016 ) and Yang et al. ( 2018 ) . compared to its prepared answer. We set the maximum generation step to be 16.

We also seek to use LLMs to decide whether the questions are single- or two-hop. We do so by prompting LLMs to predict answers when given (1) both documents (“both”); (2) the first document (“first”); and (3) the second document (“second”). We drop questions that are not answerable in “both”. We keep questions when the prediction from “both” agrees with that from either “second” or “first” even if they differ from the prepared answers. For these questions, we use the predicted answers as ground truths for the rest of experiments. Empirically, we observe this to be a reliable way to increase the amount of synthesized data without sacrificing the quality and these questions are in general single-hop questions.

When deciding the number of hops, we treat all the “topic” questions as two-hop questions as they mostly require comparing facts about two concepts, and use the LLMs’ predictions to decide the number of hops for “hyper”. In particular, we classify the “hyper” questions that are only answerable in “both” as two-hop questions and those that are answerable by “first” or “second” as single-hop. We will leverage this property later when post-processing generated queries.
Query Generation
As shown in Figure 4 , we prompt LLMs to generate retrieval queries given Wikipedia document pairs, generated questions, and the answers from last step. The goal is to generate a sequence of candidate queries, which will later be verified against retrieval corpora using a retriever. We also consider the original question as a candidate query in addition to the model-generated ones. The original questions are used as a backup query at the first hop, i.e., they are included only if the model-generated queries are all classified as invalid in the later verification step. We set the maximum generation step to be 64.
Query Verification
We take the query candidates and verify whether the queries can retrieve desirable documents from the entire Wikipedia document collections. In this work, we use the DRAGON retriever (Lin et al., 2023 ) and the flat index from FAISS (Johnson et al., 2019 ) . 5 5 5 Since we only use a subset of Wikipedia documents as retrieval corpus, using flat index is still efficient in our experiments. We compute similarities among documents using dot product of embedding vectors.

When verifying queries, we seek to find whether a query is valid or a duplicate to another valid query. A query is seen as valid if one of the prepared document pairs is in the top-ranked documents. Queries will be seen as duplicates if they retrieve the same document in the document pair. That is, given a prepared document pair (d_{1},d_{2}) , queries q_{1} and q_{2} , and a retrival function \text{topk}(\cdot) that returns a set of top-ranked documents given a query,

q_{i} is valid if d_{1}\in\text{topk}(q_{i}) or d_{2}\in\text{topk}(q_{i}) where i\in\{1,2\} ;

q_{1} and q_{2} are duplicates if d_{1}\in\text{topk}(q_{1})\cap\text{topk}(q_{2}) or d_{2}\in\text{topk}(q_{1})\cap\text{topk}(q_{2}) .

We drop the invalid queries and keep the shortest query if there are duplicates. We also drop questions if we fail to generate valid queries to retrieve (1) both documents for two-hop questions; or (2) the document leading to answerable predictions for single-hop questions (e.g., the first document in the document pair if the questions are answerable in the “first” setting). We drop the “hyper” questions if their answers are not in the retrieved documents at the last hop. We retrieve top 7 documents in experiments. 6 6 6 We use 7 documents to ensure enough space to include all these documents without needing to truncate them.
Extend to Fact Verification
To show that our approach can generalize to other tasks that require multi-hop reasoning, we extend our approach to the fact verification task. We follow the task setup in FEVER (Thorne et al., 2018 ) where models are asked to classify whether a claim is “supported”, “refuted”, or can not be judged due to “not enough information”.

In this setting, we also seek to generate a claim, intermediate queries, and an answer. Since facts described in a claim typically come from multiple documents that are closely related, we mostly follow the same procedure as described in previous sections except that we only consider the “hyper” document pairs. We use the same prompt for different categories as it improves model performance in our preliminary experiments. We hypothesize that this is due to the fact that FEVER is a classification task and providing different task examples within a prompt helps models learn the differences among categories. We use 8 examples in the prompts and show the complete set of prompts in Appendix C .
Training Data.
We synthesize approximately 1.5 million multi-hop questions and 1.9 million claims. We use neucleus sampling (Holtzman et al., 2020 ) with a top- p probability of 0.9 for decoding when generating the data. Development sets are 5k instances samples from each set. The dataset statistics are summarized in Table 1 .
Finetuning.
We finetune LLaMA of two parameter sizes (7B and 65B) on the generated data. During finetuning, we only compute cross-entropy losses on the query and answer strings. We also mix in plain Wikipedia text. Approximately 20% of data examples in each minibatch are plain text and we finetune LLaMA on it using vanilla language modeling loss. The finetuning and evaluation experiments are conducted separately for multi-hop QA and fact verification. The best model checkpoints are selected based on the perplexity on the synthesized development sets. We finetune models for 20k steps with a learning rate of 2e-5.
Evaluation Benchmarks.
We evaluate finetuned models on three MQA datasets (HotpotQA, MuSiQue (Trivedi et al., 2022b ) , and 2WikiQA (Ho et al., 2020 ) ) and one fact verification datasets (FEVER). For all these datasets, we use their entire official development sets as test sets. For MuSiQue, we follow Press et al. ( 2022 ) to use the subset of two-hop questions. For FEVER, we use both the development and test sets in Thorne et al. ( 2018 ) as the test set. We report the dataset sizes in Table 2 . For multi-hop question answering datasets, we report exact match (EM) and F_{1} scores. For fact verification, we report accuracies. When averaging scores across datasets, we first take the average of EM and F_{1} for the MQA datasets and then compute the overall average. Unless otherwise specified we use greedy decoding during evaluation.
Retrieval Corpus.
When generating data, we use the preprocessed Wikipedia dump from HotpotQA. For evaluation datasets, we use the preprocessed Wikipedia dumps provided with the datasets for HotpotQA and FEVER. For MuSiQue and 2WikiQA, we follow Trivedi et al. ( 2022a ) to use all the documents appeared in the datasets as their respective retrieval corpus. We summarize the number of documents for each dataset in Table 2 . We note that our retrieval corpus for MuSiQue and 2WikiQA are smaller than those reported in Trivedi et al. ( 2022a ) likely due to the difference in handling duplicate documents, where we simply pick the first document appearing in the datasets. We use the first 100 tokens 7 7 7 We use spaCy (Honnibal et al., 2020 ) tokenizer. in each Wikipedia page.
Baselines.
We compare to three kinds of baselines:

Prompting based approach: SeflAsk (Press et al., 2022 ) and DSP (Khattab et al., 2022 ) . They are the most competitive few-shot approaches that explicitly issue queries. We re-implement these two approaches using LLaMA;

Prior work on MQA question generation: Pan et al. ( 2021 ) heavily rely on hand-crafted functions to ensure the complexity of generated questions;

Prior work on query generation for MQA: Qi et al. ( 2019 ) use lexical overlap between the retrieval context and the next document(s) expected to retrieve as queries.
Compare to prior work on few-shot prompting.
We report our results and the results from prior work in Table 3 . We apply self-consistency (Wang et al., 2023b ) , which samples multiple outputs and then ensembles final predictions based on majority voting, to the finetuned models, 8 8 8 We use top- k sampling and set the temperature to be 0.7 and k to be 40. We sample 20 outputs per data instance. which results in additional improvements in model performance. We note that some of prior approaches (e.g., MCR) can be applied to our finetuned models to further improve model performance (e.g., in a way similar to self-consistency).

In general, we find that finetuning on the synthetic data significantly improves model performance for both the 7B- and 65B-parameter LLaMA. We also observe that LLaMA 7B shows much weaker performance compared to LLaMA 65B when we apply SelfAsk and DSP, which require strong in-context learning capabilities that are often missing in small language models. Interestingly, applying our approach effectively reduces the performance gap between LLaMA 7B and LLaMA 65B. While our results are not directly comparable to those from prior work (due to the differences in evaluation setup), we still include them in the table to show that with our approach LLaMA 65B achieves competitive results than prior work that employs much larger models.
Compare to prior work on few-shot multi-hop question generation.
We report results in Table 4 . We finetune LLaMA 7B on the 100k questions generated by Pan et al. ( 2021 ) . 9 9 9 Questions are downloaded from the authors’ code repository: https://github.com/teacherpeterpan/Unsupervised-Multi-hop-QA We also add the few-shot examples that are used to prompt LLMs during our data generation to the training data to ensure fair comparison. As Pan et al. ( 2021 ) do not consider intermediate queries, we also finetune LLaMA 7B on 100k questions generated in this work without using queries (“Question”). In both experiments, we retrieve top 15 documents and use the original questions as queries. We find that our generated questions lead to better performance for HotpotQA and 2WikiQA but is worse than Pan et al. ( 2021 ) on MuSiQue. Since our approach requires little effort in tuning the data generation functions, these results demonstrate the effectiveness of our approach in generating multi-hop questions. We also experiment with a “Question+Query” setting where we finetune models on both questions and their intermediate queries. We observe significant improvements and the final results outperform prior work by a large margin.
Compare to prior work on query generation.
We adapt the authors’ original implementation 10 10 10 https://github.com/qipeng/golden-retriever to generate queries for 100k question answer pairs synthesized by our approach. To measure the retrieval performance, we also report precision and recall for the retrieved documents. In particular, a query prediction is deemed as positive if the ground truth document is within the top-ranked documents. As shown in Table 5 , our approach outperforms prior rule-based approach by a significant margin.
Effect of Data and Model Sizes.
To investigate the effect of data and model sizes, we additionally finetune OPT models (Zhang et al., 2022 ) of 125M and 1.3B parameters on our synthetic datasets, and we vary the amount of the finetuning data (i.e., 0, 100k, 500k, and full). As the general trends are similar across different datasets, we report the average performance for each model when finetuned with a particular amount of data. We note that for multi-hop question answering datasets for which we have two metrics, we take the average of exact match and F1 scores as the dataset performance. The results are shown in Figure 5 . Generally, the synthetic data helps model performance, but larger models still benefit more from the finetuning. The most significant gains are from the initial 100k examples, after which the improvements start to plateau. We will leave the finding of the exact optimal amount of finetuning data for future work.
Effect of Filtering Steps.
We look into the effect of our filtering steps by finetuning LLaMA 7B models on the unfiltered question answer pairs and unfiltered queries. We report results in Table 6 . We note that the filtering step for “QA Pairs” corresponds to the question answering step, and the filtering step for the other setting corresponds to the query verification step. In the former setting, similar to previous experiments, we directly retrieve top 15 documents using input questions. In general, we find that the filtering steps help improve model performance significantly.
Effect of Diverse Relationships between Documents.
We investigate the effect of finetuning models on data generated from diverse document relationships. We report the results in Table 7 where we find that diverse document relationships improve multihop QA performance.
Conclusion
We propose a LLMs-based data synthesis framework for open domain multi-hop question answering that demands less than 10 QA pairs. The framework requires less hand-crafted features than prior work while still achieving better performance. We show that our approach is general by extending to fact verification tasks. Our results on three multi-hop question answering and one fact verification benchmarks show that our approach leads to significantly smaller models that rival the performance of previous methods. The analysis shows (1) the importance of the filtering steps and diverse relationships among documents; and (2) our approach benefits models of various sizes.
Limitations
We highlight three limitations on our work: (1) our approach depends on synthesizing large amounts of data, which are expensive even if we used LLaMA 65B which are much smaller than PaLM 540B and GPT-3.5; (2) our approach finetunes language models and thus is not applicable to the closed-source language models, e.g., GPT-3 and PaLM; and (3) our approach depends on the availability of powerful LLMs for synthesizing finetuning data.
Figure 1: An illustration of the overall pipeline of our proposed approach. Each data instance in our synthesized dataset consists of a question, queries and their corresponding retrieved documents, and an answer. We first prompt LLMs to synthesize questions and queries, finetune models on the synthesized data, and then evaluate the finetuned models on downstream tasks that require iteratively querying retrieval corpora.; An illustration of the overall pipeline of our proposed approach. Each data instance in our synthesized dataset consists of a question, queries and their corresponding retrieved documents, and an answer. We first prompt LLMs to synthesize questions and queries, finetune models on the synthesized data, and then evaluate the finetuned models on downstream tasks that require iteratively querying retrieval corpora.