Achieving Human Parity in Content-Grounded Datasets Generation
Authors: Asaf Yehudai
Boaz Carmeli
Yosi Mass
Ofir Arviv
Nathaniel Mills
Eyal Shnarch
Leshem Choshen
Venue: ICLR 2024 poster (2024)
Keywords: Data Generation
content grounded long-form question-answering
summarization
Primary area: generative models
TL;DR: Novel Method for Content-Grounded Datasets Generation Achieving Human level Performance

Abstract: The lack of high-quality data for content-grounded generation tasks has been identified as a major obstacle to advancing these tasks. To address this gap, we propose Genie, a novel method for automatically generating high-quality content-grounded data.
It consists of three stages: (a) Content Preparation, (b) Generation: creating task-specific examples from the content (e.g., question-answer pairs or summaries). (c) Filtering mechanism aiming to ensure the quality and faithfulness of the generated data. We showcase this methodology by generating three large-scale synthetic data, making wishes, for Long-Form Question-Answering (LFQA), summarization, and information extraction. In a human evaluation, our generated data was found to be natural and of high quality. Furthermore, we compare models trained on our data with models trained on human-written data -- ELI5 and ASQA for LFQA and CNN-DailyMail for Summarization. We show that our models are on par with or outperforming models trained on human-generated data and consistently outperforming them in faithfulness. Finally, we applied our method to create LFQA data within the medical domain and compared a model trained on it with models trained on other domains.

Published as a conference paper at ICLR 2024
GENIE :: ACHIEVING HUMAN PARITY IN CONTENT-GROUNDER DATASETS GENERATION
Asaf Yehudai & Boaz Carmeli 9, Yosi Mass 9, Ofir Arviv 9, Nathaniel Mills 9 Assaf Toledo 9, Eyal Shnarch $ Leshem Choshen o 0 IBM Israel Research Lab 9, Hebrew University of Jerusalem & MIT (Ase f. Yehudai, leshem. choshen] .omc com
ABSTRACT
The lack of high-quality data for content-grounded generation tasks has been iden- tified as a major obstacle to advancing these tasks. To address this gap, we pro- pose Genie, novel method for automatically generating high-quality content- grounded data. It consists of three stages: (a) Content Preparation, (b) Genera- tion: creating task-speeific examples from the content (e.g.. question-answer pairs summaries). (c) Filtering mechanism aiming to ensure the quality and faith- fulness of the generated data. We showcase this methodology by generating three large-scale synthetic data, making wishes, for Long-Form Question-Answering (LFQA), summarization, and information extraction. In a human evaluation, our generated data was found to be natural and of high quality. Furthermore, com- pare models trained on our data with models trained on human-written data ELI5 and ASQA for LFQA and CNN-DailyMail for Summarization. We show that our models are on par with or outperforming models trained on -unnnngeeeertted data and consistently outperforming them in faithfulness. Finally, we applied our method to create LFQA data within the medical domain and compared a model trained on it with models trained on other domains.
INTRODUCTION
Content-grounded generation is needed in various tasks, such as Retrieval-Augmented Generation (RAG), and content-based virtual assistants. In such tasks, the model is expected generate response based on a given content (i.e., information). For example, answer question given a document that includes information needed for the answer. Zheng et al. (2023) found those types of tasks to be the second most common use cases of Language Models.
Creating datasets with elaborate responses, which rely on long content, requires an expensive and demanding manual process. This may explain why such datasets are scarce even for popular tasks such as questionnansweenn available resources, such news providers (Hermann et al. 2015) and Reddit user posts (Fan al. 2019). This lack of high-quality content-grounder data has been identified as one the obstacles to advancing long-form QA (Stelmakh al. 2022) and domain-specinf summarization (Zhu et al. 2020), among other content- based tasks.
generation. Morcover, most existing datasets were collected from noisy
To address this gap, suggest Genie, Generate information & elucidate method for creating synthetic training data for any domain and any content-groundee task. We propose a three-step pro- cess: (a) Content Preparation, (b) Generating, and (c) Filtering. Preparation is fairly straightforward, as the data may be noisy; it is best to clean it. The generation is done using a few-shot prompting ap- proach with a large language model (LLM). See an example in App. D Finally, since the generation is automatic, we filter its outputs to ensure their faithfulness, well-formedness, and overall quality.
Genie offers flexibility and can generate synthetic data for different domains and content- montent-grounded generation tasks. We apply it to the tasks of long-form QA (LFQA), summarization ($ 3). and information extraction (IE) (App. C by creating wish-QA, wish summarizaiio,,, and wish-IF. We
We wished for cool name and that is what we've got.

---

Published as a conference paper at ICLR 2024
Content Preparation
Few-Shot Learning Prompt
Task description ExamentExample
W
H
2R
Content:Example Content Here>
Corpus
Extracted Content
LLM
Reward Model
Falthfulness Model
Task Data
Generated Task Example
Question:
dry The lowest point on dry land is the Sea,
Answer:
Jordan. 418 (1,371 feel) below sea level.
3: Filtering
2: Generation
Figure 1: Genie's 's three steps are: (1) Extract content from the source data (2) Prompt an LLM to generate task- task-specific examples based the provided content (3) filter low-quality and unfaithful examples to ensure data quality.
then show in manual evaluation that it generates high-quality data that is natural, faithful, and lexically diverse ($4). For the task LFQA, we compare the performance of models that were trained with wish QA generated by Genie to those trained with the same amount of data, generated by humans ($5). We show that the former models outperform or are on par with the latter models. Additionally, faithfulness scores show that models trained our synthetic data are more faithful to the grounding content. Those results show the overall efficacy and faithfulness of our data as training data compared to that of comann-generuee data. We replicate success with summarization, showcasing the generality of the method. We publicly release all three wishes datasets.
AUTOMATICALLY CURATING DATASET FOR CONTENT- SIOEND-GROUNDED TASKS
In Figure we illustrate Genie's three steps for automatically curating high-quality content- grounded datasets: Content Preparation, Generation, and Filtering. We refer to a content-grounded data point, like question-answer pair or summary, as example.
2.1 CONTENT PREPARATION
In the preparation step, we obtain the grounding content by extracting passages from raw documents using rule-based methods. This step the least general of approach it relies the specific format in which the data is stored. If the data already exists in easy-to-use passages, can be used is. For example, broken lines, found in table, conveniently extracted from another dataset. As the general case, we describe the extraction of content passages directly from web pages.
Implementation details. We crawled Wikipedia pages using browser emulation to allow dynamic content to be retrieved. Then we pass the full HTML DOM through filters to remove noise (c.g.. headers, footers, sidebars, etc.). We are left with the main page content which then transformed into Markdown, preserving the document structure (e.g., lists, tables. links, image references, arti- cles, and sections). From this structure a table of contents is derived and based on it we break the Markdown page into passages.
2

---

Published as a conference paper at ICLR 2024
2.2 GENERATION
In the generation step, we prompt large language model to generate a synthetic example. For that, we use the in-context capabilities the model. We prompt the model with four content-example pairs followed by the extracted content from the corpus with no example (see the prompt in appendix (D. The L.M generates a new example to match the extracted content.
Implementation details. We decode greedily, which encourages the models to produce more grounded responses (Honovich et al. 2022b). In addition, we create two variants of the data, one by generating examples using Falcon-40B (Penedo al., 2023) and another by generating with Llama-2-708B (Touvron et al., 2023). In general, the results using the different models have similar tendencies, with Llama being slightly better (see replications with Llama in an appendix {B). As Falcon is purely pre-trained, without additional steps we mainly report results relying on Falcon, to showcase that our method is not dependent on further alignment and instruction steps.
2.3 FILTERING
In the filtering step, score each content-example pair for its format, faithfulness (i.e., grounded), and quality. For each such aspect, we implement a scoring function and filter low-scoring pairs.
Format. We filter out examples where parts of the template are missing (e.g., in QA, when the prefixes signifying the start of the question or the answer are absent). Furthermore, we filter ex- amples that too short (less than ten words) or too long (surpassing 1.5 times the length of the grounding content for LFQA, and 0.25 for Summarization).
For QA, use [document], [question], and [answer] prefixes before each corresponding element. For summarization [document], [summarize], and [summary] with [summarize] representing the specific summarization instruction. It is important to note that did not fine-tune these prompts.
Faithfulness. To validate that the gendereeee examples are grounded the content, we adopt an off- fhf-thh-sheff faithfulness metric and filter low-scoring examples. When deployed with trustwor- thy data, this can serve as a measure of correctness.
We test faithfulness by mapping the problem into Textual Entailment (Dagan et al. 2005) Nat- ural I Language Inference (Bowman et al. 2015) (NLJ) problem. NLJI involves two input sentences: hypothesis and a premise. The objective is determine whether the hypothesis can be inferred from the premise, contradicts it, or is neutral with respect to it. NLI models were widely utilized for faithfulness consistency evaluation (Honovich et al., 2021 Dziri et al. 2022), and most simply by taking the grounding text as the premise and the generated example as the hypothesis (Maynez al. 2020). Here, we use the fine-tuning T5-11B NLI model presented in in/Honovich al. (2022a) for assessing the generated example faithfulness.
Quality. An important aspect of our methodology involves evaluating the quality of the generated examples, specifically quantifying their relevance to the corresponding task. Note that the task may be constant throughout a dataset (as often the case of summarization) or be dependent upon an instruction (such as the question in question answering). To judge the quality automatically we use a reward model.
Reward models are trained human preference data to give high reward for answers that human annotators prefer. Such models can quantify quality human-like way, considering dimensions that are hard to isolate and measure independently by dedicated metrics. Reward models used as quality scores for Reinforcement Learning optimization (Ouyang et al. 2022). and also serve as reference-less evaluation metries for text generation tasks (Touvron et al. (023).
Here, we use the reward model for both purposes and rely on the Open-Assistant model (Köpl et al. 2023), using the DeBERTa-v3 architecture (He et al. 2021). We filter generated examples whose score is below 0.5 by the reward model -3deradm-aoo--eeeeeaaaaa----ee
? We chose 0.5

---

Published as a conference paper at ICLR 2024
threshold based on experimentation. Similarly, use trueettrueennl nl examples deemed unfaithful by it.
model to filter
EXPERIMENTAL SETUP
Here we describe the datasets we utilized in our content- content-grounded generation tasks: LFQA and summarization ($3.1). Subsequently, we outline the various synthetic datasets we generated ($3.). and finally, discuss the models employed for training and the evaluation metries ($3.4)
3.1 DATASETS
ELIS. (Explain Like I'm Five) (Fan et al. 2019) comprises open-ended questions and extensive responses authored by users within the Reddit forum of the same name. To these questions and answers, retrieved documents were added as grounding content. In their manual analysis, they have found that the content sufficient to answer 65% of the questions and have information relevant to 92% of the questions. In this work, use the KILT version of the dataset (Petroni al. 2020
ASQA. (Answer Summaries for Questions which are Ambiguous) (Stelmakh et al. 2022 is dataset that pairs ambiguous questions from the AmbigQA dataset (Min al.) 2020) with metic- ulously crafted long-form answers generated through crowdsourcing. To add grounding they have used the same method presented in ELI5, but specifically retrieved documents from Wikipedia.
NQ. (Natural Questions) (Kwiatkowski et al., 2019) is dataset of real user questions sourced from the Google search engine. It includes questions and their corresponding passages (named long answers) from Wikipedia which provide potential answers and contain extractive short answers. This dataset does not have long-form answers, and here we will use only its documents for our synthetic data generation process $3.2 and will compare synthetic questions with the questions from NQ
CNN-DailyMail. (Kwiatkosski al. 2019) is dataset commonly used for text summarization. consists news articles from CNN and the DailyMail along with their human-written summaries.
3.2 GENERATING SYNTHETIC DATASETS
The datasets described above were used to create datasets of synthetic data;
Wish-QA-NQ. To create this dataset, draw upon NQ passages (Kwiatkowski al., 2019), for our synthetic data generation process. These passages are well-suited for our process because they were originally extracted from Wikipedia pages by annotators and typically consist of well- structured paragraphs, each centered around a specific topic.
Wish-QA ELI5/ASQA. For the creation of a dataset that mimics the conditions of ELI5 and ASQA, where answers can be derived from multiple documents, we rely on the top three retrieved passages from either the corresponding corpus. These passages are used as the grounding docu- ments for constructing this synthetic dataset.
In addition, we make a new wish dataset entirely from crawled data:
Wish-QA. stands for Wikipedia from Scratct is novel data constructed following the gen- eral approach for crawling and processing as detailed in Section $2.1 It represents a realistic data generation use case from unprocessed content. We note that the extracted passages may exhibit noise and lack of coherence and conciseness.
http://wwwggnnaccccc/gggggooeeee xxl true nli mixture
Wish-QA also the general name for all synthetic QA datasets
4

---

Published as a conference paper at ICLR 2024
3.3 MODELS FOR EXTRINSIC EVALUATION
In the Extrinsic evaluation, our goal is to compare the performance of models trained on our synthetic content-grounded data with those trained on data generated by humans. To ensure a fair comparison, we maintain an equal number of examples from each dataset (10,000) and employ identical models for training, using the same set of hyperparameters. The models we use for training are Flan-xl (Wei) et al. 2 and 1lama-2-1 1-ie--2------- 3b-Chat (Touvron et al., 2023). These models serve as the foundation for facilitating comparisons across architectural variations, including Encoder-Decoder and Decoder- only models, as well as different variations of instruction fine-tuning and alignment training.
3.4 EVALUATION METRICS
We evaluate the performance with ROUGE as a lexical similarity metric Lin 2004). as model-based reference-based metric (Zhang al. 2019b), and Reward model as model-based reference-less metric. We reuse the ANLI Faithfulness metric and reward mentioned in the filtering for evaluation. For faithfulness evaluation, we also calculate the K- K-Precision lexical similarity metric (Adlakha al. 2023). Different performance metrics (Post, 2018: Zhang al. 2019a] and more)
BERT-Score
showed similar results in initial trials, showing reliability of different forms (Perlitz et al. 2023).
ROUGE. Following the conventional approach of assessing generated text quality, including long- form answers (Fan et al. 2019). we report the ROUGE-L score (Lin 2004).
BERT Score. Zhang et al. 2019b) is a semantic smmilarity-based metric that leverages pre-trained language models to predict if the model response is semantically equivalent to the gold answer. Kasai et al. (2022) have shown BERT Score is effective in evaluating many generation tasks.
K-Precision. Following Adlakha et al. (2023) we report K-Precision, as it showed the highest correlation with human judgments from all lexical metrics. The metric follows the intuition, that in faithful response most words need to come from the content.
4 INTRINSIC EVALUATION
In this section, we perform intrinsic evaluation and validation of Wish-QA. We conduct a micro Turing Test, presenting synthetic and human questions side by side. We show that the questions generated synthetically are more natural than most of those found in available datasets. We also test the whole workflow and show that the filters contribute the generated data quality and that Genie is cost and time-efficient and creates diverse data.
Naturalness Evaluation. To assess the naturalness our guestions, we conducted a human- evaluation experiment. In the experiment, expert annotator was provided with two questions: one human-created and the other synthetic. Both questions were based on the same content. The annotator's 's task was identify the question they believed we sampled 100 questions from ELIS, ASQA, and NQ, along with their 100 synthetic counterparts.
muman-written. For this experiment,
The results in Table D (and App. $ B indicate that for ELI5, the synthetic question was selected as the numan-writted one in 72% of the cases, for NQ it was 63%, and for ASQA it was 49%. These results suggest that our synthetic questions are more natural and human-like than questions collected from sources like Reddit and Google Search engine. Additionally, they are indistinguishable from questions written by experts, such as those in the ASQA dataset. As a side finding, we also find that the ASQA dataset is of higher quality than the others, which experiments below replicate.
Multi-Dimensiin Quality Assessment. In this assessment, we aimed to investigate the qualities of the generated data and the impact of the filtration processes. We focused on the following dimen- sions: relevance and clarity of the questions, and faithfulness and overall quality of the answers. To accomplish that, we randomly selected 100 questions from the unfiltered and filtered Wish- QA. For each content-queet woweeettt-uustiinnannaaeae answer triplet, we asked annotators to answer a list of questions as shown in
A non-author, native English speaker with MA degree.
5

---

Published as a conference paper at ICLR 2024
The first two assessment questions aim to assess the relevance and clarity of the question. The clarity question is inspired by the findings of Min et al. (2020). which revealed that more than half of naturally occurring factoid questions are ambiguous. Following that, we include three ques- tions related to the answer quality. These questions are designed to ascertain whether the answer adequately addresses the question while remaining faithful to the underlying content. Lastly, ask for an overall quality rating on a 5-level Likert scale.
Table [
Human assessment results in Table I demonstrate that the filtration process had a significant impact on the relevance of the questions. Although our filtration setup does not directly assess the questions, we find that our faithfulness filter together with the reward filter provides an indirect signal about the relevance of the question. We also observed an improvement in the percentage of answers that were found to address the question. Faithfulness results show decent improvement, but there still room for enhancement. Annotators' interviews reveal that despite the presence of unfaithful cases in the dataset, their granularity was often more subtle. In some instances, the model added missing pieces of information that were subsequently found to be factually correct.
We observe slight improvement in the clarity of questions, coupled with almost all answers ad- dressing the questions. This highlights that our answer is a single relevant response from wide space of plausible answers, a well doll-documented phenomenon in LFQA (Krishna et al. (02T). Lastly, identify improvement in the overall score, which leads us to the conclusion that the filtering process substantially contributes to the quality and faithfulness of our dataset.
Table I: Multi-Dimensionall Quality assessment for synthetic data generated from scratch. Results show a large improvement in question relevance and the percentage of answers that address the question, answers that are faithful, and overall answer scores.
Quality Review Question
Wish-QA w/o filters
Wish-QA w/ filters
Is the question relevant to the content? Is the question clear? (not ambiguous) Does the answer address the question? Is the answer faithful the content? Grade the overall quality of the answer
67%
92%
63%
67%
80%
98%
53%
76%
3.48
4.58
Diversity. Our synthetic data is built on top of large-scale content that covers many different dis- tinct topics. As result, our data contain diverse lexicons. We compute vocd-D (McCarthy & Jarvis 2010) to measure the lexical diversity of our data. We found that the lexical diversity of all synthetic data is higher than their guman-generated counterparts (see Table (6). We also see that most response lengths are similar to the ones in the human writing datasets.
Scale. With 300K samples overall (full statistics in App. A). our dataset collection balances scale and quality. ELI5 is of a similar size but noisy, and ASQA carefully annotated but much smaller.
Monetary and Time Cost. Genie more cost-efficient and time-efficient than the traditional approach of crowd-sourced dataset curation. The cost of API's calls of models like the ones used typically ranges from $0.02 to S0.04, while the cost of an expert annotator to create question is approximately $4.45 (Stelmakh et al. (022). According to this rate, the 300K examples in our synthetic dataset would have cost over SIM. The time it takes to generate 10 examples is less than a minute, i.e. much faster than the time that it would take human to read the context.
EXTRINSIC EVALUATION
Finding the synthetic data to be of high quality, we test its usefulness for improving training. We present quantitative results from extrinsic experiments, evaluating models trained on synthetic and human-gennnn generated data on the ASQA and ELI5 test sets.
In Table (and App. $ B) present Flan-xl results trained on human and synthetic data. We note that here by synthetic in-domain we refer to the case where the train and test come from the same dataset, either FL.5 or ASQA.

---

Published as a conference paper at ICLR 2024
Results indicate that synthetic data is competitive alternative even when human human-gemenneeeee generated data already exists. In all cases, we see substantial gains from training on the synthetic data. For example, Rouge-L almost triples from 10.5 to 28.2 for Synthetic NQ. This gain is over the already strong multitask haseline (Flan) that trained on thousands of tasks, many of which are forms of question answering.
Moreover, the synthetic data provides better or comparable results in all metrics even for cases where train and test data come from the same dataset. While for ASQA Rouge-L and Bert-Score are slightly lower than the in-domain training data, the synthetic data is even better than the human data on the rest of the scores on ELIS. We conclude that, no human- -aman-generated data exists, automatically generating it has the potential to be as good.
ASQA performs better on both ASQA and ELI5 test sets. This observation implies that ASQA is, on the whole, a superior dataset compared to ELI5. This aligns with the substantial annotation efforts invested in the creation of ASQA, in contrast the noisy and automatically scraped ELIS data. However, it is important to note that this meticulous curation has led to a considerably smaller dataset for ASQA, totaling approximately 6k examples including the development and test sets (compared to 272k examples in EL15). This emphasizes the contribution of our approach which allows large-scale high-quality data generation.
Another strong support for the effectiveness of our data generation approach is exemplified by the model's ' outputs being favored by the preference reward model, achieving comparable or higher results than the gold standard of both datasets.
Wish-QA seems to work well even with the noisy content. Wish-QA-NQ data outperformed the synthetic in-domain data across all metrics. This can be due to the quality of the Wish-QA-NQ being favorable or that a signal document generation setup is slightly preferable.
The performance CNN-DaillMaiil, presented in Table 4 shows that Wish-summarization data improves upon the strong Flan-xl baseline, in Bert-Score and Reward score but not on ROUGE-L. Overall, the dataset seems comparable, attesting the flexibility of the method.
5.1 FAITHFULNESS RESULTS
Overall, Table 3 suggests training on our synthetic data leads to more content-faithful models. Mod- els trained on Wish-QA-Ne QN-QQ and Synthetic data, and Wish-QA QA were more faithful than those trained on ASQA and ELIS data by k-Precision and ANLI metrics. This aligns with Krishna et al. (2021), indicating LFQA models generated answers that are not grounded in the retrieved documents, and assert that this is one of the hurdles for filed progress.
Flan-xl achieves the highest Faithfulness scores followed by the synthetic datasets. Flan's 's achieve- ment can be the result of its shorter and almost extractive answers. Taking into account that it is also substantially underperforming. we deduce that the synthetic datasets achieve the best trade-off across performance and faithfulness.
The faithfulness results for CNN-DailyMail are consistently high. As we observed, Flan-xl tends to produce more extractive responses. Since CNN-DailyMail primarily contains extractive summa- rization, it's no surprise that exhibits high faithfulness scores. However, the model trained on our data, which doesn't emphasize extractiveness as hard requirement, outperforms Flan-xl in terms of k-Precision, matches in terms of NI.,, and achieves the highest average level of faithfulness.
In summary, our quantitative analysis affirms that the utilization of synthetic data substantially hances answer quality in both ASQA and ELI5 datasets. Our approach not only matches human- generated responses but also quantitatively surpasses them in terms of reward, highlighting its po- tential for generating higher-quality answers. Additionally, our method ensures high faithfulness and grounding in the generated responses, setting it apart from existing datasets.
DOMAIN ADAPTATION
We have demonstrated that method can generate synthetic data as good duman-generan data. Next, we hypothesize generating data directly in the target domain more effective than from another domain for given task. To investigate this hypothesis, define our test set as
7

---

Published as a conference paper at ICLR 2024
Table 2: Performance comparison of Flan-xl models trained human-generateed and synthetic data. The results reveal that our synthetic data consistently outperforms or achieves comparable perfor- mance to human-generated data, as indicated by ROUGE-L and Bert-Score metrics. Additionally, by reward score, models trained on our synthetic data exhibit superior or comparable performance the gold standard responses.
Test-set
ASQA
ELI5
Train Set
ROUGE-L
Bert-Score
Reward
ROUGE-L
Bert-Score F1
Reward
Flan-xl
10.5
49.7
28.8
6.2
46.7
9.2
ASQA
31.4
66.0
68.6
13.5
52.2
24.4
ELI5
18.7
58.7
37.2
13.1
51.3
11.3
Wish-QA
28.0
67.5
85.1
13.8
55.2
26.7
Wish-QA-NQ
28.2
64.8
80.3
13.2
54.0
30.3
Wish-QA in-domain
27.0
63.4
73.3
13.1
52.8
22.7
Gold
72.1
30.3
Table 3: Fiathfullnes performance Comparison of Flan-xl Models Trained Human-Created and Synthetic Data. The results demonstrate that our synthetic data consistently outperforms both human-generated data and gold responses, indicated by the k-Precision, and ANLI metrics. Flan- xl stands out with the highest scores, which can be attributed to the extractive nature of its responses.
Test-set
ASQA
ELI5
Train Set
k- k-Precision
ANLI
k- E-Precision
ANLI
Flan-xl
98.2
88.7
89.2
84.9
ASQA
67.5
55.7
52.2
34.3
ELI5
52.9
33.5
29.0
5.6
Wish- QA
77.9
74.9
58.5
37.9
Wish-QA Nash-QA-NQ
79.3
75.5
60.4
43.3
Wish in-domain
81.9
79.1
68.3
52.8
Gold
46.3
25.3
20.6
2.7
PubMed-QA, which focuses LFQA in the medical domain. Accordingly, we create synthetic auestinn-answeriig data on PubMed papers (Wish-QA-MED) as task data in the target domain. We then compare the performance of models trained Wish-QU Mish-QAA-EDD dataset with those trained on Wish-QA-NQ data, as well as with models trained on the human-created ELI5 and ASQA datasets.
The results in Table 5 demonstrate that the synthetic dataset outperforms EL15 and is comparable to or slightly better than ASQA in ROUGE-L and Bert-Score. Additionally, there is a more substantial gap in terms of reward and faithfulness.
Interestingly, Wish-QA-NQ and Wish Wish-QA-MER QA-MED achieve similar results, echoing the finding that Wish-QA outperforms other datasets. This suggests that oft-of-domain data holds little disadvantage over in-domain data and can often surpass One explanation may be that providing the content with the task (e.g. QA) makes the model rely less the training domain. Supportive evidence the finding of Onoe et al. (2023), who found that, in their task, update strategies lag behind the performance of simply concatenating the content the prompt. This may that the model relies on the content more than was previously thought (Neeman et al. 2023).
Table 4: Performance Comparison of Flan-xl Models Trained Human- Created and Wish- Summarization Data. The results reveal that our synthetic data achieves comparable performance to generatee data.
Test-set
CNN-DailyMail
Train-set
ROUGE-L
Bert-Score
Reward
k-Precision
ANLI
Flan-xl
30.2
70.9
96.3
97.6
98.7
CNN-DailyMail
33.3
72.7
96.5
97.0
99.1
Wish- Dish-Summarization
28.6
71.3
97.5
98.2
98.7
8

---

Flan-xl Published as a conference paper at ICLR 2024
Table 5: Performance of Flan-xl Models PubMed test data. The results reveal that syn- thetic data consistently outperforms or achieves comparable performance to noman-geeneatie data in general and faithfulness metrics. Results suggest that in-domain data don't t provide additional improvement for content- srotent-groonded generation, but may help the faithfulness of the model.
PubMed
Train-set
ROUGE-L
Bert-Scort
Reward
K-Precision
ANLI
12.8
53,8
10.7
60.6
38.2
ASQA
20.5
61.4
37.3
77.2
60.8
ELI5
15.0
56.3
16.8
32.2
2.2
Wish-QA-MED
22.1
61.6
39.4
78.2
81.8
Wish-QA-NQ
22.0
62.9
44.5
84.2
73.1
The faithfulness scores are inconclusive, while ANLI indicates that in-domain synthetic improves faithfulness, the k-Precision says otherwise, suggesting least parity.
We conclude that Genie can be beneficial in creating human-level data for many tasks and domains, however, it can be that I.FQA flexible terms of its training data domain. We leave for future research to check this finding and to show tasks or dimensions that exhibit improvement due to target domain data and can benefit from our method.
7 RELATED WORK
Our work is far from the first propose synthetic data for training experimentation (Choshen & Abend, 2019; Agarwal et al. 2020). Recently, generating data from a large language model to train smaller et al. (022). Our method does not focus on distillation. Apart from using a stronger model for the synthetic data, those methods differ from ours as the learned model mimics a diverse of skills, rather than becoming an expert on a task.
suggested
weak form of distillation improve smaller models (West
Still, there are few synthetic methods for specific tasks. Most notably, methods that rely on 2-step process, generation, and filtering. WWest et al. (2022) presented 2-step pipeline for Symbolic Knowledge Distillation, rather than creating content-grounded data, Kim al. (2022) apply this method to create a social dialogue dataset. In Unnatural Instructions and Self-Instruct (Honovich) ict al. b022b; Wang et al. 2022), they applied this method for the creation of an instruction dataset. Their method relies on model knowledge for content- grounnddgr tasks. Similarly, Bitton et al. (2023) q2d approach uses 2-step process for creating
ceermation-scckingg dialogs. Those works share similar mechanisms with method but differ in the content-gre srntent-groonde aspect of our work.
The dialog inpainting approach (Dai et al. 2022), shares a common objective with ours, to generate content- grounded question answering. They add questions between the document sentences to cre- ate a dialogue. This approach ensures the groundedness the dialogue but comes at the cost of less fluent and neutral conversation. In approach, we generate the question and answer using the LLM and verify its groundedness and quality to allow both faithfulness and naturalness.
DISCUSSION
Our work introduces Genie, an efficient and cost-effective automated approach for curating content- grounded datasets. Our method incorporates a novel filtering mechanism to ensure data quality. We demonstrate that our synthetic wish-QA and wish-summarizztion data achieves parity with expert human datasets in both intrinsic and extrinsic evaluations. Furthermore, we illustrate that our data surpasses Human--ritten datasets in terms lexical diversity and faithfulness. We have also proven the applicability of our method to noisy crawled data.
We want to emphasize the immense potential this approach holds for facilitating the development of content-focused datasets and, consequently, generative models, minimizing the need for costly human annotation. Therefore, our method democratizes the creation of such datasets and models, making them more accessible to the entire community.

---

Published as a conference paper at ICLR 2024
REFERENCES
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. Eval- uating correctness and faithfulness of insllwition-followinng models for question answering. arXiv preprint arXiv:2307. 16877, 2023.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami AI-Rfou. Knowledge graph based syn- thetic corpus generation for knowledge-enhaneed language model pre-training. arXiv preprint arXiv:2010. 126888 2020.
Yonatan Bitton, Shlomi Cohen-Ganor, Ido Hakimi, Yoad Lewenberg, Roee Aharoni, and Enav Weinreb. q2d: Turning questions into dialogs teach models how to search. arXiv preprint arXiv:2304. 14318, 2023.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. arXiv preprint arXiv: / Se::::0020000 2015.
Leshem Choshen and Omri Abend. Automatically extracting challenge sets for non-local phe- nomena in neural machine translation. Conference on Computational Natural Language Learn-
ing (CoNLL), pp. 291-303, November 2019. doi: 10. 1/9911080/// URL https:
(laclantholoyy
/tgg/K199 1028
dd Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer. 2005.
Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. Dialog inpainting: Turning documents into dialogs. In International Conference on Machine Learning, pp. 3558-556.. PMLR, 2022.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. Evaluating attribution in dialogue systems: The begin benchmark. Transactions of the Association for Computational Linguisties, 10:1066-1083, 2022.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli Eli5: Long form question answering. arXiv preprint 190iv:197.00911 2019.
Pengeheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangle embedding sharing. arXiv preprint 1.X0522111.09543, 2021.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 22: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7856-7870, Online and Punta Cana, Dominican Republic, November http:///22...mmmmmaaaa....
2021. Association for Computational Linguistics. doi: 10. htttp:://acllanthollog/. 200 emn wawnnpmmaaa.mm inmainn... 619
URL
2021
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Re-evaluating factual consistency evaluation. arXiv preprint MEXXv:2244.0499 2022a.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689 2022b.
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, and Noah A. Smith. Bidimensional leaderboards: Generate and evaluate language hand in hand. In Proceedings the the 2022 Conference of the North American Chapter of
the Association for Computational Linguisties: Human Language Technologies, pp. 3540-3557 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 18.53v53/v1/ 2022. main.259. URL wt.pttttpp...................... //acl http wgw.ooyg..ro//222222 naacl n. 259
10

---

Published as a conference paper at ICLR 2024
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, et al. Soda: Million-scale dialogue distillation with social commonsense contextualization. arXiv preprint arXiv:2212.10465 2022.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversation--demcraatiin democratizing large language model alignment. arXiv preprint arX1w:2304.00327,, 2023.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. Hurdles to progress in long-form question answer- ing. In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pp. 4940-4957, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/11/2221..m maw.22l-aaalmmmian3.3 URL
lttps::/aclaatthooogy org/ nwwclaaallaommmmm sasacc--mannn mwi...i.................. 393
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Chin-Yes New Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl ma.n. 173. URL https://acLanthoologyoooo 2020. acl-macl 173.
Philip M McCarthy and Scott Jarvis. Mtld, voed-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381-392, 2010.
Sewon Min, Julian Michael, Hannanch Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004. 2020.
Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. Dis-
entQA: Disentangling parametric and contextual knowledge with counterfactual question answer- ing. Annual Meeting of the Association for Computational Linguistics, pp. 10056-10070, July 2023. doi: 10.18653/v1/2223.ac-oooo acl-long.559, URL htttps::/aclattttlloooooo htt.:/aaanntttooooooo///2//22332 org/2023. acl acl-iong. 559
Yasumasa Onoe, Michael JQ Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge. arXiv preprint arXiv:2305. 230.00165,.. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint axXiv:2006001166 2023.
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. Efficient benchmarking (of language models). arXiv preprint 1tt.::2308.11999 2023.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao. James Thore, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowl- edge intensive language tasks. arXiv preprint arXiv:2009.02252, 2020.

---

Published as a conference paper at ICLR 2024
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Con-
ference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, Octo-
ber 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6311 URL https: a/cl acl antho woghchhoooy...ooo oww.orggwwl.. 6319
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Wei Chang. Asqa: Factoid questions meet long-form answers. arXiv preprint Se0i::2204.0009900 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09.8088..8,,,,, 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint ar arXiv:2212.00500 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Xim- ing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 4602-4625, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: (0.18653/v1/0022.mm www.:o2.nnnaalmmnnn naacl-mmiin341 URL http:///clonttcloooooro///////// haacl-main.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore:
Evaluating text generation with bert. ArXiv, abs/1904.09675, 2019a. URL https://api,
semanticscholas: wtt.ccooo.....rg//ooouus.aaaaaaa 12:
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 09675, 2019b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat-Ing m: A large-scale real-world Ilm conversation dataset. arXiv preprini arXiv:2309.119900 2023.
Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael Zeng, and Xuedong Huang. Make lead bias in your favor: sero-shot abstractive news summarization. In International Conference on Learning Representations, 2020.
A DATASETS STATISTICS
While the amount of synthetic data is capped merely by the actual amount produced and the con- tent available, meaning the others can generate more data, we supply the actual sizes we release. Statistics are shown in Table
B COMPLEMENTARY RESULTS.
We provide additional human experiments (Table 7 demonstrating that using clean content from exciting datasets can further improve the synthetic data quality.
We also provide the results synthetic datasets created with the stronger Lamma model in Table and PubMed on Table
We also include here results with Bomma-2-3B--hhee as the base model. General results are pre- sented in Table 1n Faithfulness results are in Table 10
12

---

Published as a conference paper at ICLR 2024
Table 6: Data statistic for several datasets including the size of the data, the number of words in the response, and the average lexical diversity. Overall, our dataset spans 300K+ samples, similar to the size of ELI5, and Maal which were collected from available resources and noisy by nature. On the other hand, our data is 50 times larger than the carefully annotated ASQA data. We also can see that most response lengths are similar the ones in the human writing datasets, and the lexical diversity of all synthetic data is higher than their human writing counterparts.
samples
# words response
lexical diversity
ELI5
272K
107.0
70.9
ASQA
6,316
73.3
65.8
CNN- Mailil
311K
44.8
40.1
Wish
27K
65.1
54.3
Wish-QA-NQ
88K
73.4
62.3
Wish-QA-EL.S
78K
98.3
71.8
Wish-QA-ASQA
6140
87.5
67.0
SusmaiSumarizzaioon
37K
66.6
76.0
Wish-QA-Med
69K
84.2
71.8
Table 7: Human assessment for Wish-QA without and with filtering, and Synthetic NQ. Results
show big improvement in question relevance and the percentage of answers that address the ques- tion, answers that are faithful, and overall answer scores.
Quality Review Question
Wish- QA w/o filters
Wish-QA w/ filters
Synthetic NQ
Is the question relevant to the content?
67%
92%
98%
the question clear? (not ambiguous) Does the answer address the question? Is the answer faithful to the content? Grad the overall quality of the answer
63%
67%
91%
80%
98%
98%
53%
76%
88%
3.48
4.58
4.74
C INFORMATION EXTRACTION RESULTS.
To test the efficacy the method in another task, we conducted an experiment focusing on informa- tion extraction (IE). For this experiment, utilized the information extraction part of the Databricks dataset, which we divided into train and test sets (1000/50). Synthetic information extraction data was generated over Google-NQ passages, requiring only new of few-shot examples demon- strate the task, without any other modifications. The table below 12 shows that our model improved in all metries compared the Flan-xl xl baseline, except for ANLI. Furthermore, achieves higher scores than the model trained on human data in Reward and K-Precision, while slightly lagging behind in ROUGE-L, Bert-Score, and ANLII The average of the faithfulness metries indicates com- parable performances in this dimension for the models trained on human and synthetic data. These experiments reinforce our belief that our method is general and can be readily applied to various tasks, yielding improved results through the generation of high-quality synthetic data.
Table 8: Results on ASQA and ELI5 including models trained with Llama-2-70B-chat synthetic
data.
Test-set
ASQA
ELIS
Train Set
ROUGE-L
Bert-Score FI
Reward
ROUGE-L
Bert-Score Fl
Reward
Plan-xl
10.5
49.7
28.8
6.2
46.7
9.2
Human in- in-domain
31.4
66.0
68.6
13.1
51.3
11.3
Human out of domain
18.7
58.7
37.2
13.5
52.2
24.4
Wish-QA
28.0
67.5
85.1
13.8
55.2
26.7
Wish- Wish-QA-NQ Falcon
28.2
64.8
80.3
13.2
54.0
30.3
Wish-QA-Me Ilama
28.1
64.8
81.7
13.6
53.9
44.1
Wish-Qu QA in-domain Falcon
27.0
63.4
73.3
13.1
52.8
22.7
Wish-QA in-domain Llama
28.0
64.6
76.4
13.7
53.6
30.5
Gold
72.1
30.3
13

---

Published as a conference paper at ICLR 2024
Table 9: Full results on PubMed dataset. Here we include results where Llama-2-70B is the data
generator.
PubMed
Train-set
ROUGE-L
Bert-Score
Reward
x-Precision
ANLI
Flan-xl
12.8
53.8
10.7
60.6
38.2
ASQA
20.5
61.4
37.3
77.2
60.8
ELI5
15.0
56.3
16.8
32.2
2.2
Wish-QA-Med Falcon
22.1
61.6
39,4
78.2
81.8
Wish-QA- Falcon
22.0
62.9
44.5
84.2
73.1
Llama
21.4
62.1
30.0
74.6
92.4
Wish-QA-NQ Llama
21.4
62.5
51.2
80.6
83.0
Table 10: Faithfulness results ASQA and ELI5 when Llama-2-13b-Char is the base model.
Test-set
ASQA
ELI5
Train-set
k-Precision
ANLI
k-Precision
ANLI
Llama-13b-chat
40.4
65.5
23.3
24.6
ASQA
77.3
73.1
48.0
37.3
ELI5
59.0
42.6
33.3
13.8
Wish-QA-NQ
81.1
76.2
58.2
45.1
Wish-QA-ASQA
79.2
73.1
51.8
40.3
Wish-QA-ELIS
77.6
75.9
52.0
46.1
Table 11: Results on ASQA and ELI5 when Llama-2- 1aama---13b-Chatt is the base model.
Test-set
ASQA
ELI5
Train-set
RougeL
Bert-Score
Reward
RougeL
Bert-Score
Reward
Llama-13b-en 13b-chat
21.4
57.6
62.7
11.8
51.2
46
ASQA
29.0
65.2
91.1
12.4
53.5
66.9
ELI5
13.4
55,0
86.4
11.4
52.8
66.7
Synthetic NQ
26.3
64.4
86.9
12.5
53.5
52.6
Synthetic ASQA
26.3
63.6
87.1
12.3
53.1
57.7
Synthetic ELI5
26.3
63.6
89.2
12.4
53.0
57.1
Table 12: Information Extraction results on Databricks dataset.
Model
ROUGE-L.
Reward Bert-Score
K-Precision
ANLI
Flan-xl
42.3
67.3
35.9
39.1
89.1
Databricks
54.6
77.2
70.1
56.7
89.9
Wish-IE
47.9
74.5
87.4
60.0
86.8
14

---

Published as a conference paper at ICLR 2024
D PROMPT
We provide examples of the prompts used to generate data in Fig 2
15

---

Published as a conference paper at ICLR 2024
An illustration of our data generation prompt. In black is the few-shot prompt we give the model. In pink new QA that the model generated based the the provided content.
Figure 2:
Instruction: Given the next [document], create [question] and [answer] pair that are grounded in the main point of the document, don't add any additional information that is not in the document. The [question] is by an information-seeking user and the [answer] is provided by helping Al Agent.
[document]: Scrumptious Sweet Co. factory...
### Response:
[question]: Where was the movie Chitty Chitty Bang Bang filmed?
[answer]: Chitty Chitty Bang Bang was filmed in variety of locations in England and Germany. Some..
[document]: The series stars Eugene Levy Johnny Rose
### Response:
[question]: What is the plot of the show Schitt's Creek?
[answer]: The show Schitt's Creek about a wealthy family who loses their fortune and is forced to rebuild their lives in small town. The show follows the family as they adjust to their new life in the town and learn to appreciate the simple things in life.
[document]: 2016's countdown broke several Hottest 100 records...
### Response:
[question]: What was the most popular song on the 2016 Hottest 100? [answer]: The most popular song on the 2016 Hottest 100 was "Never Be Like You" by Flume. This was the first time that an electronic dance music producer topped the countdown.
[document]: In Greek mythology, Persephone
Response:
Who is Persephone in Greek mythology and what her significance? [answer]: In Greek mythology, Persephone, also known as Kore, the daughter of Zeus and Demeter. She..
[question]:
[document]: Terry, the dog that played Toto in the 1939 screen
### Response:
[question]: What breed of dog was Toto in the Wizard of Oz?
[answer]: Terry, the dog that played Toto in the 1939 screen adaptation of The Wizard of Oz, brindle Cairn terrier.
16


Reviews:
[{"review_id": "DqiYF98tAg", "reviewer": "Reviewer_xd6k", "summary": "The paper focuses on LLM-based synthetic data generation for content-based generation tasks (such as long-form QA).\nThe paper uses LLMs with in-context task-specific examples and new contents to generate synthetic content-related data (eg. question-answer pairs). This data is then filtered based on various factors (reward model scores, format etc.). Through multiple evaluations certain synthetic data created in that form is found to be on par with human data.", "questions": "I am open to increasing the score if I am missing some context, highlights, or critical contrast compared to prior works. This could be provided in the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written.\n2. The paper demonstrates that the synthetic data generated through LLM can achieve parity with human data) for content-grounded generation tasks.\n3. Multiple forms of evaluations are done.", "weaknesses": "1. While I have not noticed any technical issues with the paper, the main weakness seems to be the novelty and the scope of the contribution. The 2-step based generation seems fairly obvious as an approach, and it seems already been tried before in a few works as elaborated in the related works. While the exploration of content-grounded generation tasks may be new, I am not sure if simply using prior methods in another task setup (without tackling any fundamental task-specific challenges not relevant in prior works) is enough to meet the bar for ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hRQ2IsIJjx", "reviewer": "Reviewer_pq6E", "summary": "The paper introduces a method for the automated generation of high-quality, content-grounded data through a three-stage process. The experiments reveal that models trained on this data either match the performance or surpass models trained on human-generated data, with an advantage in terms of faithfulness.", "questions": "why using synthetic data can improve performance on noisy test sets?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Employing Large Language Models (LLMs) to generate high-quality datasets is both logical and intuitive.\n2. Experiments showcase the method's effectiveness for both QA and summarization tasks.", "weaknesses": "1. The paper claims that high noise levels in existing datasets, such as news domains. This claim necessitates empirical validation. Additionally, the test set originates from the same noisy source. It is necessary to perform a Multi-Dimensional Quality assessment on both the original and synthetic datasets to evaluate genuine quality enhancements. The paper lacks specifics on annotation details, including inter-annotator agreements.\n2. What if using llama 70b chat synthetic data to train a llama 70b model?  If we're talking about cost and time efficiency, then using synthetic data from LLMs to train smaller models might not be the most economical approach, if LLM itself can work well. The paper argues about the method being more cost-effective than traditional crowd-sourced dataset curation, which might exceed $1M. Yet, directly sourcing text from the web can be free, and existing filtering methods can be used to ensure data quality.\n3. The paper is more like an empirical exploration into using LLMs for synthetic data generation, which is not inspiring for other works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHoT6yq3ID", "reviewer": "Reviewer_Y1CY", "summary": "This paper proposes a method for automatically generating high-quality content-grounded datasets for tasks such as question answering and summarization. The method consists of three stages: content preparation, generation, and filtering. The authors showcase the effectiveness of their methodology by generating large-scale data for synthetic long-form question answering and summarization tasks. They compare models trained on their synthetic data with models trained on human-generated data and show that their models perform equally or better in terms of quality and faithfulness.", "questions": "Please check the weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents an innovative and practical methodology for generating content-grounded datasets. The three-stage process of content preparation, generation, and filtering provides a systematic approach to ensure the quality and faithfulness of the generated data.\n2. The authors provide insightful empirical findings by comparing models trained on their synthetic data with models trained on human-generated data.", "weaknesses": "1. Content-grounding generation is a broad topic, and the paper only mentions long-form QA and summarization. However, if we look at it purely from the perspective of \"Content-grounding generation,\" most traditional NLP tasks can be transformed into the paradigm: Instruction+Input -> Output, which matches with the definition of Content-grounding generation. However,  the paper only validates this on two tasks. Furthermore, in subsequent experiments, the paper only constructs data based on specific datasets, raising doubts about the generality of the proposed method. Therefore, I tend to believe that the paper exaggerates its contributions.\n2. The data construction methods mentioned in the paper rely on the format and source (Wiki) of previous datasets, making it challenging to assess their practicality in an open-world context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)