Best Practices and Lessons Learned on Synthetic Data
Authors: Ruibo Liu
Jerry Wei
Fangyu Liu
Chenglei Si
Yanzhe Zhang
Jinmeng Rao
Steven Zheng
Daiyi Peng
Diyi Yang
Denny Zhou
Andrew M. Dai
Venue: COLM (2024)
Keywords: synthetic data, survey
TL;DR: A survey about synthetic data

Abstract: The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.

Published as a conference paper at COLM 2024
Best Practices and Lessons Learned on Synthetic Data
Ruibo Liu, Jerry Wei/ Fangyu Liu Google DeepMind, Anthropic'
Chenglei Si, Yanzhe Zhang Stanford University, Georgia Institute of Technology
Jinmeng Rao, Steven Zheng, Daiyi Peng Google DeepMind
Diyi Yang Stanford University
Denny Zhou, Andrew M. Dai* Google DeepMind
Abstract
The success of AI models relies on the availability of large, diverse, and high- quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged ing solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We empha- size the need for responsible of synthetic data to build more powerful, inclusive, and trustworthy language models.
promis-
Introduction
The rapid advancement of artificial intelligence (AI) technologies has led to their widespread adoption across numerous domains, from assistant agents (e.g., ACT-1, from Adept A and software development (e.g., Devin, from Cognition Lab to healthcare (Singhal al. 2022) and finance (Zheng et al.) 2022). However, the success of Al models heavily relies on the availability large, diverse, and high-quality datasets for training and evaluation. Acquiring such datasets can be a significant challenge due to data scarcity (Babbar & "cchlkoppf (219),, privacy concerns (Abay et al. 2019), and the sheer cost of data collection and annotation (Gilardi et al.) 2023a). Pessimists predict that we will run out of fresh text data in 2050 and image data 2060 Villalobos et al., 2022).
Synthetic data has emerged as promising solution to address these challenges (Nikolenko 2021). Synthetic data refers to artificially generated data that mimics the characteristics and patterns of real-world data, but is created through algorithms (Saxton et al.) 2019), generative models (Borisov et al., 2022 Meng et al. 2022), or even simulations (Vezhnevets) et al. 2023 Liu et al. 2003c), rather than being directly created by humans. By leveraging synthetic data, we can not only overcome the limitations of real-world data but also unlock
"Corresponding author(s): ruiboluu@guogle.com adai@googleecom
Work done at Google DeepMind
1 ACCTII: http::httttt/://w wttpappt..bbboo//cctttt
Devin: httpp://www.cogniiti htt.:llaaa....a..aoa..aaa... htt.://ww...a..acccoccccconnnnaa

---

Published as a conference paper at COLM 2024
the potential to develop more robust, reliable, and fair Al models [Lccini 2021 2023).
Lu et al.
One of the many benefits of synthetic data is that it can be generated at scale, providing abundant supply of training and testing data for Al models. This is particularly valuable in domains where real-world data is scarce difficult to obtain (e.g., weather data covering all conditions (Li et al. 2023a Lam et al. 2023)). Second, synthetic data can be tailored to specific requirements, such as ensuring balanced representation of different classes by introducing controlled variations (e.g,, up-weighting low-resource languages in multilingual language learning [Przystupa Abdul-Mageed, 2019)). This level of control over data characteristics can improve model performance and generalization. Third, synthetic data can help mitigate privacy concerns by creating anonymized or de-identified datasets that do not contain sensitive personal information (Howe et al. 2017 El Emam et al. (0220) This is crucial in domains such as healthcare, where patient privacy is utmost importance (Dahmen & Wei et al. 2019).
Cook 2019
Despite its promise, synthetic data also presents challenges that need to be addressed. One of them is ensuring the factuality and fidelity of synthetic data (Wood et al.) 2021 Heusel et al. 2017), as models trained on false, hallucinated or biased synthetic data may fail to generalize to real-world scenarios (Van Breugel et al. 2023: Guarnera et al. 2020). Researchers must develop sophisticated generative models and evaluation metrics to create synthetic data that accurately reflects the complex patterns and relationships found in real-world data. Another challenge is the potential for synthetic data to amplify biases or introduce new biases if not carefully designed and validated (Barbierato et al. 2022 Gupta et al. 2021). We believe rigorous testing and fairness assessments are necessary to mitigate these risks.
In this paper, we track the current state of synthetic data research and discuss current best practices and lessons learned. The rest of the paper is organized as follows. Section 2 provides an overview of synthetic data generation techniques and their applications in model training, presenting case studies and empirical evidence. Section 3 discusses the usefulness of synthetic data in evaluation. Section discusses the challenges and limita- tions of synthetic data, and in Section 5 we outline potential solutions and future research directions.
Synthetic Data in Training
Synthetic data, which is generated by mimicking authentic data collected from the real world, has proven to be an effective and relatively low-cost alternative of real data. This section explores several notable domains that leverages synthetic training data.
2.1 Reasoning
Math. Recent adv advancements in mathematical reasoning for language models (LMs) have led to the development of various approaches to improve performance on math-related tasks. One approach is to train on math-targeted pre-training data, such as Minerva (Lewkowycz
al. 20022,, Llemma (Azerbayev et al. 2023), and DeepSeekMath (Shao et al. 2024). Another mainstream method is to generate synthetic questions and answers to imitate the fraining or validation set of target benchmarks. For instance, WizardMath (Luo al. 2023a leverages series of operations to increase the complexity of questions and answers using GPT-3.5, while MetaMath (Yu et al. 2023) bootstraps the questions in MATH and GSMSK by rewriting them in different ways, such as semantic rephrasing, self-verification, and backward reasoning. GAIR-Abel (Chern et al. (202)) found that the format of the augmented answers crucial to final performance, with answers that begin with paraphrasing of the question followed by step-by-step solution showing better performance than those in vanilla format. Xwin- Math (Li et al. 2024) further scaled up synthetic SFT data to one million examples and found that the LLM 7B model (Touvron et al. 2023) can still benefit from data scaling. MMIQC (Liu & Yao 2024) composed bundle of datasets that infuse SFT style data (via question-answer rephrasing or directly taken from MetaMath) with a subset of high-quality mathematical pre-training data, such as Open MetMahhh (Paster et al. 2023
2

---

Published as a conference paper at COLM 2024
Scaling up the generation of synthetic math data is straightforward process, but ensuring the correctness of the generated math remains a significant challenge for practitioners. AlphaGeometry (Trinh et al., 2024) is recent attempt to address this issue by training neural model using TO0 million synthetic data points. The model proposes solutions and guides symbolic deduction engine in verifying the correctness of each branch when solving complex geometry problems. By combining the power of synthetic data with a rigorous verification process, AlphaGeometr achieves a problem-solving ability comparable to that of human Olympiad gold medalist, demonstrating the potential of this approach in tackling complex mathematical reasoning tasks.
Code. Different from Math, synthetic data for code reasoning can naturally combine the execution results with structured code, as one requirement of correct code is being executable. In coding-enhance models, CodeRL (Le et al. 2022) presents an actor-critic approach to improve pretrained language models with feedback signals on synthetic code samples. Haluptzok et al. (2022) propose a self-improvement strategy where the models generate their own synthetic puzzle-solution pairs. These pairs are then verified and filtered by a real interpreter before being used to finetune language models. Shypula et al. (2023) further propose a framework that leverages a simulated environment and adaptation strategies like self-improvement synthetic data generation and CoT prompting for code optimization. Peeegeeeoooiioie
Yang et al. (2024) developed InterCode, a framework designed to enhance interactive code generation within reinforcement learning environment, where code serves as actions and execution feedback serves as observations. Reflexion (Shinn et al. 2024 employs external or internally simulated linguistic feedback signals to improve the code reasoning capabilities of language models. Regarding synthetic SFT data, Code Alpaca comprises a dataset of 20K code instructions automatically generated by applying SELF-INSTRUCT (Wang et al. 2022a) to ChatGPT across 21 seed tasks. WizardCoder (Luo et al.) (023b) introduces Code Evol- Instruct to guide ChatGPT with heuristic prompts to enhance the complexity and diversity of synthetic data. Meanwhile, Magicoder (Wei et al. (023c) developed OSS-INSTRUCT, which generates 75K diverse synthetic instruction samples from open-source code snippets.
Other reasoning tasks. Synthetic data also leads to impressive performance in other reasoning tasks. For instance, Wei et al. (002a) augmented existing natural language datasets by replacing natural language labels with arbitrary symbols, generating over 500k synthetic examples. Using these synthetic data for supervised finetuning significantly improved model performance on unseen in-context learning and algorithmic-reasonigg tasks. STaR (Zelikman et al. 2022) generates synthetic saain-of-though rationales and filters out those leading to wrong answers for finetuning language models to improve their reasoning. In the domain of physics reasoning, Mind's Eye (Liu et al., 2022) takes a novel approach by training text-to-code model with synthetic "text-description code" data. This enables the model to convert textual questions into rendering code, which then executed in a physical engine (i.e., DeepMind MuJoCo (Todorov et al.) 2012 The rendering results are injected into the context, allowing even small language models armed with Mind's 'ss Eye to achieve performance comparable to models 100 times larger.
rendering
2.2 Tool-using and Planning
Learning tool-using through synthetic trajectories. Synthetic data is also a powerful approach to enable LMs to learn tool-using abilities through simulated trajectories, as collecting real-world human tool-using data might be time-consuming, and the actual distribution of calls to tools might be skewed. LaMDA (Thoppilan et al., (2022,, for instance, was trained not only on web documents but also on interaction data between crowdworkers and the model itself, with the synthetic data annotated with calls to appropriate tools. This training process allowed LaMDA to develop the ability to use calculator for arithmetic, a search engine for real-time information seeking, and a machine translator for translation. Similarly, Toolformer (Schick et al. 2024) learns to decide which APIs to call and what data, while Galactica (Taylor et al.)
arguments to pass by training on cemplate-generatee 2022) infuse API-calling data into pre- trainin mixture. ToolAlpaca (Tang et al.) 2023 is novel framework designed to automatically generate diverse tool-use corpus, by building
3

---

Published as a conference paper at COLM 2024
multi-agent simulation environment and letting agents select and use tools iteratively. These examples demonstrate the potential of synthetic trajectories in enabling LMs to acquire tool-usinn abilities and enhance their reasoning capabilities across various domains.
Learning to plan in synthetic environments. An important feature of the agent in Au- tonomous Machine Intelligence (LeCun 2022) is planning ability of decomposing complex tasks into subtasks and finishing the subtasks hampati et al.; 2024). Synthetic data can be a valuable tool here as it can serve as the teedback signal collected from simulator (Park et al. 2023), and learning on it can make the agent aware of affordances (Ahn et al. 2022 Lingg et al. 2022). For example, Inner Monologue (Huang et al., 2022) leverages natural Language form feedback generated by the simulated environment to teach LLM-based robots planning. They find that such feedback significantly improves high-level instruction completion on both simulated and real-world domains. To compose large number of realistic planning tasks (e.g., "Rearrange objects table to match a given scene. VIMA (Jiang et al. 2022) creates a multi-modality simulated environment called VIMA-Bench, which supports extensible collections of objects and tex- tures. In the Minecraft game, Voyager (Wang et al. 2023) deploys number GPT-4 based agents to interact with the synthetic environment and finds that the agents can unlock new skills faster and complete planning more efficiently with the help of synthetic feedback.
reward-optimal way (Kamb-
2.3 Multimodality
Reverse rendering from vision to text. Vision-languagg alignment data focuses on ac- curately grounding visual input to an LLM (usually via a vision encoder). Web-scraped image-caption pairs have been the most popular MM alignment data in the past few years since CLIP (Radford et al. 2021) and ALIGN (Jia et al. 202T). However, web-scraped image- text pairs are usually noisy and only have coarse-gres coarse-grained correspondence, insufficient for grounding details of images in language. In domains such as documents, screens, figures, and diagrams, such fine-grained alignment can most conveniently be obtained from data synthesis pipelines built with image rendering engines. Pix2Struct (Lee et al. (2023) uses web servers to render HTML code into website screenshots, and the training task is derender masked screenshot to the full HTML code. MatCha (Liu et al., 2023b) and DePlot (Liu al. 2023a) render tabular data into charts with Python plotting libraries and pretrain foundation model by giving the rendered image and producing the code and/o or the tabular data. Si et al. (2024) and Laurençon et al. (2024) train on synthetically generated HTML and CSS files for the task of converting webpage screenshots into code implementation. The models finetuned on the synthetic data can generalize reasonably well on realistic data scraped from the Internet. |Borkman et al. (2021) propose to use physics engines or game engines (e.g,, Unity) as the synthetic data generator to help computer vision research.
Multi-modality instruction following. Downstream applications of multimodal LLMs require reasoning and instruction following capabilities. Such data are usually long-form question response pairs and are expensive for humans to create. LLaVA (Liu et al. 2024b uses existing image captions to prompt GPT-4 (in text-only mode) for writing diverse and long-form prompt-answer pairs. During multimodal LLM training, images and prompts are used as input while the captions and bounding box information can be hidden. Besides image captions, other sources of image attribute information such as object bounding box (Zhao et al.) 2023), OCR (Zhang et al. 2023c) and derendered charts (Masry et al.) 2023 Carbune et al. 2024) can all fit into such as image attributes + text LLM rewriting synthetic data pipeline.
2.4 Multilingual
Back-translation augmentation. Many multilingual language models use taak-transsttionn
as a data augmentation method, creating synthetic parallel training data from monolingual data sources (Sennrich et al. 20066) Zheng et al.) 2020; Caswell et al., 2019 Marie et al. 2020 Bi et al. 2021 Liao et al. Pham et al. 2021: Xu et al. 2022). For example, Sennrich et al. (0016) back back-translate monolingua target data into source language data, providing
4

---

Published as a conference paper at COLM 2024
additional parallel training samples for substantial translation task improvements. Re- searchers have also explored different sampling methods for back-translation (e.g, beam search, constrained sampling, unconstrained sampling) and their comparative effective- ness (Sennrich et al. 2016; Edunov et al. 2018 Graça et al. 2019 Bannard & Kallison-Burchhe 2005). Xu et al. (2022) emphasize the importance of the weight and quality of synthetic data for optimal NMT performance using tack-translation. They propose a method to optimize the ratio between search methods and gamma score to balance estimated importance weight and quality. However, some limitations exist with back-translation-aases synthetic data generation. For example, the quality and diversity of synthetic data depends on the performance of the back-translation method. If the synthetic data is too noisy or not diverse, the performance gain would be limited (Epaliyana et al. 2021 Chauhan et al. 2022).
Generating multilingual questions and answers at scale. Recent studies explore the generation and utilization of synthetic multilingual question-answer (QA) pairs to improve language models' performance in multilingual and cross-lingual question answering (Asai
et al. (202)) Kumar et al. 2019 Chi et al. (2020) Riabi et al. Li & (Callison-Burde 2023
Abulkhanov et al., 2023). One approach is to translate existing monolingual questions and/ or answers into other languages (Asai et al. 2021). Another involves using Question
Generation (QG) models to produce synthetic questions in a cross-lingual fashion based on
answers and/ or source texts (Kumar et al. 2019 Chi et al. 2020 Riabi et al., 2021). Recent efforts also focus on jointly generating questions and answers in multiple languages for greater flexibility (Shakeri et al., 2021 Li & Aillioon-Burch 2023). For example, (Shkerii et al. (2021) finetune a pretrained multilingual 15 model (Xue et al. 2020) on a mixture of a QA generation task and multilingual masked language modeling task to produce synthetic QA pairs in multiple languages. These efforts generally show that language models trained on synthetic QA pairs demonstrate improved performance on multilingual QA and information retrieval benchmarks.
2.5 Alignment
Instruction Following, Synthetic data can serve as a promising approach for training instruction-folloiingg models, particularly in scenarios where real-world data is scarce, expensive, or challenging to obtain. Self-instruct (Wang et al., 2022a) and Stanford Al- paca (Taori et al., 2023) are both using LLMs to generate instruction tollowing data which covers wide range of scenarios. They first pick small set of "seed instruction following samples" " and then ask the LLMs to imitate the format to generate more demonstrations. One concern of this type of method is how to keep the generated data high quality, which involves the complexity of queries (Liu et al. 2023d), the diversity of semantics (Ding) et 2023), and the scale of the synthetic dataset (Yuan et al.] 2023). To this end, Xu et al. (2023) propose Evol-Instruct which adds complexity to simple instructions via prompting. Mukherjee et al. (2023) leverage LLMs to revise the instructions and responses iteratively to include high-quality explanation traces in the FLAN dataset (Wei et al. 2022), and they find the trained model has improved performance in many NLP'tasks UltraChat (Ding et al., 2023) is large-scale and multi-round synthetic dialogue dataset, which is generated by two separate ChatGPT Turbo API models one one serves as the user role while the other serves as the assistant. They instruct the user model with carefully designed prompts to mimic real human user behaviors.
Many language models are supervised finetuned to learn how to follow instructions, but in learning this behavior, they may inadvertently also learn to be sycophantic (Perez et al. 2023), tailoring their responses to follow user's 's viewpoint, even if that viewpoint is not objectively correct (Wei et al. 2023b). Sharma et al. (2024) find evidence that the preference models (i.e., the reward model used for RLHF training) and even humans prefer sycophantic responses sometimes. On this front, Wei et al. (2023b) generates synthetic data to encourage models to be robust to user opinions and adds these data in a finetuning step to reduce sycophantic behavior on held-out prompts.
Mitigating hallucination. Many widely-used language models utilize supervised fine- tuning (SFT) to learn to align their interactions with users (Wang et al. 2022b Zhang et al.)
5

---

Published as a conference paper at COLM 2024
2023b). In particular, there exist many methods of generating synthetic SFT data that can improve capabilities such as reasoning and aligmment (Wei et al.; 2023a bb. It has been shown, however, that these synthetic data can induce hallucinations into language models by containing nontrivial amounts of hallucinated answers or by forcing models to learn to answer questions that they do not know the answer to (Zhang et al., 2023d). These cases demonstrate that synthetic data, if not applied correctly, can actually increase hallucinations in language models.
On the other hand, recent work has also shown promising results in mitigating hallucina- tions using synthetic data. For example, GPT-4 (OpenAl 2023) was trained using reward model that leveraged synthetic hallucination data in order to perform reinforcement learn- ing (Zhang et al. 2023d). This method resulted in significant improvement in performance on the TruthtulQ A {Lin et al. 2022) dataset (Zhang et al. (023d). Similarly, JJoes et al. (2023) designed a synthetic task where hallucinations can be readily evaluated, utilizing this task to optimize LLM outputs by learning continuous postfix via prefix-tuning, Tian et al. (2023) uses automated fact-checking and confidence scores to rank factuality scores of model response pairs, which are then used to finetune language models with DPO (Rafailov et al., 2023) to improve their factuality. Continued research in using synthetic data to mitigate hallucinations is still limited, however, by the lack of synthetic tasks for which hallucinations can be scalably evaluated.
Aligning with shared human preference and values. Directly finetuning on value-aligned or human-pretee -urmfr-peeferred data is straightforward method for aligning language models, but this method often requires substantial human annotation, which can be prohibitively expensive at scale. Additionally, such annotation frequently exhibits varying styles and inconsistent quality, particularly in the case of poorly annotated samples at the lower end of the quality spectrum (Metal 2023 Gilardi et al., 2023a). To address these practical challenges, an advanced technique known as 'reinforcement learning from human feedback (RLHF)" has been proposed (Leike et al. 2018: Christiano et al. 2017 Ouyang et al. 2022). This approach involves training a reward model with human data to act as a proxy of human judgment, which guides the optimization of the LM generation policy.
Recent studies have proposed a mixture of synthetic data and real human data to train more robust reward models (Gao et al., )023). Constitutional AI (Bai et al. 2022) proposes to use a small set of principles to steer the Al generated critiques and feedback, and use such synthetic data to replace the real human data in the typical RLHF pipeline. The model trained with this RLAIF (i.e., reinforcement learning from AI feedback) method shows similar strong performance as RLHF baselines. In general, synthetic data offers powerful solution for human values and preferences alignment by allowing researchers to generate large-scale, diverse, and controlled training datasets in a low-cost way (Cui
et al. 2023 Ganguli et al. 2002)). By simulating wide range of scenarios involving ethical dilemmas (errez et 2022),, social interactions (Liu et al. 2023c), and cultural norms (Ziems et al. 2023), synthetic data enables comprehensive and systematic testing of AI models alignment with human values (Askell et 2021). This approach helps identify and mitigate issues related to bias (Liu et al. 2021 Ntoutsi et al., 2020), fairness (Zhao al. 2018 Landers & Behhredd 22023), ) and unintended consequences before Al systems are deployed in real-
world settings Ye et al., 2024).
However, it is important to acknowledge that low-fidelity synthetic human preference data might be limited in accurately reflecting nuanced human judgment (Argyle et al. 2023). Consequently, the resulting models may be less robust under "jail-breaking attacks" (Huang et al. Deshpande et al. 2202), and may reveal strategically deceptive behavior even through safety training (Pan et al. 2022; Steinhardt 2022 Everitt et al. (002)). To mitigate these risks, researchers must continuously refine and improve the quality and diversity of synthetic data, incorporating more complex and comprehensive scenarios that better capture the intricacies of human values and preferences. Additionally, combining synthetic data with real-world data, and creating synthetic data in interactive environment which can be synced with the real world, are promising remedies. As the need for effective Al gov- ernance and regulation grows, synthetic data will play an increasingly vital role in enabling
6

---

Published as a conference paper at COLM 2024
scalable oversight mechanisms that promote trust, accountability, and the development of Al technologies that are aligned with human values and societal expectations.
3 Synthetic Data in Evaluation
Synthetic data is widely used in evaluations of different perspectives:
Factuality. AI systems may generate information responses that are not grounded in factual knowledge or data, leading to the creation of misleading or false content, formally known as hallucination (li et al. 2023). Factuality evaluation aims ensure the consistency of the knowledge in the Al system's s output with the knowledge provided by its training
data and knowledge base (ji et al. 2023 Zhang al., 2023d). Early statistical-based halluci- nation evaluation methods rehied on n-grams to directly calculate the overlap of vocabulary between the input and output content (Dhingra et al. 2019 Wang et al. 2020). However, these methods have limitations, as they only consider lexical overlap and do not account for semantics or sentence meaning (li et af., 2023), making them unsuitable for evaluating more complex forms of hallucination. Subsequent assurance methods shifted from statistical approaches to model-based methods, which are more robust compared to token-difference- based methods (Honovich et al. 2021). While these model-based evaluation methods are more advanced than their predecessors, they still have limitations. For example, the models can only output the degree of hallucination and may struggle to pinpoint specific errors (Falke et al. (019). Feng et al. (023a) propose to combine LLMs generation with random walks on knowledge graphs generate synthetic evaluation data for factuality, which is aware of entities and relations on the graphs. WWei et al. (2024) created synthetic dataset called LongFact for long-form factuality evaluation and used Google Search as the ground- ing source and LLM for the automated judgement, to achieve human-level accuracy but with significally lower cost (Min al. 2002)..
Safety. Red teaming is powerful technique for evaluating the safety and robustness of Al models (Ganguli et al. 2022 Casper et al., 2023b). By generating diverse and realistic scenarios designed to elicit unaligned or harmful outputs (Casper et al. 2023a red teaming
can expose vulnerabilities and weaknesses in AI systems (Perez et al. 2022). For example, Perez et al. (202)) use LMs to generate datasets for evaluating the behavior of other LMs. They end up producing 154 high-quality datasets which are verified by humans, and discover new cases of inverse scaling where LMs get worse with size. |Hbbinger et al. (2024) leverage synthetic data to trigger backdoor attacks to LMs at scale; they find LMs can exhibit deceptive behavior and create a false impression of safety under such attacks, and standard "safety training" could not remove such deception easily. These methods demonstrate the feasibility of using AT assistance to scale up human oversight (Bowman et al. 2022) over complex problems and unseen domains.
Assisting human evaluation. Recent studies have shown that in many cases, synthetic judgements from large-scale LMs (LLMs) can serve qualified, fast, and low-cost alterna- tives to actual human evaluation GGilardi et al. 2023b). Using GPT-4 as the judge, Alpaca Eval (Li et al.) 2023b) and MT Bench (Zheng et al. 2023) are two popular benchmarks that measure the comprehensive abilities of LM-based ChatBot. In coding tasks, synthetic environment is a common choice to aid human evaluation, as humans can make the as- sessment more efficiently via actual executions and analysis on running logs. Gu et al. (2024) propose CRUXEval, a code execution reasoning benchmark consisting of 800 Python functions generated by CodeLLaMA-34B Similarly, Liu et al. (2024a) introduce CodeMind, a framework to gauge the code reasoning abilities of LLMS on Independent Execution Reasoning (IER), Dependent Execution Reasoning (DER), and Specification Reasoning (SR). All these evaluations based on synthetic data show strong correlation with real human judgements.
7

---

Published as a conference paper at COLM 2024
4 Challenges and Limitations of Synthetic Data
While synthetic data offers numerous benefits and applications, it is crucial acknowledge and address the potential challenges and limitations associated with its use. This section delves into three significant concerns surrounding synthetic data:
Misuse of synthetic data might proliferate misinformation. The potential misuse of synthetic data is a significant concern that must be addressed to ensure the responsible development of Al systems. Current Al models become increasingly capable of generating
human-like data ranging from text [Gemini-Team et al. 2024 2023), images (Saharia et al.
2022: Ramesh et al. 2022), songs to even videos (e.g., OpenAl SORA 4 This can be particularly dangerous when synthetic data is used to impersonate real people, manipulate public opinion, or influence political processes. Moreover, the dissemination of synthetic data-driven misinformation can erode trust in legitimate information sources, making it increasingly difficult for people to distinguish between truth and falsehood (Byman et al. 2023 Rid 2020). To mitigate these risks, it is crucial for researchers, developers, and policymakers to establish clear guidelines and best practices for the ethical generation and use of synthetic data, including robust mechanisms for detecting and countering synthetic misinformation (Groh et al. 2022). By proactively addiesgngg these challenges, we can harness the beneñits of synthetic data while minimizing its potential for harm.
Synthetic data might cause ambiguity in Al alignment. The increasing use of synthetic data in aligning Al models (e.g, Constitutional Al (Bai et al. 2022)) can introduce significant ambiguity and uncertainty. The goal of AI alignment is to ensure that AI systems behave in ways that are aligned with human values and intentions. However, synthetic data, which is artificially generated rather than collected from real-world sources, may not accurately represent the nuances and complexities of human values and preferences (Zhou et al., 2024). This discrepancy can lead to Al models learning from data that is biased (Feng et al.) 2023b Liu et al., 2021), ungrounded (Liu et al., 2022 Patel & Pavlick 2022), or misrepresentative of realdworld scenarios (Weidinger et al. 2021 [ji et 2l.]3)23). As result, AT systems trained on synthetic data may exhibit behaviors that are misaligned with human expectations, potentially leading to unintended consequences or even harmful actions (Zou et al. 2023 Anderljung et al. 2023). Moreover, the ambiguity introduced by synthetic data can make it challenging to interpret and understand the decision-making processes of Al models (Lightman et al., 2023), further complicating the task of ensuring alignment. To mitigate these risks, it is crucial for researchers to carefully consider the limitations and potential drawbacks of using synthetic data in alignment research and to develop robust methods for validating and testing AT models trained on such data.
Training with synthetic data makes evaluation decontamination harder. The use of synthetic data in model training poses significant challenges to fair evaluation. Evaluation benchmarks are often created by referring to public text sources, such as coursework web- sites or forums. Consequently, it is arguable that all publicly available benchmark test cases might occasionally be included in the pre-training data of LLMs (Hoffmann et al. 2022] Gao et al.) 2021). The use of synthetic data exacerbates this issue rather than mitigating it. Although the community has proposed several techniques to detect such evaluation contamination, such as min-k% prob (Shi et al. 2003), which checks the probabilities of k long-tail tokens, these token-level decontamination methods are inadequate when the model is trained with synthetic data. Synthetic data might include rephrased versions of the bench- mark data (Oren al. 2023 Mattern al., 2023), rendering token-level decontamination ineffective. In addition to developing more advanced evaluation contamination detection techniques, we recommend that model developers invest in creating and maintaining in- house and protected evaluation benchmarks. These proprietary benchmarks should be carefully safeguarded to prevent leakage and ensure the integrity of the evaluation process.
Make songs with Suno Al: https://ppppppmu ai/ Sttpph/ttpp:/pgaaaa.ccc///reeees htt.:/aaarraaaaaaeoeeeeeeeererra war--aaa-ollleee hooorddssmulato
8

---

Published as a conference paper at COLM 2024
5 Directions for Future Work
As the field of synthetic data continues to evolve, there are several promising directions for future research and development. This section outlines three key areas that warrant further exploration:
Synthetic data scaling. The impressive performance of many over-trained small language models (e.g,, Mistral series models (Jiang et al., 2023), and Gemma series models (Gemma- Team et al. 2024), inter alia) demonstrates the necessity of training with large amount of tokens (even passing the compute-optimal chinchilla law (Rae et al. 2021). However, whether we have similar conclusions on the training with synthetic data is still open question, as the quality of synthetic data may not be as consistent as real-world data (Yu et al. 2024). Future research should investigate the scaling laws for synthetic data and determine the optimal balance between the quantity and quality of synthetic samples. This exploration could help us understand the most effective strategies for leveraging synthetic data in training large-scale language models, potentially leading to more efficient and cost-effective approaches (Muennighoff et al. 2024).
Further improving quality and diversity of synthetic data. While existing methods for generating synthetic data have shown promise, there is still room for improvement in terms of creating high-quality, attributed synthetic samples that closely mimic real-world data.
Future research should focus on developing new advanced techniques (or based on existing
ones such as Generative Adversarial Networks (GANs) Goodffeloo et al. 2020) or Diffusion Models (Ho et al., 2020), inter alia) that can control and manipulate specific attributes of the generated data, enabling the creation of diverse and customizable synthetic datasets. Additionally, researchers should explore methods that can incorporate domain-specinf knowledge to ensure the generated data adheres to the underlying constraints and patterns present in the target domain (e.g., via Retrieval Augmented Generation (RAG) (Lewis) et al. 2020: Borgeaud et al, 2022) ) while maintaining the data quality. By advancing the state-of-the-art in attributed synthetic data generation, we can unlock new opportunities for privacy-preserving analysis (Assefa et al. 2020), and model training across various fields, from healthcare (e.g., synthetic medical images (Frid-Adar al. 2018 Wei et al., 2019)) and finance simulated trading trajectories (Zheng et al. 2022) to social sciences (Argyle
al. 2023
Park al. 2023) and beyond.
Towards high-fidelity and more efficient scalable oversight. As AT models become increasingly complex and autonomous, it becomes challenging to monitor and assess their behavior using traditional oversight methods that rely on human supervision or real-world data (Amodei et al. 2016). Future research should explore the use of synthetic data for high- fidelity scalable oversight of these advanced systems. Existing methods typically simulate certain scenario in social iterations, such debate (Leike et al. 2018), reflection (Zhang) eet al. 2023a), or revisions (Liu et al. 2023c) to obtain synthetic data, while new approaches could cover more comprehensive scenarios and more modalities (Sun et al., 2023), recent studies have found many issues of simulation that only covers a narrowed down (Cheng et al., 2023) or over-simplified (Zhou et al., (0244 scenes. Looking forward, another growing direction could be how to achieve scalable oversight more efficientll-giverr that we have the full control over the synthetic data generation, we can probably provide more targeted oversights with less synthetic data. As the need for effective AI governance and regulation grows, synthetic data will play an increasingly vital role in enabling more trustworthy scalable oversight mechanisms that promote robust, accountable, and safe deployment of Al technologies for the benefit of society (Askell et al. 2021
Bowman et al. 2022).
The emergent self-improvement capability. We typically choose the most capable model to generate synthetic data, as its generation is of higher quality. However, an intriguing ques- tion arises: can a model generate synthetic data that is better than the data it was trained on, thus enabling it to improve itself? This concept of relf-improvement through synthetic data generation is an exciting avenue for future research. If a model can generate higher-qualin data than its original training set, it could potentially bootstrap its own performance by iter-

---

Published as a conference paper at COLM 2024
atively learning from the enhanced synthetic data (Chen et al. 2024 This self-imppovemmet capability could lead to the emergence of more advanced Al systems that can autonomously
refine their skills and knowledge over time (Burns et al. 2023 Huang et al.
2023b). Although
recent work shows encouraging progress in this direction (Chen et al. 2024), the upper bound of self-improvement and the underlying reasons for its elfectiveness remain open questions. Future research should investigate the theoretical underpinnings and practical feasibility of self-improvement through synthetic data generation in more diverse scenarios, examining the necessary conditions, potential limitations, and associated risks. By unlocking the potential of emergent self-improvement capabilities, we could enable more adaptable, efficient, and autonomous learning processes (LeCun 2022)
2024 Yuan et al.
Conclusion
Synthetic data has emerged as promising solution to address the challenges of data scarcity, privacy concerns, and high costs in AI development. By generating realistic and diverse datasets, synthetic data enables the training and evaluation of AI models at scale across various domains. As we approach human-level or even superhuman-leve intelligence, obtaining synthetic data becomes even more crucial, given that models need better-than- average-human quality data to progress. However, ensuring the factuality, fidelity, and lack of bias in synthetic data remains a critical challenge.
Future research directions on synthetic data could focus on improving the fidelity and controllability of generative models and developing standardized evaluation and contam- ination protocols and tools. We could also explore the integration of synthetic data with other techniques and its application in other domains. Despite the challenges, the potential benefits of synthetic data in advancing AT research are significant. By leveraging synthetic data responsibly and effectively, we can build more powerful, inclusive, and trustworthy AT systems that benefit society as whole.
References
Nazmiye Ceren Abay, Yan Zhou, Murat Kantarcioglu, Bhavani Thuraisingham, and Latanya Sweeney. Privacy preserving synthetic data release using deep learning. In Machine Learning and Knowlledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, Septemiber 10-14, 2018, Proceedings, Part 18, pp. 510-526. Springer, 2019.
Dmitry Abulkhanov, Nikita Sorokin, Sergey Nikolenko, and Valentin Malykh. Lapca: Language-agnostic pretraining with cross-lingual alignment. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2098-2102, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. Xiv preprint, abs/ abs/2220.00199 2022. URL /ttps:////// wttp:///...r/////////////// /ttp////p/////aarrooo//////////0 01691
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. ArXiv preprint, abs/1606.06566 2016. URL
httpe://arrrv.oor//aa////// 06565
Markus Anderljung, Joslyn Barnhart, Jade Leung, Anton Korinek, Cullen O'Keefe, Jess Whit- tlestone, Shahar Avin, Miles Brundage, Justin Bullock, Duncan Cass-Begga et al. Frontier ai regulation: Managing emerging risks to public safety. ArXiv preprint, abs/ //////207.07777 2023. httpst//s///s org/ abs 2307 03718
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31 (3):33337-35 2023.
10

---

Published as a conference paper at COLM 2024
Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Hajishirzi. One question answering model for many languages with cross- voss-lingual dense passage retrieval. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, vir- fual, pp. 7547-7560, 2021. URL https://prrcceeednggg www.eeedddinnn..eii...... wtt.errppp...ppppee////aa... 3df07fdaeiab273aat 1ww0////aaa100000bbbbbb00aaa-a
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. ArXio preprint, abs/2112.0088 2021. URL
00861
Samuel A Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al in Finance, pp. 1-8, 2020.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. ArXiv preprint, abs/ 23100.16131 2023. URL.htts:///axiv org/ abs/2310.10631
Rohit Babbar and Bernhard Schölkopf. Data scarcity, robustness and extreme multi-label classification. Machine Learning, 10869):339-1355, 2019.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Consti- tutional ai: Harmlessness from ai feedback. ArXiv preprint, /b212202,0 2022. URL
httpp///aaaavv.oor//a//222110000 08073
Colin Bannard and Chris Callison-Burch. Paraphrasing with bilingual parallel corpora. In Kevin Knight, Hwee Tou Ng. and Kemal Oflazer (eds.), Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACI'05), pp. 597-604, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. doi: 10.3115/1219840 1219914. URL /tLL:ttop://a/lltttoooog/// /r/llp/gg///eaaattoooooo////////
Enrico Barbierato, Marco I. Della Vedova, Daniele Tessera, Daniele Toti, and Nicola Vanoli. A methodology for controlling bias and fairness in synthetic data generation. Applied Sciences, 12(9):4619, 2022.
Wei Bi, Huayang Li, and Jiacheng Huang. Data augmentation for text generation without any augmented data. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers), pp. 2223-2237, Online, 2021. As- sociation for Computational Linguistics. doi: ittpp://aclaathology org/2021 acl-long. 173
URL
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Bapting Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 2206- 2240. PMLR, 2022. URL httpps:::prpoceediiiiii
mlr wttppppwsseeeeeeppppppppppptp wwweee//vvie//brreeau//// wwtp:////breeaaa.a..omaaa html
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergii Kasneci. Language models are realistic tabular data generators. ArXio preprint, abs/2210.000 22100.6280, 2022. U :ttps:///rriiii
11

---

Published as a conference paper at COLM 2024
Steve Borkman, Adam Crespi, Saurav Dhakad, Sujoy Ganguly, Jonathan Hogins, Y. C. Jhang, Mohsen Kamalzadeh, Bowen Li, Steven Leal, Pete Parisi, Cesar Romero, Wesley Smith, Alex Thaman, Samuel Warren, and Nupur Yadav. Unity perception: Generate synthetic data for computer vision. ArXiv preprint, abs/2107.042 2021. URL https
pwrziiv.rr//baaa. httg/abb/21007 04259
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress scalable oversight for large language models. ArXiv preprint, abs ttbs/2221.0334// 2022.
httph::t//raaatte wwwp:ii..rgggbb//////// 03540
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold As- chenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to- strong generalization: Eliciting strong capabilities with weak supervision. ArXiv preprint, 20s//231,,9390,, 2023. URL https: https://arxii www./oggg/////////////...ee.ee// wtt./////...sse0
Daniel L Byman, Chongyang Gao, Chris Meserole, and VS Subrahmanian. Deepfakes and international conflict. Brookings Institution, 2023.
Victor Carbune, Hassan Mansoor, Fangyu Liu, Rahul Aralikatte, Gilles Baechler, Jindong Chen, and Abhanshu Sharma. Chart-based reasoning: Transferring capabilities from Ilms to vlms. ArXio preprint, 12596
2024. URL htttts://aaxiiiii wttp://.rgg//bb/////e/e
Stephen Casper, Tong Bu, Yuxiao Li, Jiawei Li, Kevin Zhang, Kaivalya Hariharan, and Dylan -adfield-Menell. Red teaming deep neural networks with feature synthesis tools. In shirty-seventh Conference on Neural Information Processing Systems, 2023a.
Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadeield-Menell. Ex- plore, establish, exploit: Red teaming language models from scratch. ArXiv preprint, (bss/230.009444 2023b. httphhtps:///et wwgps/iiiirrr///oooo aas///bs//// 2ww.//946600022222
Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pp. 53-63, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/ /vl W19-5206. URL pttps:://claanthooog .rg Mt///2995555
Shweta Chauhan, Shefali Saxena, and Philemon Daniel. Improved unsupervised neural machine translation with semantically weighted back translation for morphologically rich and low resource languages. Neural Processing Letters, 5703):1707 2022.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning converts weak language models to strong language models, 2024. 2024.
Myra Cheng, Tiziano Piccardi, and Diyi Yang. CoMPosT: Characterizing and evaluating caricature in LLM simulations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10853-10875, Singapore, December 2023. Association for Computational Linguis- tics. doi: (0.18653/vv 2022..emmllmmainnnmmm emnlp mannpp-minn.e 669
URL atttps//acclantholggy
Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and Pengfei Liu. Generative ai for math: Abel. /attps://gitinn com/GAIR CAAIAAAIIIINNPP 2023.
Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, and Heyan Huang. Cross- lingual natural language generation via pre-training. In The Thirty-Fourth AAAI Conference
on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7570-7577 AAAI Press, 2020. URL _ https: h/aa::://aaaaa/aae htt.//aaa...rgg/////// wwt.:/oo//oj///oooeeee ind/x.iddxxxi.. http://www.p///AAAtarrridee//e/a
12

---

Published as a conference paper at COLM 2024
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, wttp:cc/pppppp/////eeeeeeee /
USA, pp. 4299-4307, 2017. URL https://procceddiggg 2f0p:a/0000001111122202000000ee0
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback, 2023.
Jessamyn Dahmen and Diane Cook. Synsys: A synthetic data generation system for healthcare applications. Sensors, 19(5):1181, 2019.
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned languag models. ArXiv preprint, abs/2304.05335, 2023. //L////tte////rrro/o//////////// wwtp://////b.....oooo 05335
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. Handling divergent reference texts when evaluating table-to-text genera- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4884-4895, Florence, Italy, 2019, Association for Computational Linguistics. doi: 10.18653 / 11/119-1483 URL httpt:ttcc/aaattttoooooo
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high- quality instructional conversations. ArXiv preprint, 2ts/23305.14333 2023. URL https
//arxiv.org/ aww./a//...... 14233
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back- translation scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489-500, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 20.18653/vvl. 11/////1DD1--1055 URL htttps://acclaatholoog org/ 1045
Khaled El Emam, Lucy Mosquera, and Richard Hoptroff. Practical synthetic data generation: balancing privacy and the broad availability of data. O'Reilly Media, 2020.
Koshiya Epaliyana, Surangika Ranathunga, and Sanath Jayasena. Improving back- translation with iterative filtering and data selection for sinhala-english nmt. In 2021 Moratuwa Engineering Research Conference (MERCon), pp. 338-443. IEEE, 2021.
Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198(Supp) 27)::435-6455 2021.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. Ranking generated summaries by correctness: An interesting but challenging appli- cation for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2220- Florence, Italy, 2019. As- sociation for Computational Linguistics. doi: 10.18653/v1/P191 //aclanthologg..rggg//oo
URL https
Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov. FactKB: Generaliz- able factuality evaluation using language models enhanced with factual knowledge. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 933-952, Singapore, December 2023a. Association for Computational Linguistics. doi: htttps:/acclatthllee
URL
org/ 2023. emnlp-main.
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. ArXiv preprint, abs/ abss/305..nm 3300500823, 2023b. URL https: / /arxivv org/ 08283.
13

---

Published as a conference paper at COLM 2024
Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Synthetic data augmentation using gan for improved liver lesion classification. In 2018 IÉEE 15th international symposium on biomedical imaging (ISBI 2018), Pp. 289-299 IEEE, 2018.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ArXiv preprint, abs//220.07888 2022. URL httph:tt///// /attps://arxii. wwg.oogg//////..oooooo /209 .07558
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. ArXiv preprint, abs/2101.0002/ 2021. URL
htttpe///axxxv.org//b//////////
Leo John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization.
In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Gemini-Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/2312.1111 11805, 2023. URL
https://arxiv.ct org/ hwwpg//bbs2231... 11112...1885 11805
Gemini-Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Jean-aptstte Alayrac, Radu Soricut, Angelíki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, abs/ 2e550/033.055500, 2024. URLhttpps //arxiv org/ /bs/ 2403. 2403.05530 05530
Gemma-Teams Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. ArXiv preprint,
abs/2403.08295, 2024. URL http::tt// //arxiv/ org. abs///3b/4 08295
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd work- ers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30): c2305016120, 2023a.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd work- ers for text-annotation tasks. Proczedings of the National Academy of Sciences, 120(30): ppaa.2305001122 URL https: htttps://ww..r http://wwwwj// pnas. wwwpppa.......o..p..p org/doi/
e2305016120, 2023b. doi: 10.1073 10.1073/pna..2305// abs/10. 1ia/10.1073/paass 2tw.1/////maa22005100011211 2305016120
Ian Goodfellow, Jean
Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communii cations of the ACM, 63(11):1399144 2020.
Miguel Graça, Yunsu Kim, Julian Schamper, Shahram Khadivi, and Hermann Ney. Gen- eralizing back-translation in neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pp. 45-52, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/W199 https://accaanthologgy org/W19-5205
URL
Matthew Groh, Ziv Epstein, Chaz Firestone, and Rosalind Picard. Deepfake detection by human crowds, machines, and machine-informen crowds. Proceedings of the National Academy of Sciences, 119(1))221000111,, 2022.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. ArXiv preprint, abs 2401.03065, 2024. URL h/tpps:///axxxi http://.rg//bb////////// 03065
Luca Guamera, Oliver Giudice, and Sebastiano Battiato. Deepfake detection by analyzing convolutional traces. In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition workshops, pp. 666-667, 2020.
14

---

Published as a conference paper at COLM 2024
Aman Gupta, Deepak Bhatt, and Anubha Pandey. Transitioning from real to synthetic
data: Quantifying the bias in model. ArXio preprint, abs/ /ess/21000004144,, 2021. URL https
//arxiv. http///ss///a...a/aa/eaaaaaaaaaa 2105 04144
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves to program better. ArXiv preprint, abs/2220.14502,, 2022. URL https
//arxiv org wtw./////b///..//////////// 14502
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6626 6637, 2017. URL ptsppppopp// hewpeeeeiiinnnnnn.niiipee neurips sc/paper/20111 2017 http//aahh/////
Abstractt html
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https://pproceedinggs www.gs...urip..oo www.:nuuurs.c.ppppeeeeee wttpcc/cppppp... httpppappppppp//eppeeepp
Awstobbttsrraatt....... html
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of ooppuall large language model training, Advances in Neural Information Processing Systemss 35:30016-30030, 2022.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. q²: Evaluating factual consistency in knowledge -egollgudedg dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7856-7870, Online and Punta Cana, Domini- can Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 org/20211 eenlp-main 619
emnlp-main.619,
Bill Howe, Julia Stoyanovich, Haoyue Ping, Bernease Herman, and Matt Gee. Synthetic data for social good. ArXia preprint, abs/ 2017. URL https://arxix..... abs/1710.0820
Fan Huang, Haewoon Kwak, and Jisun An. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. ArXiv preprint, abs/2302.07736, 2023a. httpsss::///axi http://.ora/bbb//////e https//2322.0777...
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051-1068, Singapore, December 2023b. Association for Computational 22t/2/22..mmmmmmae..
Linguistics. doi: 2023. mat.3mmnn-mmmi.66
URL /ttp::
org/
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. ArXiv preprint, abs/ 22060056,,, 2022.
/tRp:stts://arx arg/abs//200) 2207 05608
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDi- armid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive lims that persist through safety training. ArXio preprint, abs/2400.05566, 2024. URL httph:ttt////e //tpx/vaavvvev//a org/ abs/ 2401. 05566
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys (CSUR), 55(12):1138, 2023.
15

---

Published as a conference paper at COLM 2024
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-languaage representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virfual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4904-4916. PMLR, 2021. URL/ptp://proceddngg mlr ppwss//139/jaa//// 2tmml.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ArXiv preprint, abs/2310.06825, 2023. URL https
//axxiv.org/aas//3100.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipula- tion with multimodal prompts. In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
Erik Jones, Hamid Palangi, Clarisse Simões, Varun Chandrasekaran, Subhabrata Mukherjee, Arindam Mitra, Ahmed Awadallah, and Ece Kamar. Teaching language models to 06827
hallucinate less with synthetic tasks, 2023. URL
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Sid- dhant Bhambri, Lucas Saldyt, and Anil Murthy. Llms can't 't plan, but can help planning in Ilm-modulo frameworks, arviv preprint Xiv:2402.00817, 2024.
Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. Cross-lingual training for automatic question generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1863-4872 Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v 11/5//11119-14L.. URL
Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Learning skillful medium-range global weather forecasting. Science, 381(6667) 2023.
Richard N Landers and Tara S Behrend. Auditing the ai auditors: A framework for evaluat- ing fairness and bias in high stakes predictive models. American Psychologist, 78(1):36, 2023.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screen- shots into html code with the websight dataset, 2024. URL /tttps:/arxiv.. org abs/ 2403.09000 09029
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 15:21314-21322 2022.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 0, 2022-06-27 Open Review, 62, 2022.
Kenton Lee, Mandar Joshi, Tulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893-18912 PMLR, 2023.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. ArXiv preprint, abs/ 1811.0787,, 2018. (tRLLhhtts///aaat http:/::////xxxa/a///o/e/ abs/ 07871
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augnented generation for knowledge-intensiee NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina
16

---

Published as a conference paper at COLM 2024
Balcan, and Hsuan- Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL
www.eddiiinnnnnnnss...mm httpi/ps..epppppe/22//////////// html
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlal Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022. URL htttts:://ariiiii www.aaawwww..iiaaaaaaaaraaaaaaae
14858
Bryan and Chris Callison-Burch. Paxqa: Generating cross-lingual question answering examples at training scale. ArXiv preprint, abs/2304.112066, 2023. URL. https://ra/
org/abs/ 2204. 12206
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong, math ca capabilities. ArXio preprint, abs /2403.040, 2024. URL htttt:ttts::///i/ http///axxvxvrrraa/raaaaaaasa orgg/bs///003 047006
Lizao Li, Rob Carver, Ignacio Lopez-Gomez, Fei Sha, and John Anderson. Seeds: Emulation of weather forecast ensembles with diffusion models. ArXiv preprint, abs abs/2306.14066, 2023a. U 14066
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of
instruction-follooingg models. htttps://githbb wwtp://suuoottttttteeoeeae 1ab/ hatpslaaaaaaaaaaaaaaaaaaaaaaaaae _val| 2023b.
htttpss//gttuubbccm//tttata/
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. ArXiv preprint, abss/22090077/ 2022. https::::///rte wwtpaa/www...././/////./..../ 07753
Baohao Liao, Shahram Khadivi, and Sanjika Hewavitharana. Back-translation for large- scale multilingual machine translation. In Proceedings of the Sixth Conference on Machine Translation, pp. 418-424, Online, 2021. Association for Computational Linguistics. URL https://aclanttologg..oooo
5.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ArXiv preprint, 0oss/2305.20055 2023. URL
wttpv//.orr/ab///300// 20050
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3214-3252, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653 /122:/1122222222/l//// 229. URL
htttps://ccaattholoogg wwg. acl- 229
Changshu Liu, Shizhuo Dylan Zhang, and Reyhaneh Jabbarvand. Codemind: A framework to challenge large language models for code reasoning. ArXiv preprint, abs/2402.09664. 2024a. URL h org/ aaep//bbb//22..../// 09664
Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. Deplot: One-shot visual language reasoning by alot-to-table translation. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 10381-10399, 2023a.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Eisenschlos. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12756- 12770, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
17

---

Published as a conference paper at COLM 2024
Haoxiong Liu and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing. ArXie preprint, abs/2401 0t///2401..00000 2024. URL battps://arxee org/ abs/2401.09000
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. Mitigating political bias in language models through reinforced calibration. In Thirty-Fift AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innooative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 14857- 14866. Press, 2021. URL httphhttpss/ssss.. aaai. orgg iid/xxndxxx... php/ AAAT/ att.aae/iiceceeee///eeeeee
17744
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai. Mind's eye: Grounded language model reasoning through simulation. ArXiv preprint, abs 2ass/222005559, 2022. URL htttss///axiiv org/ abs/ /b200 05359.
2210
Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models in simulated human society. ArXiv preprint, abs/2305.16966 2023c. URL pestps://arxxe org ht///bbb/2300 16960
Wei Liu, Weihao Zeng, Keging He, Yong Jiang, and Junxian He. for alignment? a comprehensive study of automatic data selection in instruction tuning. ArXiv preprint, abs/ 221 2023d. URL http:ttss///rrai htttppss/aarrir.crgaaa//222222 15685
What makes good data
Yingzhou Lu, Minjie Shen, Huazheng Wang, Xiao Wang, Capucine van Rechem, and Wenqi Wei. Machine learning for synthetic data generation: review. ArXiv preprint, abs/2302 200.02300,4062, 2023. URL http::tt:/// https://axxiv //axxiv www.aaiii....gaaaaa orgg/bs//3000/0 23022040222222222
Fernando Lucini. The real deal about synthetic data. MIT Sloan Management Review, 63(1): 144, 2021.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv preprint, abs/ /ooss/230.000 09988,, 2023a. URL httpt::t///arteee http//xxxvvv..ro org/ wtrg/abs/2308 2tt.//00088000033333
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. ArXiv preprint, bals//230.0088 2023b. URL https
//arxiv. org abs/ 230e.//336...m 08568
Benjamin Marie, Raphael Rubino, and Atsushi Fujita. Tagged back-translation revisited: Why does it really work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5990-5997, Online, 2020. Association for Computational Linguistics. doi: 10..633/v1/2200ac-mmiii 2020 21200acl-main.. 532
URL httpp://aclaanthooogg org/
Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. UniChart: A universal vision-language pretrained model for chart comprehension and reasoning.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14662-14688 Singapore, 2023. Association for Computational Linguistics. doi: 10. 10.18653/vv htt222222.mmmmmmaimi... ittps://accaantholgy 002/2023. emnlp-main.... 906
URL
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatric Membership inference attacks against language models via neighbourhood comparison. Arviv preprint, abs/2305.18462, 2023. URL 18462
18

---

Published as a conference paper at COLM 2024
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models:
Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462-477, 2022.
Meta. Meta and microsoft introduce the next generation of llama. https://ai.net meta com/
blog/llama-2 2023.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. Xiv preprint ar arX:::2305.1422 2023.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksan- dra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Subhabrata
Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. ArXiv preprint, abs/ 0td/2330600200 2023. URL
2306. 02700
Sergey Nikolenko. Synthetic data for deep learning, volume 174. Springer, 2021.
Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria- Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. Bias in data-driven artificial intelligence systems introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3),e1356, 2020.
OpenAL. Gpt-4 technical report, 2023.
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. Proving test set contamination in black box language models. ArXio preprint, abs/2310.17623, 2023. URL /tttps://axxiv.... www.aos.///oeeo.oo.ooo..ooo... /tt.ab///3...110.. 17623
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. ArXiv preprint, abs/ 2222222322255 2022. URL https://a//: org/ abs/2203. (tte/2233.02155)
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
URL httpps://peenreview
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. Arxiv preprint, abs/ Abs//230.06686, 2023. URL /tttps://aaattt
www.oooggg/////eoeeeee..oeeeeeee /210 06786
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces.
In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httpp:///pearreviev...o www.:oruntnoooooeooo gJEEMssHHK
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9419-3448, Abu Dhabi, United Arab Emirates, 2022. Association for Com- putational Linguistics. URL http:htts:///clll httattss///alaatlollooo..o wwg//022.........oo... 2022. emnlp-main 225
19

---

Published as a conference paper at COLM 2024
Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with modee-written evaluations. In Findings of the Association for Computational Linguistics: ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 13387-13434. Association for Computational Linguistics, 2023.
Hieu Pham, Xinyi Wang, Yiming Yang, and Graham Neubig. Meta back-translation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httLittt///oopeeeeeieee mtt::foorummmmddd3jjmmmmppppp
Michael Przystupa and Muhammad Abdul-Mageed. Neural machine translation of low- resource and similar languages with backtranslation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pp. 224-235, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.18653 /v155//1/// //W199-431. URL htttss//aclaatho.l. wwrg..rgg//. W19g/119-5431
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language su- pervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Interna- fional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 7748-8763.. PMLR, 2021. URL /
http://proceeddin
ord2aa html
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- -aan-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & wttps//...rg/aa////...... insights from training gopher, 2021. URL ptttps://arxive org/ http///bb22211.12 11446
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023. URL http::///appi.smmntiiccccoee
org/CorpusID
258959321
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv preprint, abs/ hbs//220.06125, 2022. URL
owwoooggggbo..o wwwp///...... 06125
Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano. Synthetic data augmentation for zero-shot cross-lingual question answer- ing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7016-7030, Online and Punta Cana, Dominican Republic, 2021. Asso- ciation for Computational Linguistics. doi:
wttp:////11122..emmppp.m atttps:/accaanthll.oe hwtpchhoooggorrr/2222 org/2021. eennl-main..6.
URL
Thomas Rid. Active measures: The secret history of disinformation and political warfare. Farrar, Straus and Giroux, 2020.
20

---

Published as a conference paper at COLM 2024
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Ad- vances in newral information processing systems, 15:36479-36499 2022.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathe- matical reasoning abilities of neural models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019,
URL/pttts://openeeviee wwtp://eeeeeeeeeeeeeeeoeeeo 11551155XX
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Anmual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 86-96, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10. 10.18653/ /v1 /v/11610099 wwwpaaantoooooooo//11----11
URL https
Siamak Shakeri, Noah Constant, Mihir Kale, and Linting Xue. Towards zero-shot multilin- gual synthetic question and answer generation for cross-lingual reading comprehension. In Proceedings the 14th International Conference on Natural Language Generation, pp. 35- 45, Aberdeen, Scotland, UK, 2021. Association for Computational Linguistics. URL attta://acclamtholog..oorgg www.or///.......111..... inlg-1.4
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelffh International Conference on Learning Representations, 2024.
2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Alexander Shypull Aman Madaan, Yimeng Zeng, Uri Alon, Alon,Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. perpriingeeefrommaanniiiprooooio code edits. ArXiv preprint, abs/2302..7222 2023. URL
htttpe//aarrav.oog//bb///2/// 07867
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-nd engineering?, 2024. URL/attps://art wwwaariii.../raaaaa /rg/abs/ 2403.03163
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. ArXiv preprint, abs/ 2b1//2222.,3138, 2022. URL https
//arxiv.org/ wttp://pppp....2........ 13138
Jacob Steinhardt. Ml systems will have weird failure modes. waw.aaaeeeeeeed----aaaaee--eeeee 2022.
21

---

Published as a conference paper at COLM 2024
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. ArXiv preprint, abs/ htss/2309.4525, 2023. URL btttps://axxiv.org/bbs//////
14525
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. ArXiv preprint, hos//330.053001, 2023. htt.hhtts::/aaat
wwwp/rgg///////////eoe 05301
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloinn Ilama model. htnde///tt////tttbbb.om////
wwwabbstaooodd..oo aaapaaafdddaadaaaaaaaaaaaaaaaaae 2023.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArX preprint, abs/ 0ttp/2211..9008 2022. WtLp:ttss://art org/ abs/
2211.09085
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Araas Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications. ArXiv preprint, abs/ /20s/220..uu 22/122200823 2022. URL httph:tt////t //axxiv www.aaawwvv....aaaaaaeae org/abs/ abtp//aa...2.... 08239
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn.
Fine-tuning language models for factuality.
In ICLR, 2023. URL https://api
semanticscholar ptrr//oorpusIDD :
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RS] International Conference on Intelligent Robots and Systems, pp- 5026-5033. IEEE, 2012. doi: 10.1109 1L.109//R05.201223366111
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXio preprint, abs/2307.0022 2023. URL
https://arxiv.. org/ /207 092888
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):47-4482 2024.
Boris Van Breugel, Zhaozhi Qian, and Mihaela Van Der Schaar. Synthetic data, real errors: how (not) to publish and use synthetic data. International Conference on Machine Learning, pp. 34793-34808. PMLR, 2023.
Alexander Sasha Vezhnevets, John P Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A Duénez-Guzmán, William A Cunningham, Simon Osindero, Danny Karmon, and Joel Z Leibo. Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia. ArXiv preprint, abs/ 0bss/2312.06644 2023. URL https //arxiv. org/ abs/ http////.....ee..eaeepaaaaeeeeee
2312. 03664
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? analysis of the limits of scaling datasets in machine learning. Xre preprint, abs/2211.04325, 2022. URLpttps: //arx/aa...v///////// org/abs/
2211 04325
22

---

Published as a conference paper at COLM 2024
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXio preprint, abs/ 2bss/230..6291, 2023. URLhtts:: org/ org/abs/ /bss/ 2305. 16291
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. 2022a.
11171
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with geff-generaeed instructions. volume /bbs/221..1002 2022b. httpsttss:///// //atxss htgg/bb//20 2212. 10560
Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen. Towards faithful
neural table-to-text generation with content-matching constraints. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pp. 1072-1086, Online, 2020, Association for Computational Linguistics. doi: 10.18653 1t0::///112222aa-mmmmo 101. URL pattts::/aclaantologg) wrg/ooggg/0000 2020 acl-main 101
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. https://openrreiee id GEggGZZG
Jerry Wei, Arief Suriawinata, Louis Vaickus, Bing Ren, Xiaoying Liu, Jason Wei, and Saeed Hassanpour. Generative image translation for data augmentation in colorectal histopathol- ogy images. In Advances in Neural Information Processing Systems, 2019.
Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc V. Le. Symbol tuning improves in-context learning language models. volume /ts//2305.82995 2023a. URL httthtts////xx/>> org/ abs/2305.0820
Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. Simple synthetic data reduces sycophancy in large language models, 2023b. URL https::/arrti:
brg//bs//535 /2088 03958
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. Long-form factuality in large language models. 2024.
http:p/ss//ppp...mmmaaccccccoooe wrwp:arroo//Co/ooeoeeaeoa 268724304
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. ArXiv preprint, abs/ 2232202222 2023c. URL httttstt/aaxxiiii org/
021/20
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. ArXiv preprint, abs/ 2b122.2.,,4399, 2021. URL https://arxiv.. http:://aaxxv..o/aab/a/////ss 02359
Erroll Wood, Tadas Baltrusaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J. Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 3661-3671 IEEE, 2021. doi: 10. /1109//CCCV8922.2021 00366.
202. 011.0010000011149992222200.03330
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. ArXiv preprint, /tts/2304.1244,, 2023. URL https:: /arxivivvvv....i wwwp://////bb////eeee 12244
23

---

Published as a conference paper at COLM 2024
Jiahao Xu, Yubin Ruan, Wei Bi, Guoping Huang, Shuming Shi, Lihui Chen, and Lemao Liu. On synthetic data for back translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologiees pp. $19-430, Seattle, United States, 2022. Association for Computational Linguistics. doi: 10.18866/v11 2022. maacl-main. 32
Utt.htttss//ccllatttoooo org/
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arviv preprint ar arXi:22010.1134,, 2020.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standard- izing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024.
Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, and Xuanjing Huang. Toolsword: Unveiling safety issues of large language models tool learning across three stages. ArXio preprint, abs/ 2b0//240..0053, 2024. URL https://arxiv.orrt abs wtw..b///22....o 10753
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv preprint, 0ttp/2309.12250 2023.
12284
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. ArXiv preprint, abs/ (bs/2440.10020 2024.
URL httthhtts//arxxtt org/ hte///bb/240000 10020
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. ArXia preprint, abbs/2308.0888 2023. URL http:////aaawvv.orr////2/0//0000
E. Zelikman, Yuhuai Wu, and Noah D. Goodman. Star: Bootstrappinggreasoning with reason- ing. In NewrlPS, 2022. URL httppp//wgg...eeea......aaaaa//p
Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for Ilm agents: A social psychology view. ArXiv preprint, abs//3100002124, 2023a. URL hhttps
//arxiv. caaxiv.org/aae aba/ 02124
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction tuning for large language models: A survey, 2023b. URL
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, ArXiv preprint, abs/ /2ts//300.1100 2023c. URL pattts://arx:ar org/ abs/ 2206233066 17107
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: survey on hallucination in large language models. ArXiv preprint, abs /bs/2239.0 2023d. URL htttp:://arxvv.orgaaaaa
01219
Bo Zhao, Boya Wu, and Tiejun Huang. Svit: Scaling up visual instruction tuning. ArXio preprint, 20000007784 2023. httptttp:e/ariv..oo////aaaa wttp://.wwww..aa//.....ooo 04087
24

---

Published as a conference paper at COLM 2024
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 20, New Orleans, Louisiana, URL
2018. Association for Computational Linguistics. doi:
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher. The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science advances, 8(18)eabk2607, 2022.
Zaixiang Zheng, Hao Zhou, Shujian Huang, Lei Li, Xin-Yu Dai, and Jiajun Chen. Mirror- generative neural machine translation. In 8th International Conference on Learning Represen- fations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https://openreviex. htw.:/neeeeeeeeeee wttpe/wwwwwweettnnnnneee weet/forummidd-Hot@@RTTT
Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, and Maarten Sap. Is this the real life? this just fantasy? the misleading success simulating social interactions with llms. ArXiv preprint, abs/ abs/2/t /4033000000000 2024.
org. abs
/ 2403. 05020
Caleb Ziems, Jane Dwivedi Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. Normbank: A knowledge bank of situational social norms. ArXiv preprint, abs/2305.17708, 2023. URL 17008
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. ArXiv preprint, abs/2307.1504/ 2023. URL https: //arxiv.///ii.ee/aa abg/ 15043
25


Reviews:
[{"review_id": "MaEDbUIU1w", "reviewer": "Reviewer_MjuL", "summary": "The paper provides best practices of synthetic data research from the lens of various tasks/case-studies/domains. It buckets the research  directions in the domains both from training and evaluation standpoint. It also describes the challenges of synthetic data generation research in the each of the domains, however falls short on the sufficient \"empirical evidence\" mentioned in the abstract. \nThe primary categories during training are: \n* reasoning\n* tool-using and planning\n* multimodal\n* Multilingual\n* Alignment. \n\nWhereas, the categories for same in evaluation are: \n* hallucination\n* safety\n* assisting human evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful reviews and great suggestions! We are glad to see you find our paper a great survey on important case-studies of using synthetic data. Below we address your questions:\n\n**Q1: Missing citation**\n\nYes we will add the citation [1] to the proper position. Thanks for reading our paper carefully!\n\n**Q2: Referring to a prior art**\n\nYes it is indeed an important work we can refer to. We will add related text and cite this work in our final version.\n\n**Q3: Why “mitigating hallucination” is under “Alignment” section**\n\nWe are following some prior arts on the definitions: For example, in [2], the author defined the “mitigating hallucination” problem as one of the “Knowledge Alignment” problems.\n\n**Q4: Really enjoying the “direction of future work” section**\n\nThanks! We are so glad you love reading this section! Hope it helps your future research!\n\nFinally, we thank the reviewer again for the insightful comments!\n\n[1] [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/abs/2307.09288)\n\n[2] [​The Knowledge Alignment Problem: Bridging Human and External Knowledge for Large Language Modelse](https://arxiv.org/abs/2305.13669)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Provides necessary and important case-studies for usage of synthetic data that helps different LLM benchmarks.\n\n* For every case-studies, the paper references the important synthetic data papers, which can serve as an interesting survey of some sorts.", "reasons_to_reject": "* While the paper does provide various case-studies, it doesn't *\"delve\"* much into the empirical evidences on various techniques.\n\n* There are various techniques highlighted for each case-study in the paper, it would be good to have some remarks on \"best practices\" on each of them at least on some empirical level. \n\n* The crucial \"take-home\" message is missing for each domain.", "questions_to_authors": "* (Possible) missing citation: ? in \"Aligning with shared human preference ....\" paragraph\n\n* Reference to, and discussion on \"Wang et al, Data Management For Large Language Models: A Survey\" can help improve the paper\n\n* It is slightly unclear why \"mitigating\" hallucination has been clubbed under alignment subsection.\n\n* I *really* enjoyed the \"directions of future work\". Kudos!", "extra_scores": {}}, {"review_id": "TzyhgHkQey", "reviewer": "Reviewer_5Caz", "summary": "The paper presents a survey of existing methods on synthetic data research, and its various applications. The paper discusses synthetic data generation techniques across various tasks like reasoning, multilinguality, multi-modality, etc. Next, the paper discusses the use of synthetic data in evaluations. Finally, challenges and future directions for research in this area are discussed. The paper is well-written and easy to follow.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reading our work carefully and the high rating! We are glad you think our work is a timely contribution to the community! Below we address questions: \n\n**Q1: Typos**\n\nYes we will of course fix these typos in our final version. Thanks again for reading our paper carefully!\n\n**Q2: Additional references**\n\nThanks for the great suggestions on additional references! Since we are not allowed to update our paper during rebuttal, we will make sure we have them in our final version!\n\nFinally, thanks for the careful reading and great suggestions! Hope our work can help your future research!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work is a timely contribution as the NLP community is rapidly moving towards using synthetically generated data for scale. \n\n2. The work comprehensively touches on various application areas and discusses a detailed list of literature for each of these areas. Each section is a great starting point for a future researcher looking to leverage automatically generated data.", "reasons_to_reject": "1. While most of the literature reviewed in this paper understandably pertains to the the post-LLM era, synthetic data generation strategies have had a rich history even before the era of transformers. For instance, annotation projection [1,2], systematic perturbation[3,4], bi-/multi-linugal pivoting [5], to name a few. A section discussing these methods would make the work more complete. \n2. The discussion on the consistency of LLM outputs (or the lack thereof) deserves to be mentioned in detail. Several works have shown inconsistency as a problem [6,7,8]  \n\n[1] Inducing Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora (Yarowsky et al., HLT 2001) \\\n[2] Multilingual Projection for Parsing Truly Low-Resource Languages (Agić et al., TACL 2016)\\\n[3] The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languages (Wang & Eisner, TACL 2016)\\\n[4] Synthetic Data Made to Order: The Case of Parsing (Wang & Eisner, EMNLP 2018)\\\n[5] Paraphrasing with Bilingual Parallel Corpora (Bannard & Callison-Burch, ACL 2005)\\\n[6] CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (Ravichander et al., EMNLP 2022)\\\n[7] Consistency Analysis of ChatGPT (Jang & Lukasiewicz, EMNLP 2023)\\\n[8] Language models are not naysayers: an analysis of language models on negation benchmarks (Truong et al., *SEM 2023)", "questions_to_authors": "- Typographical errors:\\\n    - Page 2. Para 2: ‘’synthetic’’ -> ‘’synthetic data’’\n    - Page 6: Missing citation under ‘’Aligning with shared human preference and values’’\n- Some additional literature:\n    1. Section 2.4: \n        - Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (Singh et al., 2024)\n    2. Section 2.5:\n        - SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (Manakul et al., EMNLP 2023)\n    3. Section 4:\n        - The Generative AI Paradox: \"What It Can Create, It May Not Understand\" (West et al., 2024)", "extra_scores": {}}, {"review_id": "biY5FiNRYN", "reviewer": "Reviewer_gT5a", "summary": "This paper surveys recents results in the topic of synthetic text data used to train and evaluate machine learning algorithms such as large langage models (LLM) for natural language and code. The results are grouped in topics such as use-case and type of tasks on which synthetic data is being used to enhance performance. A number of limitations are discussed and several directions for future work are proposed.\n\nThe work provides a useful survey of how synthetic text data is being used. However, I believe that the paper would benefit from better scoping, articulating its methodology, and applying a critical lens to the field to nuance some of the « hyped »  statements.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful reviews and constructive feedback! We are glad our paper is useful in providing a comprehensive reference for future research on synthetic data for LLMs. Below we address your questions:\n\n**Q1: Adding formal definition of synthetic data**\n\nYes we will have a better definition and categorization of different types of synthetic data (mainly categorized by the generation approach) in our final version.\n\n**Q2: Privacy concerns on synthetic data**\n\nThanks for the great question and many useful references! We agree with the reviewer that whether synthetic data can is still an open question --- our original claim was mainly based on some prior art that claimed synthetic data can mitigate the concerns about personal privacy (e.g., Harnessing large-language models to generate private synthetic text). In our final version we will leave the room for further discussion on this and will include the work you mentioned as references for different opinions. Thanks again for the thoughtful question!\n\n**Q3: Current situation of template-based synthetic data generation**\n\nAlthough now generating synthetic data with LLMs seems to become the dominant method, we still see in some applications template-based generation can still play a role. For example, Google DeepMind used sympy + python to create millions of synthetic arithmetic QAs to aid the LM to gain better mathematical abilities [1]. In general there seems to be no clear boundary; it more depends on the performance of the method on specific applications.\n\n**Q4: Adding tables of public available synthetic datasets**\n\nGreat suggestion! We will add one in our final version.\n\n**Q5: Adding more related work in the early years**\n\nThanks for the suggestion! It is indeed important to cite some related work in early years so that our readers can better understand the past and present of synthetic data research. We will include more in our next version!\n\n**Q6: Adding more discussion**\n\nThanks for the suggestion! We agree with the reviewer that how to make sure training on synthetic data would not cause “model collapse” is still a challenge for synthetic data research. We will add more discussion on this in our final version.\n\nFinally, we really appreciate the reviewer’s thoughtful comments and constructive suggestions! Hope our rebuttal can solve your concerns and questions. Thanks!\n\n[1] [Analysing Mathematical Reasoning Abilities of Neural Models](https://github.com/google-deepmind/mathematics_dataset)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a useful survey of use cases/tasks on which synthetic text data has been successfully used to train/evaluate ML algorithms. Thus, the paper can serve as a reference for readers interested in understanding this space and in re-using existing synthetic text datasets similar use cases.", "reasons_to_reject": "- The scope of the paper is not clear from the Title, Abstract and Introduction. The paper presents itself as a survey of synthetic data in general, but its scope is more narrow: synthetic text data mainly generated using hand-engineered templates and algorithms (but not generative algorithms). This is not a bad thing : a review of synthetic text data can still be very useful. However, clarifying the scope is essential to reaching the intended audience, because what most people understand by « synthetic data » is artificial data generated by an AI model, e.g., a GAN or a Bayesian network, which was trained on real data.  \n\n- The paper is very extensive on the applications of synthetic text data, but is missing a taxonomy/categorization of the synthetic data generation models. The introduction does not delineate the boundary between human-aided techniques (such as algorithms/simulations) and fully data-driven synthetic data (« rather than being directly created by human. » ). A table of available synthetic datasets grouped by method and/or use case is missing.\n\n- The methodology used to identify the tasks to which synthetic text data is applied is not clear. It is not clear if the review is exhaustive of the field, and how « good » use cases are delineated from « bad » use cases.", "questions_to_authors": "Questions:\n\n- How does your work relate to other surveys in this space, e.g., https://arxiv.org/abs/2205.03257? \n\nMain comments:\n\n- Please consider adding formal definitions and corresponding terms for the various types of synthetic data considered in this work, such as template-generated, simulated, algorithm-generated synthetic data, etc.\n\n- The statement that synthetic data can create « anonymized » datasets « that do not contain sensitive personal information » is incorrect. An important missing reference is Stadler et al. [A] which demonstrates that synthetic data is not immune to inference attacks (such as membership inference and attribute inference), one criteria for anonymisation as per the Working Party 2014’s Opinion on Anonymization techniques. The notion that synthetic data is inherently privacy-preserving has been seriously challenged (see also Houssiau et al, 2022 [B] and Meeus et al., 2023 [C]). Unless formal privacy protections are put in place (differential privacy guarantees), synthetic data will leak private information, especially about vulnerable records. How to achieve synthetic data that is truly privacy-preserving while remaining useful is an open problem. \n      \n- How much of the literature in this space relies on template/algorithm-generated synthetic data, with prior knowledge infused by humans and how much of it uses LLM-generated synthetic data ? When is it better to use one over the other?\n\n- Please consider adding a table with the datasets which are publicly available. This could help this paper become a useful starting point for identifying already existing synthetic text datasets.\n\n- The paper surveys mainly results from the last couple of years, however synthetic data similar to the focus of this work has already been used in the past, see e.g., Kocijan et al.[D]’s dataset for solving the Winograd Schema Challenge. Please consider including citations from older work and incorporating their contribution in the review.\n      \n- The question asked in the «emergent self-improvement capability » paragraph seems to have already been (negatively) answered in Shumailov et al. [E] Please consider including a discussion of this paper and updating the conclusions in light of their finding.\n\n[A] Stadler, T., Oprisanu, B., & Troncoso, C. (2022). Synthetic data–anonymisation groundhog day. In 31st USENIX Security Symposium (USENIX Security 22) (pp. 1451-1468).\n\n[B] Houssiau, F., Jordon, J., Cohen, S. N., Daniel, O., Elliott, A., Geddes, J., ... & Szpruch, L. (2022, October). TAPAS: a Toolbox for Adversarial Privacy Auditing of Synthetic Data. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research.\n\n[C] Meeus, M., Guepin, F., Creţu, A. M., & de Montjoye, Y. A. (2023, September). Achilles’ heels: vulnerable record identification in synthetic data publishing. In European Symposium on Research in Computer Security (pp. 380-399). Cham: Springer Nature Switzerland.\n\n[D] Kocijan, V., Cretu, A. M., Camburu, O. M., Yordanov, Y., & Lukasiewicz, T. (2019, July). A Surprisingly Robust Trick for the Winograd Schema Challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 4837-4842).\n\n[E] Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023). The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493.", "extra_scores": {}}, {"review_id": "wQsbwSJYyE", "reviewer": "Reviewer_XqGy", "summary": "This survey paper presents a comprehensive summary of the uses of synthetic data, issues with its quality, and possible research directions that are relevant to explore in the realm of developing and utilizing synthetic data.\nThe discussion of applications where synthetic data is used and can benefit is very thorough and gives an almost complete view of the ways researchers generate and use synthetic data nowadays in training models, the section about the use of synthetic data for evaluation is less detailed since it doesn't refer to the fact that data that is synthesized to train models is also being used to evaluate the models by using some sort of partition to train, test and dev datasets. \nThe last two sections are a good starting point for researchers to examine when they consider developing synthetic data and/or working with it, though I thought that some of the points there are obvious and may need to be explained from a different perspective (for example, how improving the quality of synthetic datasets different than improving the quality of non-synthetic ones). The part about the self-improving models through synthetic dataset creation is very interesting though.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews! \n\nWe agree with the reviewer that it would be better to include additional discussions on the less-known issues of synthetic data, and raise more open questions for further discussion. Specifically, we plan to add more discussion on the privacy protection of synthetic data, and potential techniques that can mitigate model collapse if the synthetic data is not carefully produced. We hope these additions can further improve our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A helpful survey for researchers interested in creating synthetic data or developing methods for generating synthetic data.\n* Some of the future research directions mentioned in the paper are thought-provoking.", "reasons_to_reject": "If this was not a summary paper, I wouldn't write this comment as a reason to reject the paper, but since it is then some of the points mentioned in the paper are obvious and the authors could have focused more on the less-known issues with synthetic data, and raise more questions in the future research direction section.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Overall, reviewers found the paper useful: they appreciate having a post-LLM era review paper, and felt this would be a good resource for the field. However, there were some points that didn't fully land with them, so they gave useful suggestions to help the paper reach its full potential. In particular, the paper's scope and contextualization should be clarified for the next version, as recommended by reviewers gT5a and 5Caz. In particular, attention should be paid to addressing changes since the last review paper mentioend by 5Caz, and including related work mentioned by all reviewers. It could also be nice to include a take-home message as suggested by reviewer MjuL, at the end of each section.

Everyone participated in rebuttal, and useful info was exchanged.
Decision: Accept