Forewarned is Forearmed:  Harnessing LLMs for Data Synthesis via Failure-induced Exploration

Authors: Qintong Li, Jiahui Gao, Sheng Wang, Renjie Pi, Xueliang Zhao, Chuan Wu, Xin Jiang, Zhenguo Li, Lingpeng Kong
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: data synthesis, preference learning, LLM alignment

Abstract:
Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data, leading to impressive performance across a range of downstream applications. Current methods often rely on human-annotated data or predefined task templates to direct powerful LLMs in synthesizing task-relevant data for effective model training. However, this dependence on manually designed components may constrain the scope of generated data, potentially overlooking critical edge cases or novel scenarios that could challenge the model. In this paper, we present a novel approach, ReverseGen, designed to automatically generate effective training samples that expose the weaknesses of LLMs. Specifically, we introduce a dedicated proposer trained to produce queries that lead target models to generate unsatisfactory responses.  These failure-inducing queries are then used to construct training data, helping to address the models' shortcomings and improve overall performance. Our approach is flexible and can be applied to models of various scales (3B, 7B, and 8B). We evaluate ReverseGen on three key applications—safety, honesty, and math—demonstrating that our generated data is both highly effective and diverse. Models fine-tuned with ReverseGen-generated data consistently outperform those trained on human-annotated or general model-generated data, offering a new perspective on data synthesis for task-specific LLM enhancement.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730680505465

thread_type:
official_review

<thread_object>
creator: reviewer_yadx

<content>
summary:
This paper introduces REVERSEGEN, a method for generating training data by identifying model weaknesses through failure-inducing exploration.

strengths:
* Novel approach to data synthesis through failure exploration
* Comprehensive evaluation across three important tasks (safety, honesty, math reasoning)

weaknesses:
* The paper lacks clear justification and motivation for why generating failure-guided data would improve model performance
* No theoretical framework explaining why failure cases would be more valuable than standard training data
* Table 6 shows similar result w and wo failure induction in math reasoning task, does this mean failure induction does not benefit math reasoning tasks?
* No analysis of computational costs or token/API budget comparisons with baseline methods
* Reward mechanism not clearly explained
* Insufficient baseline comparisons, especially for mathematical reasoning task

questions:
* Could you provide theoretical or empirical justification for why failure-inducing data would be more valuable than standard training data for model improvement?
* How does REVERSEGEN's performance evolve with increasing iterations? What determines convergence?
* Could you analyze why failure induction appears less effective for mathematical reasoning (Table 6) compared to other tasks?
* What are the computational requirements (tokens, API calls, training time) compared to baseline methods?
* Could you provide more details about the reward mechanism design and validation process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692543297

thread_type:
official_review

<thread_object>
creator: reviewer_hfrr

<content>
summary:
The paper proposes a technique called ReverseGen for generating failure-inducing examples for an LLM. The technique uses a proposer model that is fine-tuned using pairs of positive and negative examples. The evaluation shows that the generated data can be used to fine-tune and improve models in safety, honesty, and mathematical reasoning.

strengths:
- The paper introduces a new technique for synthesizing failure-inducing data for a target LLM 

- The technique is effective in 3 distinct domains of safety, honesty, and mathematical reasoning and shows improvement on SOTA on each. I appreciate the inclusion of Table 5 and Table 7 with examples in the evaluation section

weaknesses:
The writing of the paper, especially the separation of the technical section and implementation details needs to be improved. Many of the evaluation choices seem a bit arbitrary and need to be organized better to understand how each of the applications of the proposed technique fits into a single framework (if they do). 

- Line 126: “ these studies tend to focus on particular tasks or inputs, and overlook the task generalization and the data salience on model enhancement. Our work differs from these prior works in that we focus on generating failure-guided data for target models”
Can you be more specific in comparison with each of these prior works? It is unclear to me how ReverseGen differs from many of these works mentioned in the related works


- Line 176: The term “solvable” and “unsolvable” is defined here and never mentioned here. The term “solvable” for positive examples is quite unclear. Use a more appropriate name for this.


- Line 202: Section 3.2 is the technical section and it starts talking about “gpt-40-mini”. I would recommend authors to separate the implementation details from the technical section. “Gpt-40-mini” is a specific model and used to label the responses, define the model used for labeling as a parameter of the technique that’s instantiated as a specific model. 

## Minor:
- Line 189: “We begin by warming up the proposer model $M_{prop}$ with the initial task-specific instruction set” - A bit unclear wording. Can be more technically precise, especially in the technical section of the paper

- Line 190: “3-shot prompts” - this seems like implementation detail as well which would be more appropriate if it was in the evaluation section

- Line 204: What’s R(.)? I don’t see it defined anywhere before

- Line 220: “$M_{ref}$ is the reference model that remains unchanged”: this doesn’t really define what is $M_{ref}$

- Line 227: typo - two dots “..”

- Line 275: why were these hyperparameters chosen?

- Line 284: “Responses from the target model are generated using greedy decoding for deterministic quality measurement, with the temperature parameter set to 0” - Doesn’t greedy already mean temperature doesn’t matter?

- Line 287: “Instructions for the SFT data are produced by proposer models, while responses are derived from the target models using tailored prompts for safety tasks, generated by gpt-4o-mini for knowledge-intensive tasks.” -> the sentence is too long and hard to understand. What are “knowledge-intensive tasks” in this context?

- Line 319: “well-safe”?

- Line 429: “ReverseGen solely relies on synthetic data” - Doesn’t it use MMLU as the initial instruction seed?

- Line 523: “This may result from inadequate assessment of REVERSEGEN’s proposed questions by the current benchmark or the need for more effective fine-tuning algorithms for difficult questions.” - a bit unclear

questions:
Since the technical section and implementation details are somehow mixed, it is really hard to understand some of the details of each of the experiments. I have the following questions about the evaluation:

- In what experiments, GPT-40-mini used? How was this decided? Is there an ablation study on its use?
- For the honesty calibration and mathematical reasoning experiment, what number of "proposer fine-tuning" iterations are used in ReverseGen? 
- Why is Llama3 used in 4.4 but not in the previous experiments?

Can you add a table that maps each of the variables in the technical section to the specific choice made in each experiment? This would make it easier for the reader to understand each experiment. For instance, the columns could be $M_{prop}, M_{tgt}$, number of examples, seed data, number of iterations, use of GPT-4o-mini, and other implementation details for each experiment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703920656

thread_type:
official_review

<thread_object>
creator: reviewer_29jz

<content>
summary:
This paper proposes a data synthesis approach by training a data generator and leverage the performance of the target model as a training signal. Specifically, the predictions of the target model are used to construct a preference dataset (target model's failure cases are preferred) for the training of the data generator which performs DPO on top of those preference data.

strengths:
* The paper is overall well written and easy to understand.
* The proposed approach is novel. It performs RLHF to train the data generator, with the target model works as a preference provider.

weaknesses:
The experiments miss 2 significant baselines. 
* To verify the effect of the proposed RLHF approach, there should a baseline finetuning the data generator (proposer LLM) with a collection of failed samples, and generate a dataset.
* A strong LLM (i.e., gpt-4o) plays an important role in the proposed method when obtaining the oracle label, so there should another baseline directly prompting the gpt-4o multiple times to generate a synthetic dataset.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719218285

thread_type:
official_review

<thread_object>
creator: reviewer_u19j

<content>
summary:
The authors propose an iterative finetuning method for finetuning a target langague model by using synthetic data generation from a proposer language model which proposes harder and harder questions to a target language model. This is in effect a curriculum learning approach which trains a target model on harder and harder samples. The proposer model is also trained to propose harder and harder questions by using errors in the target model’s answers.

A proposer language model generates few-shot candidate questions. Then the target model predicts answers to these questions. The answers are then compared to gpt-4o-mini's answers, which is used as a gold-standard. If the answers agree then this question is placed into the negative set {x^-}. If the answers from the target model does not agree with the gpt-4o-mini then the question is placed in the positive set {x^+}. These sets are then used by DPO is used to finetune the proposer model to produce harder and harder samples by leveraging the positive and negative sets. Finally, the proposer model generates synthetic questions which are deemed hard for the target model, labels are generated by the proposer model or gpt-4o-mini. The target model is then trained with SFT.

strengths:
- The idea of using the language model prediction errors to create a set of easy and hard examples to train a proposer model with DPO is novel and a nice idea.

- Interesting results that show that harder to predict data points aka using a curriculum harder and harder questions is beneficial for some red teaming and honesty benchmarks. This is interesting since, in comparison other papers such as [1], show that hard samples actually hurts target model performance albeit in a different dataset domain.

- Wide range of experiments: red teaming, honesty and mathematical reasoning to demonstrate that the method can generalize to multiple domains.

[1] Evans, Talfan, et al. "Bad students make great teachers: Active learning accelerates large-scale visual understanding." arXiv preprint arXiv:2312.05328 (2023).

weaknesses:
- Weak results on mathematical reasoning which do not demonstrate considerable improvement in performance. Nor are many other similar iterative methods which have some sort of synthetic question-answer prioritization compared to [2, 3].

- No ablation experiments, what is the performance with 1 iteration versus 5? What if you generate 1k or 10k samples to populate the positive and negative sets for DPO training?

[2] Lee, Nicholas, et al. "Llm2llm: Boosting llms with novel iterative data enhancement." arXiv preprint arXiv:2403.15042 (2024).
[3] Jiang, Yuxin, et al. "Lion: Adversarial distillation of proprietary large language models." arXiv preprint arXiv:2305.12870 (2023).

questions:
- What is $R(\cdot)$ in line 204?

- Performance for ReverseGen in Tables 1 and 2 has not saturated with increasing iterations and it seems like you are under reporting results. What would happen if you ran this for 4 or 5 iterations? At what point would performance saturate? What if you generated 10000k instruction candidates and performed t=1 iteration versus 2000k instruction candidates with t=5 iterations?

- When using harder and harder samples to train a model for example in active learning, or data selection, where data points are prioritized with a larger loss then this can cause a negative feedback loop with a catastrophic drop in performance from a target model. Did you observe similar artifacts in your experiments?

- In Tables 1, 2, 3 you have a rows ‘without failure induction’ but you do not describe what this ablation is?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732740299431

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated Manuscript and Response to All Reviewers

comment:
We sincerely thank all the reviewers for their feedback and constructive comments. We appreciate the reviewers' recognition of the novelty and significance of preference learning from failure feedback for data synthesis (R#1-u19j, R#2-29jz, R#3-HFrR, R#4-yaDx), the clear presentation of our paper (R#2), as well as the comprehensive and effective results across diverse tasks (R#1, R#2, R#4). 

At the high level, we aim to:
1. investigate whether iteratively preference learning from model failures can effectively produce **new challenging samples**, particularly for models pre-trained on task-related data, and
2. evaluate the impact of training models on challenging samples by applying ReverseGen to three distinct tasks.  These guide our methodology and experimental design.    

In response to the reviewers’ comments, we have conducted **four** additional experiments (including two baselines and two analyses) and revised the manuscript accordingly. **Changes are highlighted in blue in the updated PDF.** The updates are summarized as follows: 

1. **Sections 2.1 & 3**: Added detailed explanations of prior works and revised the methods section for clarity, reducing implementation details (R#3). 
2. **Section 4.1 & Appendix C**: Enhanced the reward process and parameters with a detailed paragraph and table (R#3, R#4). 
3. **Figure 2**: Analyzed iteration times for ReverseGen performance (R#1, R#4). 
4. **Table 3**: Clarified the experimental setup (R#3). 
5. **Table 5**: Included results for the iterative LLM2LLM approach (R#1). 
6. **Sections 4.4 & Appendix G.1**: Added explanations for math performance and provided a detailed results analysis (R#1, R#4). 
7. **Appendix A**: Added motivation for failure-inducing exploration (R#4). 
8. **Appendix E.2**: Provided analyses of iteration times and preference training data (R#1). 
9. **Appendix G.2**: Included additional details on prompting powerful models (R#2). 

We have addressed the queries and suggestions raised by each reviewer in detail and provided clarifications where necessary. We greatly appreciate the reviewers’ valuable contributions, which have helped us improve the quality of our manuscript. Should further clarification be required to assist in advancing our score, please do not hesitate to let us know. 

Thank you once again for your review and feedback!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733144555466

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We sincerely appreciate the time and effort you have dedicated to evaluating our paper. Your insightful feedback and constructive comments have been invaluable in improving our manuscript and addressing areas of potential confusion. We have carefully addressed each of your comments, provided additional experiments, and updated the manuscript to clarify our points.

As the discussion deadline approaches (in less than 23 hours), we would be grateful to receive any additional feedback you may have.

Best regards,

The Authors of 8852
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents ReverseGen, a novel approach for generating training data by identifying model weaknesses through failure-induced exploration. The method uses a proposer model fine-tuned via preference learning to generate challenging examples that expose target model failures. Key strengths include: (1) A novel framework for data synthesis through failure exploration, (2) Comprehensive evaluation across safety, honesty and math reasoning tasks, and (3) Demonstrated improvements over baselines. Main weaknesses initially centered on unclear theoretical justification, limited baseline comparisons, and analysis of computational costs. However, the authors provided extensive experimental results and analysis during rebuttal that adequately addressed these concerns. The paper makes a meaningful contribution to data synthesis for LLM training and warrants acceptance.