Title: UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data Augmentation

Diffusion Models

Training Data Valuation

Few-Shot Learning
Abstract: Data augmentation using generative models has emerged as a powerful paradigm for enhancing performance in computer vision tasks. However, most existing augmentation approaches primarily focus on optimizing intrinsic data attributes -- such as fidelity and diversity -- to generate visually high-quality synthetic data, while often neglecting task-specific requirements. Yet, it is essential for data generators to account for the needs of downstream tasks, as training data requirements can vary significantly across different tasks and network architectures. 
To address these limitations, we propose UtilGen, a novel utility-centric data augmentation framework that adaptively optimizes the data generation process to produce task-specific, high-utility training data via downstream task feedback. Specifically, we first introduce a weight allocation network to evaluate the task-specific utility of each synthetic sample. Guided by these evaluations, UtilGen iteratively refines the data generation process using a dual-level optimization strategy to maximize the synthetic data utility: (1) model-level optimization tailors the generative model to the downstream task, and (2) instance-level optimization adjusts generation policies -- such as prompt embeddings and initial noise -- at each generation round. 
Extensive experiments on eight benchmark datasets of varying complexity and granularity demonstrate that UtilGen consistently achieves superior performance, with an average accuracy improvement of 3.87\% over previous SOTA. Further analysis of data influence and distribution reveals that UtilGen produces more impactful and task-relevant synthetic data, validating the effectiveness of the paradigm shift from visual characteristics-centric to task utility-centric data augmentation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile most existing generative data augmentation methods prioritize the generation of high-fidelity and diverse images, this paper argues that practical tasks and model architectures often necessitate tailored training data distributions rather than generic visual quality. To address this, the authors propose Utility-Generated Data Augmentation (UtilGen), a framework that explicitly optimizes synthetic data to enhance task-specific performance, rather than merely adhering to visual standards. UtilGen introduces three key components: 1) Task-Oriented Data Valuation, which is a meta-learned weight allocation network that quantifies the task-specific utility of synthetic data. 2) Model-Level Generation Capability Optimization, which finetune the generator model (stable diffusion) using  DPO. 3) Instance-Level Generation Policy Optimization, which optimizes prompt embedding and input noise to maximize the utility weights of generated images. Extensive experiments across 8 datasets demonstrate that UtilGen outperforms prior state-of-the-art (SOTA) methods by an average of 3.49% in performance, validating its effectiveness in aligning synthetic data with downstream task requirements\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n•\tThe paper's main strength is its proposal of a utility-centric paradigm for generative data augmentation. This is a departure from the conventional focus on visual fidelity and diversity, offering a more targeted and more effective approach to data augmentation.\n\n•\tA meta-learned weight allocation network provides an efficient, retraining-free solution for task-oriented data valuation.\n\n•\tModel-level and instance-level optimization provides a comprehensive framework for tailoring both the generative model and the individual generated samples for maximizing utility weights.\n\n•\tPaper provides rigorous experimental results on various datasets, showcasing an average of 3.49% performance improvement compared to previous SOTA approach\n\n•\tPaper provides all necessary ablation study and results which provide valuable insights into why the proposed method is effective.\n\nWeakness:\n\n•\tThe proposed framework's complexity arises from its integration of multiple modules and training stages (e.g., TODV, MLCO, ILPO, DPO, textual inversion), which may pose challenges for practical implementation\n\n•\tWhile the paper emphasizes the efficiency of its utility evaluation method, the overall framework, e DPO fine-tuning and instance-level optimization, could be computationally intensive. The computation cost is given in the appendix.\n\n•\tThe performance of UtilGen is likely dependent on the quality of the initial classifier and the representativeness of its feature, as the utility weightage network uses the feature from the classifier model. The paper could have explored the sensitivity of overall method on choice of different classifier such as CNNs vs pretrained transformer models.\n\n•\tPaper could also compare their approach against simple augmentation or self-supervised pretrained models.\n\nQUESTIONS:\n1.\tHow sensitive is the performance of UtilGen to the choice of the initial classifier architecture and training/val data selection?\n\n2.\tAre there any scenarios or types of datasets where use of UtilGen might not be effective?\n\n3.\tThe framework seems to have several hyperparameters for each components. How sensitive is the method to the choice of these hyperparameters, and do you have any guidelines for setting them?\n\n4.\tHow does this approach perform when we use pretrained vision transformer models as the classifier?\n\nLIMITATIONS:\nYes, listed in appendix", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of designing data-generation models that are well aligned with a downstream task. A key element of this setup is a weight network, which learns to define weights associated with data samples, reflective of their utility for a given task, e.g., classification. Once this weight network is learned, it is used to refine the synthetic data generation, such that it yields high-weight data. Emphasis is placed on diffusion models. Additionally, the noise generation and text-prompt embeddings are also optimized, again leveraging the weight network.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses an important problem and provides an effective procedure\n2. The demonstration figures are helpful in giving intuition (Fig 1) and in providing a concise summary of the approach (Fig 2)\n\nWeaknesses:\n\n1. Aspects of the paper are not very clearly written (Fig 2 was important for understanding the idea, as some of the text description could have been more clearly written)\n2. It is said that an advantage of this approach is that it doesn't need classifier-based methods to evaluate the quality of the generated data. However, the method used to learn the weight network relies heavily on training aligned with a downstream task, e.g., classification. There is a need for sufficient training and validation data to make this model (weight network) effective. While it is true that the later stages of the setup do not need a classifier, the entire approach rests on the quality of the weight network (stage 1), and a lot of data may be needed to learn that well.\n\nQUESTIONS:\n1. I found the discussion in Sec 3.1 not very clear, particularly Eqs (2) and (3). In (3), you are optimizing \\phi, it seems, parameters associated with the weight network. It seems therefore that you are optimizing \\omega_i. These weights impact the learning of \\theta in (2). Please provide some clarification of what you are saying mathematically here, as it is not very clear. I was helped by Fig 2, which at least was able to give me the idea of what you are trying to say.\n\n2. Typos:\n\na) Line 114: \"preferences preferences\"\nb) Line 119: \"during training process\"; missing a \"the\"\nc) Line 146: \"MLPO\"; I think you mean MLCO\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a data augmentation technique that uses synthetic samples from a diffusion model. The authors introduce a task dependent objective for sample generation. First they formulate a bilevel optimization problem to train a simple MLP that assigns weights to samples in the cross entropy loss, with the MLP computing a utility gain for each sample. Next they leverage this utility network to generate negative and positive samples for direct preference alignment. They also propose model level and instance level tuning of class specific prompt embeddings and initial noise for each generated sample based on the utility MLP. Experimental results show improved performance when a target model is trained on synthetic data alone and on a mix of real and synthetic data using the proposed framework.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n• The paper is well structured; topics flow logically.  \n• Formatting is clean; figures and tables are of high quality.  \n• The proposed method is novel and improves over prior work.  \n• Ablations and experimental design follow a logical sequence and address readers’ questions as they arise.  \n\n**Weaknesses**\n\n1. The implementation details require further clarification on several points of the method:\n\n   a. How is the bilevel optimization solved? Obtaining $\\theta^*(\\phi)$ requires gradient unrolling to backpropagate through $\\phi$, which can demand substantial memory.  \n\n   b. How is the prompt optimized during the denoising stage? Diffusion models involve multiple denoising steps and backpropagating through prompt parameters may incur high computational cost.  \n\n2. The authors claim that UtilGen enhances diversity but the connection between the utility MLP and diversity promotion is not clear. It would be useful to report a diversity metric such as pairwise distance in feature space (or any other) to quantify the diversity of synthesized samples.\n\nQUESTIONS:\nI also have the following questions and requests:\n\n1. I am curious about the performance gap of UtilGen compared with other baselines under different synthesis budgets. How does the gap change as more data is augmented?\n\n2. Why not first perform textual inversion to adapt the diffusion model to each class and then use the synthesized samples via the textual inversion (concatenated to the real data) to train the utility MLP in the first step? This would reduce distribution shift for the MLP when later inferred for diffusion alignment and DPO labeling.\n\n3. I would appreciate if the authors could explain how sample weights relate to the model’s cross entropy loss. Is there a trend between sample weights and the uncertainty of the $ \\theta $ model, and if so how does this trend evolve across different optimization stage of $\\theta$?\n\n4. Following on the previous point, although it may be outside the current method’s scope and that is absolutely fine if it is not addressed because of time constraints, it would be interesting to evaluate performance when the utility MLP is trained iteratively:\n\n   1. First augment the dataset by a factor of 1×.\n\n   2. Fine tune the MLP to produce updated sample weights after training on this augmented set.\n\n   3. Synthesize new samples using the improved MLP to achieve an 2× augmentation factor and so on to 3x and beyond...\n\n   This experiment would reveal whether adapting the MLP for weighting at different training stages yields additional gains.\n\n\nI thank the authors for the good work and I would be willing to raise my score upon addressing the questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a utility-centric data augmentation framework aimed at improving the task-specific effectiveness of synthetic data used in computer vision models. While existing data augmentation methods using generative models primarily focus on visual fidelity and diversity, UTILGEN shifts the paradigm to focus on task-specific utility—how useful synthetic data is for enhancing downstream task performance.\n\nTo achieve this, UTILGEN employs a meta-learned weight allocation network that efficiently evaluates the task-specific utility of each synthetic data sample without requiring full training cycles. These utility scores then guide a dual-level optimization process: 1. Model-Level Optimization: Adapts the generative model itself to the downstream task using methods like Direct Preference Optimization (DPO). \n2. Instance-Level Optimization: Fine-tunes generation policies (e.g., prompt embeddings, initial noise) to produce more useful data samples.\n\nTheir framework is evaluated across **eight** benchmark datasets and shows an average improvement of 3.49% in accuracy over state-of-the-art augmentation methods. Notably, a ResNet-50 trained solely on UTILGEN-generated synthetic data outperforms its real-data-trained counterpart on multiple tasks. The framework is also effective across different architectures (e.g., ResNeXt, WideResNet, MobileNet), demonstrating versatility and robustness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1.  Paper is very well written and easy to understand.\n\n2. UTILGEN introduces a novel shift in data augmentation—from optimizing visual fidelity and diversity to maximizing task-specific utility. This addresses a major gap in generative augmentation by aligning synthetic data generation with actual model learning needs.\n\n3. The \"meta-learned weight allocation network\" estimates task-specific sample utility without full retraining. Combined with influence function analysis and PCA-based visualization, the framework demonstrates that it generates data with stronger positive impact and clearer alignment within high-utility region. Such analysis is missing in prior methods.\n\n4. UTILGEN’s synthetic data scales effectively—performance continues to improve from 1× to 5× synthetic data, with increasing positive influence on downstream models. \n\n5. The experiments with model variants is extensive, making the results robust and effective across different multiple architectures (ResNet, ResNeXt, WideResNet, MobileNetV2).\n\n**Weaknesses**:\n\n1. The meta-learned weight network provides an efficient way to estimate task-specific utility, but its internal logic remains unclear. While data influence and distribution analyses validate its effectiveness, understanding why certain samples are rated as high-utility is not directly interpretable to user? More insights on this will aid the method's effectiveness and user's understanding.\n\n2. While UTILGEN introduces a task-aware, feedback-driven approach to data generation, it does not compare against DisCL [1], a method that also incorporates task-adaptive feedback—albeit through a curriculum learning framework. Both methods share the goal of improving downstream performance by guiding data synthesis based on task-specific signals. However, DisCL is absent from both the related work discussion and experimental baselines. Including such a comparison is important to properly position UTILGEN within the broader landscape of task-adaptive data generation methods, and to validate its generality and superiority over other task-aware strategies. \n\n3. Although UTILGEN avoids full retraining cycles via its meta-learned utility predictor, the overall framework—including dual-level optimization (model- and instance-level) and task feedback loops—remains computationally intensive. The cost-benefit tradeoff compared to collecting more *labeled real data* **isn’t fully analyzed** in the paper. \n\n4. Their results shows that synthetic data alone can match or surpass real-data-trained models on multiple benchmarks.\nHowever, This seems to differ from findings in prior work such as DataDream [2], where synthetic data is shown to be most effective when mixed with real data. Could the authors elaborate on what aspects of UTILGEN (e.g., the utility-guided generation process) you believe contribute most to this breakthrough in synthetic-only training performance?\n\n5. *Limited Reusability of Synthetic Data*: The framework still requires fine-tuning for each downstream task, including model-level generation optimization and instance-level controls. This limits out-of-the-box reusability of synthetic data across tasks. How would authors address or acknowledge this limitation ? Also Can this framework be extended to few shot setting ?\n\n**References**\n\n[1] Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion. \n\n[2] DataDream: Few-shot Guided Dataset Generation\n\nQUESTIONS:\nThe 1-5 questions asked in the Weaknesses section, would certainly affect the evaluation scores.\n\nLIMITATIONS:\nThe missing limitation related to *Limited Reusability of Synthetic Data* is discussed under the Weakness section above. Apart from this, there are no other major weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}