UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation
Authors: Jiyu Guo
Shuo Yang
Yiming Huang
Yancheng Long
Xiaobo Xia
Xiu Su
Bo Zhao
Zeke Xie
Liqiang Nie
Venue: NeurIPS 2025 poster ()
Keywords: Data Augmentation
Diffusion Models
Training Data Valuation
Few-Shot Learning
Primary area: applications
TL;DR: We propose UtilGen, a novel utility-centric data augmentation framework which adaptively optimizes data generation based on downstream task feedback to produce high-utility training data.

Abstract: Data augmentation using generative models has emerged as a powerful paradigm for enhancing performance in computer vision tasks. However, most existing augmentation approaches primarily focus on optimizing intrinsic data attributes -- such as fidelity and diversity -- to generate visually high-quality synthetic data, while often neglecting task-specific requirements. Yet, it is essential for data generators to account for the needs of downstream tasks, as training data requirements can vary significantly across different tasks and network architectures. 
To address these limitations, we propose UtilGen, a novel utility-centric data augmentation framework that adaptively optimizes the data generation process to produce task-specific, high-utility training data via downstream task feedback. Specifically, we first introduce a weight allocation network to evaluate the task-specific utility of each synthetic sample. Guided by these evaluations, UtilGen iteratively refines the data generation process using a dual-level optimization strategy to maximize the synthetic data utility: (1) model-level optimization tailors the generative model to the downstream task, and (2) instance-level optimization adjusts generation policies -- such as prompt embeddings and initial noise -- at each generation round. 
Extensive experiments on eight benchmark datasets of varying complexity and granularity demonstrate that UtilGen consistently achieves superior performance, with an average accuracy improvement of 3.87\% over previous SOTA. Further analysis of data influence and distribution reveals that UtilGen produces more impactful and task-relevant synthetic data, validating the effectiveness of the paradigm shift from visual characteristics-centric to task utility-centric data augmentation.

UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation
Jiyu Guo
Shuo Yang!" Yiming Huang' Yancheng Long'
Xiaobo 2,a
Xiu Su Bo Zhao' Zeke Xie Liqiang Nie*
Harbin Institute of Technology, Shenzhen "National University of Singapore
3 MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China
4 Central South University Shanghai Jiao Tong University
Hong Kong University of Science and Technology (Guangzhou) { 2201101260s
whittt.... www.hhit......... shuoyang@hit. edu. cn
Abstract
Data augmentation using generative models has emerged as powerful paradigm for enhancing performance in computer vision tasks. However, most existing augmentation approaches primarily focus on optimizing intrinsic data attributes such as fidelity and diversity - to generate visually high-quality synthetic data, while often neglecting task-specific requirements. Yet, it is essential for data generators to account for the needs of downstream tasks, as training data requirements can vary significantly across different tasks and network architectures. To address these limitations, we propose UTILGEN, a novel utility-centric data augmentation framework that adaptively optimizes the data generation process to produce task- specific, high-utility training data via downstream task feedback. Specifically, we first introduce weight allocation network to evaluate the task-specific utility of each synthetic sample. Guided by these evaluations, UTILGEN iteratively refines the data generation process using a dual-level optimization strategy to maximize the synthetic data utility: (1) model-level optimization tailors the generative model to the downstream task, and (2) instance-level optimization adjusts generation policies such as prompt embeddings and initial noise at each generation round. Extensive experiments on eight benchmark datasets varying complexity and granularity demonstrate that UTILGEN consistently achieves superior performance, with an average accuracy improvement of 3.87% over previous SOTA. Further analysis of data influence and distribution reveals that UTILGEN produces more impactful and task-relevant synthetic data, validating the effectiveness of the paradigm shift from visual characteristics-cente to task udility-centric data augmentation.
Introduction
Recent advances in generative models, particularly diffusion models 1 5 6 7. have significantly advanced data augmentation by enabling the creation of photorealistie images. Such samples, and empirical
text-to-image systems are capable of generating diverse and evidence has shown their potential to enhance downstream model performance (8).
Current generative data augmentation approaches can be categorized into two main paradigms: fidelity preservation and diversity enhancement. The former employs techniques such as A-R--baeed fine-tuning (9) to align synthetic data with real- world distributions [10 I1, while the latter employs
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

High Uality Data of High Utility Dfta
Data
Data
Persian Cat
Auctic Fox
Task I: Binary Classification of Persian Cats vs. Arctic Roxes
Task 2: Binory Classification of Persian Cats vs. Lions
Lion
Figure 1: Comparison of high-utility samples within the same category (Persian cats) across two different tasks. White Persian cats (left) are more useful in Task I, while golden ones (right) are more beneficial in Task 2, highlighting the diverse data requirements in different downstream tasks.
Although effective in generating visually high-quality data, these methods solely optimize the intrinsic data attributes (c.g.. fidelity and diversity), often struggling to directly optimize the task-specific utility of synthetic data. In practice, different tasks and model architectures may require distinct training data distributions for optimal performance [14), exemplified in Figure I Despite this, most existing methods lack mechanisms to adapt data generation process based on the needs of specific downstream tasks. This limitation motivates our investigation into utilityy celtttc- data augmentation, in which synthetic data is explicitly optimized to enhance task performance rather than merely meet visual standards.
varied prompts or feature perturbations to enhance data diversity 12
To go beyond the above limitation, effective mechanism is needed to assess the task-specifi utility of synthetic data, thereby providing explicit optimization signals guide the data augmentation process. However, evaluating utility through full training and testing cycles is computationally prohibitive. Therefore, the core challenges in developing utility-cettric data augmentation are: (I) How to efficiently evaluate the task-specip utility of synthetic data without exhaustive training? and (2) How to systematically improve the task-specific utility of synthetic data?
In this work, we propose UTILGEN, novel utility-centric data augmentation framework, which can adaptively optimize the data generation process to produce task sesk-spcifiic, high-utility training data based on downstream task feedback. Specifically, we introduce quantifies the task-specific utility of synthetic data through meta-learned weight allocation network [15 16 17]. The network is optimized to minimize the classifier's validation loss by adaptively weighting the losses of training samples via estimation of their utility. The trained valuation network serves an efficient utility predictor, enabling assessment of task task-specific utility for newly generated samples without the need for costly retraining cycles. Guided by the utility signals, we employ an integrated dual-level optimization strategy: (1) Model-Level Generation Capability Optimization that tailors the data generator to downstream tasks through Direct Preference Optimization (DPO), and (2) Instance- Level Generation Policy Optimization that optimizes the generation policies (i.e., prompt embedding and initial noise) to maximize the task-specific utility of synthetic data. Compared to previous advanced data augmentation methods which focus optimizing intrinsic data characteristics, our proposed method achieves an average accuracy improvement of 3.87% across eight benchmark datasets. To the best of knowledge, this is the first generative augmentation method where counterpart on
Datu Valuation, which
ResNet-50 [8] trained solely on 3x synthetic data surpasses its several benchmarks. Before delving into the details, we summarize our contributions as follows:
Motivated by the observation that training data requirements differ across tasks and network architectures, introduce novel paradigm shift in data augmentation. Instead of focusing solely on optimizing intrinsic data attributes, we emphasize enhancing the task-specific utility of synthetic data. This utility-centric approach adaptively optimizes the generation process according to downstream task needs, enabling more targeted and effective data augmentation.
To efficiently evaluate the task-specific utility of synthetic data, introduce meta-learned weight allocation network that measures the utility of synthetic data without requiring costly retraining. These utility signals drive a dual-level optimization framework that enhances both the model generation capability and the generation policies, resulting in high-utility synthetic data tailored to downstream tasks.
Our method achieves state-of-the-ar performance with an average improvement of 3.87% in accuracy across eight benchmarks, while also demonstrating exceptional versatility by delivering

---

consistent performance gains across diverse architectures (e.g., [es WideResNet (2), and MobileNet(II). Through training trajectory analysis, we validate the rationality of task utility measurement based on the weight network. Furthermore, analyses of data influence and distribution reveals that GEN.GEN generates data with higher task relevance and stronger positive impact on model performance.
2
Related Work
2.1 Training Data Valuation
Understanding the role of training data in model performance is crucial for data-efficient learning
[2 23
25 26] and optimizing model behavior [27 28 29
30
31].
To better understand
how training data affects model behavior, many studies have aimed to quantitatively assess the
influence of individual examples on model performance 132 35
R
35
36
37
38 39]. Existing
approaches for evaluating data value can be classified into two categories: (1) Retraining- Setaaining-basd
approaches, such as Data Shapley [33 34 40 41] and C-score [35,, which quantify data influence through expensive model retraining across different training subsets. For the utility evaluation of large-scale synthetic datasets, these methods become computationally prohibitive due to their inherent complexity. (2) Gradient-based methods B2
that estimate data influence by analyzing gradient interactions between training and test points, either through static snapshot analysis or dynamic trajectory examination. Although these approaches avoid model retraining, they still incur significant computational overhead, particularly when performing complex operations such as Hessian matrix inversion 445 To evaluate the utility of synthetic data, our method employs weight allocation network [5 16 17 to efficiently assess data utility, avoiding the costly retraining or complex computations required by earlier approaches.
42
43
44
2.2 Training Data Augmentation
The availability of high-quality training data has been fundamental to the success of deep learning, enabling models to capture complex patterns, learn meaningful representations, and generate accurate predictions [46] 47 48| 49 (50). The methodology of training data augmentation has progressed from
traditional techniques to advanced generative approaches [51 I0
I
52
[2]
53 544. Traditional
methods, such as mixup [55] 56). erasing [57] 58) and cropping [59 commonly rely on predefined
transformations to augment dataset diversity. However, they are inherently limited to local pixel- level modifications. While Generative Adversarial Networks (GANs) 600| enabled synthetic image generation, they often face challenges in maintaining semantic consistency and distribution alignment [61 62). Recent advances in diffusion models, such as Stable Diffusion 4 and GLIDE (63). have demonstrated superior capabilities in generating synthetic data. To enhance diversity, methods such as GIF [12, ALIA 152, and DISEF [33 employ varied prompts or feature perturbations in the latent space. Meanwhile, techniques like RealFake [T]. DistDiff (64), and DataDream [I] focus on improving image fidelity by aligning synthetic data distributions with the target domain. Nevertheless, these methods primarily address intra-class distribution alignment without evaluating which types of data better support downstream tasks. In contrast, our approach adaptively tailors the generation process to downstream tasks, producing high-utility data specifically optimized for target applications.
3 Method
In this section, the framework of our utility-centric generative data augmentation system is presented,
as illustrated in Fig. []] Specifically, the proposed approach comprises three key components: (1) -oiietOreet Data Valuation (Sec. B.7. which quantitatively assesses the utility of synthetic data for downstream tasks; (2) Model- Generation Capability Optimization (Sec. 3.2), which tailors the generative model to align with the training data preferences of the downstream task via DPO; and (3) Instance-Leve Generation Policy Optimization (Sec. 3.3 which adapts generation policy to maximize the spsk-specific utility of the synthetic data.
3

---

Slage
Task-Driven Data Valuation via Weight-Net
Stage Model-Level Generation Capability Optimization
night-Met
Welallt
Direct Preference Oplimization (DPO)
Training Data
Update WeightNet
Diffusion Modaa
WaR
generaat
Val Data
Stop
Step Update Classifer
Fvalusat
Synthetic Data
Utity Ranking
Preference Data
Learnable pomptembeddinng
Maximieig Speci- Utity Scores
Text Inversion
Level
Generation
Real Data
Diffusion Model
WeightNet
Policy
Optimization
O.
Denoise
Invers
Synthetic Data
noise
Gllden noise
Figure 2: The GTILGEN framework for feedback-drivee data augmentation, comprising three key stages: (1) Task-Oriented Data Valuation (Sec. B.I); (2) Model-Level Generation Capability
Optimization (Sec. 3.2 (3) Instance-Level Generation Policy Oplimization (Sec. 3.3
3.1 Task-Oriented Data Valuation (TODV)
is noted that individual training instances exhibit heterogeneous influences on model performance during the training process [32]. with certain data samples potentially introducing negative influences. This observation has motivated a line of research to mitigate model overfitting to data samples with negative influences, such as training sample re-weighting strategies implemented through learnable weight networks [15] 16 [7]. Moreover, it is recognized that these learned weights can be implicitly interpreted as indicators of each sample's utility for downstream tasks [13] Drawing inspiration from this insight, we propose Kask-Oriented Data Valuation, which employs weight network trained via meta-learning to quantitatively assess the utility of synthetic data for downstream tasks.
Specially, given a classifier fo. the weight w; assigned to each sample x; is derived through a loss-based process follows: wi = We (Ir) 91),
where W. denotes an MLP network with a single hidden layer. This network is trained to predict normalized weights within the range [0, 1]. where higher values reflect samples with greater utility.
To develop such weight network Wo capable of measuring task-specific data utility, we adopt bi-level optimization strategy comprising two iterative steps:
Classifier training: To enhance the generalization capability of the weight network and mitigate distribution shift in subsequent generation optimization stages, we first perform textual inversion [65] to learn class- saesispe identifiers [7:] from small set of real images per class. Using these learned identifiers, we generate synthetic data Dg with conditioned prompts G; of the form "a photo of I The synthetic data is then combined with the real training data to form merged dataset Dmerge = (D, UDg,, where DD denotes the real data. The classifier parameters 0 are optimized using a weighted loss computed over D Dmeege = ((x; v.)
N
0 (Φ) arg min
(cC([((; N
Weight network training: Given the classifier's updated parameters, the weight network parameters s are trained to minimize the loss on the validation set D, {(xz. M
M Σ
1
(0) = arg min
E 1=1
M
This bi-level optimization framework establishes dynamic feedback loop between data valuation and model training. By quantifying data utility through learned weights, the trained weight network

---

is subsequently used to guide the optimization of the data generation process. The full procedure for TODV is described in Algorithm
Algorithm 1 Task-Oriented Data Valuation (TODV)
Input: Training data D, U U = ( v.)
validation data D. = {(ry, Us ) Required: Classifier parameters 8: weight network parameters Φ; max itcration T..
I: Initialize (
(0)) randomly
2: for t= 0 to T do
Sample batch {(r, y:)] from D, UDg
4: Sample batch {(z).8))) from D.
5: Predict weights {w/())]
Hi=1 for the batch {(x,, u)) H=) that reflect their task-specific utility (Eq. (f)
6: Update g(t++2) using the loss weighted by {w] Mi (Eg. 2
7: Update (((+1) (Eq.
8: end for
Output: Optimized weight network parameter o²
3.2 Model-Level Generation Capability Optimization (MLCO)
Although standard diffusion models demonstrate remarkable capabilities in generating visually high-quality images, their outputs often fail meet the specific data requirements of downstream applications. To address this misalignment, we propose an iterative DPO framework that adapts the generative model to downstream task-specific data preferences.
Each optimization cycle begins by prompting the diffusion model with prompts of the form "a photo of [T,]", where class-specific identifiers [7,] are obtained via textual inversion [65] generate a synthetic dataset Day = {<.,)) M 1- The pre- pre-trained weight network Wo subsequently evaluates each sample's 's utility via the weight score w; W& (Cff(; 0), y.)). where [xf(x; 0), y1) is the loss of the classifier fo on sample x; with its label Based on these scores, high-utility samples x,""
and low-utility samples ot are paired to construct preference dataset:
D Ppreference = ((,, 2!) ( > ()
(4)
We then use the preference dataset D Prreference to fine-tune the diffuusonn model's U-Net & using DPO, with the optimization objective formulated according to the Diffusion DPO [66].
(V)) BB (01 [log (- (Ae) (AC., AC)))
(5)
AL ||e (x 1)l1 1|c" (xr 1)1
(6)
AC1 ||e² co (2/, 1) -1c 1)12.
Here, Gú and Ger denote the noise predictions from the trainable and reference U-Nets, respectively. The forward diffusion process g adds noise to xp at timestep where t is sampled from (0, T). ,3 balances preference alignment and KL regularization, while o(-) is the sigmoid activation in the loss. X, is the signal-to-noise ratio 66 and (\e) is weighting function III.
Through iterative DPO munittnnng, we progressively adapt the diffusion model's generative capability according to the downstream task's alignment between the model's 's output distribution and the target application requirements.
'ss preferences for the training data. This process enables better
3.3 Instance-Level Generation Policy Optimization (ILPO)
While MLPO tailors the generative model to the downstream task coarse level, ILPO performs fine-grained refinement of the generation policy by jointly optimizing the prompt embeddings and the initial noise. The overall optimization objective is formulated as:
(p G) = arg max E (C(( v))]
p.cr
5

---

Radl
50 Rcad Syslada
20
(8) Data Distribution Visualization
Utility-Aware Weight Distributions
Figure 3: (a) Feature space visualization the Flower dataset 68 shows that synthetic data achieves closer alignment with the real data distribution compared to vanilla Stable Diffusion. (b) -tieety-aware weight distributions for synthetic and real data the Flower dataset (68), showing sample utility scores for downstream tasks.
Algorithm 2 Instance-Level Generation Policy Optimization (ILPO)
Input: K classes: {c1, ... CK ): number of images generate for each class: (Ni..... Nic where N2. is the number samples generate class Ck. Required: Diffusion model g(-,)) fine-tuned by MLCO; Textual Inversion technology T((:;)) using few-shot real images per class D [#i, 1.1,,..;;; x for each cæ:
I: for each class & {1..... K} do
2: Prrompt Embedding Optimization
Initialize Pc: <- TT((s, Dx))
4: PE < Pk (Eq.8)
>> Optimize to maximize data utility
5: // Noise Optimization
6: for 1 to N₂ do
Sample noise er N/O,
ET Er by (Eq.)
Inject semantic information representing high-utility data b Generate high-utility data
9:
T:
- Pl T
10:
Daymh < D synh U [+4
11: end for
end for
Output: High-utility synthetic dataset D symh
where p denotes the prompt embedding, 6T represents the initial noise vector, g()) is the diffusion model, f(; 0) is the downstream classifier, is the ground-truth label, and Z(-) is the classification loss. The optimization process consists of synergistic components:
Prompt embedding optimization: Building upon the class-specific identifiers [I,] learned through textual inversion (65], we optimize the prompt embeddings p: = E("a photo of [7,]") using gradient- based optimization, where E(-) denotes the text encoder. This process aims to maximize the utility score predicted by the pre-trained weight network. To preserve semantic alignment during optimization, a CL.P--based regularization term LCLPP = cos ((x;), ), e;) applied, where x; the generated image, F.(.) is the CLIP image encoder and e; avg( FI (mr))) is the mean embedding of a set of target-class real images {æ.}h The complete prompt optimization objective is:
p arg max
(8)
where À controls the regularization strength. This joint optimization enhances sample utility while preserving semantic coherence with the original class concept.
The quality of synthetic images is influenced by both the text prompt and the random Gaussian noise. Yet, since each image generation requires independently sampled noise, directly optimizing noise vectors via gradient ascent to improve utility is computationally prohibitive. show that the discrepancy between denoising and inversion Classifier- Free Guidance (CFG) scales can be leveraged to implicitly inject prompt semantic information into the initial noise. Leveraging this observation, we adapt the methodology to optimize the initial noise, enabling it to incorporate semantic information of high-utilig data. Formally, the noise optimization
Noise optimization:
Recent studies 669 70

---

Table 1: Classification performance across eight datasets using ResNet-50 [88] as the backbone classifier. UTILGEN and DataDream [T] generate synthetic data guided by 16-shot real images per class, while other methods use the full real dataset as guidance for generation. The top block
shows results using synthetic data only, while the hottom includes joint training (synthetic full real dataset). Each method produces synthetic data at 5x the scale of the original real dataset.
Method
Real
Syn
IN-IK-S
IN-100-S
Cal1o1
DTD
CUB
PETs
Food-S
Flowers
Avg
Training on Synthetic Data Only
SD v2
24.35
27.96
14.74
7.92
23.43
26.05
24.02
31.41
22.49
GIF 12
28.95
31.94
20.39
13.19
27.54
29.73
25.94
56.12
29.23
GAP 72
25.84
30.94
18.70
11.01
29.49
27.46
25.31
53.66
27.80
UtilGen
A previowy SOTA
+3.37
+5.46
+5.70
+0.28
+7,94
+2.48
+3.46
+2.28
+3.87
Joint Training with Real Datn
Real Dataset
36.34
38.58
43.55
16.32
21.03
28.78
19.88
73.
34.73
v2 1
43.26
49.86
59.35
28.82
43,52
52.35
40.32
79.58
49.63
GIF 12
49.85
$4.12
67.60
33.45
43.80
58.21
41.39
84.47
54.11
GAP 72
46.58
53.14
66.97
32.87
47.46
57.86
43.77
84.84
19
DataDream
57.68
73.38
34.84
53.43
UtilGen
A real dalaset
+8..22
+22.96
+32.07
+19.74
+36.50
+35.86
+32.84
+20.28
+27.31
2 owe previous SOTA
+2.40
+3.86
+2.24
+1.22
+4,10
+3.81
+5.28
+4,02
+3.37
process is defined as:
DDIM-IIversion., (DDIIi (Et, ()),
where wi and Wer are the CFG scales for the denoising process (DIM(.) and the inversion process IDIM-Inversion(() respectively. The condition > we enables the implicit injection of semantic information into the initial noise. The entire process of ILPO is outlined in Algorithm 2
The synergistic integration of MICO (Sec (.2c and ILPO (Sec..3 enables UTIL.GNN to synthesize data that elosely aligns with real data feature distributions, as visualized Fig This approach simultaneously achieves higher utility scores (Fig B()),, demonstrating enhanced task relevance and superior data utility downstream applications.
4 Experiments
4.1 Experimental Setup
Benchmarks. We evaluate the effectiveness of UTILGEN across eight datasets spanning three classi-
fication tasks: coarse-graine classification (ImageNet- 1k-Subset (73), ImageNet- 100-Subset 73
and Caltech 101
(74)), fine-grained classification (Oxford Pets [75] Food-S [76), Flowers 102 [68].
and CUB N-B-200-20011 (77), and texture classification (DTD 78 Specifically, ImageNet- 1k-Neett---Sueer
and Food-S are subsets of ImageNet- KK and Food101 (76). respectively, each containing 100 ran-
domly selected images per class. ImageNet- 100-e-ttt- is constructed by randomly sampling 100
animal-related classes from the original ImageNet-IK 773 with 100 randomly selected images per
class. Further benchmark details are provided in Appendix
Baselines. To compare our utility-centric approach with existing methods that focus on optimizing intrinsic data characteristics, we select GIF (12] and DataDream [[]] representative baselines. Specifically, GIF [12) enhances diversity by applying feature perturbations, while DataDream [I1] improves fidelity through domain alignment using LoRA fine-tuning the diffusion model. Addition- ally. we include GAP [12], which uses feedback from the downstream model to generate adversarial prompts that maximize the model's loss on the generated images. This enables direct comparison between its loss-based feedback strategy and the utility-based feedback mechanism employed by UTH.GENN For fair comparison, we adopt Stable Diffusion v2.1 4 (SD v2.1) as the backbone generator across all baseline methods. The implementation details of UTILGEN are in Appendix

---

Drasei Duzzect 110--30--30 Cup
Closs Browel
Real
v2.1
GIF
GAP
DutaDream
UhiGen
Utilien
Figure 4: Comparison of synthetic images generated by SD v2.1, GIF [12]. GAP 72 DataDream and UTILGEN.
Table 2: Performance comparison under different syn- thesis budgets ImageNet- 100. UTILGEN scales more effectively with increasing synthetic data ratios.
Table 3: Comparison of intra-class diver- sity of synthetic data ImageNet-100) 100. Higher values indicate greater diversity.
Budget SD
GIF
12
GAP 72
DataDream
UTILGEN
Method
Mean Intra Class Diversity
12.18
15.02
0.5815
20.20
21.16
DataDream
0.5238
3x
27.96
31.94
30.94
35.48
40.94
UTILGEN
0.6054
4.2
Evaluation Results
Results on solely synthetic data. As shown in Table I (top), UTILGEN achieves the highest average accuracy of 17% when trained solely on synthetic data, outperforming the previous best method DataDream (33.30%) by a notable margin of 43.87%. It demonstratte strong performance across both coarse-grained (e.g.. 40.94% on IN-100-S (73) and fine-grained tasks (e.g.. 67.443.. 67.43% on Flowers (68). indicating excellent generalization despite training solely on synthetic data.
Results on real + synthetic Data. the joint training setting (bottom of Table I. UTILGEN maintains its lead with average accuracy of 62.04%, surpassing DataDream (58.67 67% by +3.54%. achieves particularly strong gains on fine-graine datasets (e.g., 93.62% on Flowers (68). while also delivering consistent improvements effectively complement real data across different task granularities, providing high-utility synthetic samples that enhance model performance and robustness.
coarse grained tasks. These results suggest that UTILGEN
Synthetic data scaling effects. UTH UTHLGGEN GEN exhibits strong scalability and data augmentation efficiency. As shown in Figure scaling synthetic data from xx 5x the original set consistently improves ResNet-50 [8] performance across benchmarks under both synthetic-only and joint training settings. When trained with 3x synthetic data alone, models outperform their real-data-trained counterparts on three datasets; at 5x scaling, this advantage extends to four datasets. To further examine how performance scales with the synthesis budget, we compare UTILGEN with four representative baselines under budgets. As shown Table [] UTILGEN consistently achieves the best results across all budgets, and the performance gap widens the synthesis ratio increases, demonstrating superior scalability under large-scale generation. Additional details on augmentation efficiency and computational cost provided in Appendix D
Diversity analysis of synthetic data. We evaluate the diversity of synthetic samples by computing the mean intra-class cosine distance of CLIP (ViT-L/14) features ImageNet-100) 100. Compared methods include vanilla Stable Diffusion v2.1, DataDream [ and UTILGEN. Higher values indicate greater intra-class diversity, which typically benefits generalization. As shown in Table 3 UTILGEN achieves the highest intra-class diversity, indicating that its utility- -iiidd optimization preserves sample variety and avoids mode collapse.
Reusability of synthetic data across tasks. We further analyze the reusability of UTILGENS generated data across different downstream models the ImageNet-100 dataset. Even when the weight network is trained using ResNet-50, the resulting synthetic data generalizes effectively to other models such WideResNet and CLIP. As shown in Table the performance gains remain

---

166.5
Pae
IN 100-
Dest Ducoumocy (h
Treting Acaemyy
Fool
Caltt
Mibad Wabbos Ro-ohhee
Wabez Re-Riehing
Toiking Epoch
Tusking Ejoch
(6) Syrtbetia Only
06 Systheik Red Data
(4) ToTiiin Accunsc Comparion
(3) Test Tect Compertons
Figure 5: Synthetic Data Scaling Effects across different training data regimes. (a) Models trained exclusively on synthetic data (Synthetic Data Only). (b) Models trained on combined
Figure 6: Training and test accuracy compari- son with and without the weight network using ResNet-50 [88 on the dataset (original Flowers
SD v2.1 augmented data with 1 x expansion). (a) Training convergence; (b) Test accuracy.
synthetic and real data (Synthetic- + Real Data).
Table Reusability of synthetic data across down- Table Generalization performance across di- stream models on ImageNet-100
verse network architectures on ImageNet-100).
Classifier Downstream model train weight network using synthetic data
Method
Acc. (%)
Method
ResNeXi-50
WideResNet-50 MobileNetV2
WideResNet WideResNer
DataDream
31.76
ResNet-50
GAP
27.76
32.72
CLIP
DataDream
M
71.42
DataDream
31.24
31.76
35.48
ResNet-50
UTILGEEN
UTILGEN
37.62
37,82
40.59
consistent across architectures, indicating that high-utility samples identified by UTILGEN are not tied to specific model and can be reused for diverse learning objectives.
Generalization across different architectures. To assess the versatility of UTILGENN we evaluate its performance on three architectures: ResNeXt-50 [9]] WideResNet-50 20) and MobileNetV2 [21]. As shown T UTILGEN consistently achieves the highest accuracy, with 500 images generated class, confirming the effectiveness of our approach across various network architectures.
Cost-benefit analysis compared collecting more labeled data. Compared manual data collection and annotation, UTILGEN offers a highly cost-effective solution for dataset expansion. According the Masterpiece Group manually annotating 10,000 images (e.g., 100 images per class across 100 classes) typically takes about two weeks and costs approximately $800. In contrast, generating the same amount of data using UTIL requires only about 0.94 hours and $20 on V100 GPUs rented from Google Cloud On the ImageNet- Saae--00-Suuusss dataset, using 5x synthetic data produced by UTILGEN even surpasses the real-data baseline in accuracy, while requiring only 4.7 hours and about $ $100 in compute cost, as shown in Table
Ablation study. Table presents ablation study on the IN-100-S dataset using ResNet-50 trained solely on synthetic data (500 aasssper class). We analyze the effects of MLCO and ILPO (including prompt embedding and initial noise optimization). Each component brings improvements over the baseline, and combining all three achieves the best performance, surpassing the baseline by +12.98%, demonstrating the complementary strengths of model-level and instance-level optimizations.
4.3
Mechanism Analysis
Effect of sample re-weighting on classifier training. To assess the effect of the weight allocation network on classifier training, we compare the training trajectories of models with and without dynamic weighting. As shown in Figure (a)) the weighted model converges faster and achieves higher training accuracy in earlier epochs, suggesting it effectively assigns higher weights to high- utility samples during learning. Figure ((b)) presents the test accuracy curves, where accuracy rises more rapidly and achieves final improvement of 2.94% over the bascline. This gain is attributed to the network's ability to nown-weight low-utility or noisy samples, thereby mitigating their negative
sstpa:://mg-myyanaar htw.aammarr..om/mmotttttttiioiaa
https://cloud, google. www/ ppiccing

---

3D Cal Dverisms
5D V2.1 Doninest
Nerrosioee Inflcnee
Nermalaed befhurzs Scare (c) Data Distributinn Difference
(1) Positive Indlurma Rotio
(0) Ddta Inthurnee Distribation
Figure 7: Data influence comparison between UTH.GEEN and SD v2.1, computed using influence functions. (a) Proportion of samples with positive influence scores (influence > 0) across eight datasets. (b) Influence score distributions showing that UTILL GEN generates higher density of samples with stronger influence. (c) Data density difference plot highlighting that UTILGEN dominates in high- influence regions, while SD v2.1 contributes more to low-influence areas.
Table 6:
Comparison of the manual annotation paradigm and our synthetic data paradigm terms of cost, time, and performance.
Table 7: Ablation study on ImageNet-100.
MLCO Prompt Optimization
Noise Optimization
27.96
28.68
Method
Images
Time Required
Cost Acc. (%)
36.42
37.96
Manual Annotation
10,000
~22 weeks
$800
38.58
UTILGEN (1x)
10,000
~0994 h
$20
18.04
v
39.12
39.73
UTI.GENN (5x)
50,000
<-470 h
$100
40.94
40.94
impact. These results confirm that the pre-trained weight allocation network can reliably measure sample utility and serve as a signal guide synthetic data generation.
Data influence analysis. measure data influence by jointly training models on synthetic datasets generated by UTILGEN and the baseline SD 12.1, applying Influence Function [32]. Figure (la) reveals that UTILGEN consistently produces a higher proportion of positively influential samples (influence 0) across all benchmarks. Furthermore, the influence score distribution shown in Figure b) shifts noticeably to the right, indicating that UTILGEN generates more samples with stronger positive influence. Complementing this, Figure 7 highlights that UTILGENN achieves substantially higher data density within high-inflennce regions, while simultaneously reducing sample concentration in low-utility areas. Together, these results validate the effectiveness of method in synthesizing impactful data that better supports downstream model optimization.
5
Conclusion
In this study, we propose UTILGEN, a utility-centric data augmentation framework that shifts the focus from optimizing intrinsic visual properties enhancing task-specific utility. By incorporating downstream model feedback, UTILGEN adaptively adjusts the data generation process to produce high-utility data tailored for specific downstream tasks, thereby establishing feedback loop between data generation and model training. Experiments across eight benchmarks demonstrate consistent performance gains, and in certain cases surpassing the performance of models trained solely on real data. These results highlight the superiority of the utility- approach over prior methods focusing primarily on intrinsic visual quality. These findings underscore the potential of utility- centric generation and suggest that integrating task mask-specific utility alongside traditional visual quality considerations offers a more effective paradigm for future data augmentation research.
Acknowledgment
The authors would like to acknowledge the support from the following funding sources. Shuo Yang is supported by the National Natural Science Foundation of China Young Scientists Fund
10

---

(No. 62506096). Xiaobo Xia is partially supported by the MoE Key Laboratory of Brain- inspired Intelligent Perception and Cognition at the University of Science and Technology of China (Grant No. 2421002). Bo Zhao is supported by the National Natural Science Foundation of China (Grant No. 62306046).
References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840068511, 2020.
[2] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780-8794 2021.
[3] Zhaoqing Wang, Xiaobo Xia, Runnan Chen, Dongdong Yu, Changhu Wang, Mingming Gong, and Tongliang Liu. Lavin-dit: Large vision diffusion transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20060-20070, 2025.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[5] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.1255 12598, 2022.
arXiv preprint
[6] Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232 2024.
[7] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
[8] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.0757. 2022.
[9] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[10] Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Tomr and Bo Zhao. Real-fake: Effective training data synthesis through distribution matching. arXiv preprint arXiv:2310. 10402, 2023.
[11] Jae Myung Kim, Jessica Bader, Stephan Alaniz, Cordelia Schmid, and Zeynep Akata. Datadream: Few-shot guided dataset generation. In European Conference on Computer Vision, pages 252-268. Springer, 2024.
[12] Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small- scale datasets with guided imagination. Advances in neural information processing systems, 86:76558-7666 2023.
[13] Victor G Turrisi da Costa, Nicola Dall' Asen, Yiming Wang, Nicu Sebe, and Elisa Ricci. Diversified in-domain synthesis with efficient fine-tuning for few-shot classification. arXiv preprint
2023.
[14] Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang. Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centri artificial intelligence: A survey. ACM Computing Surveys, 57(5):1-42. 2025.
[15] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances neural information processing systems, 32, 2019.

---

[16] Jun Shu, Xiang Yuan, Deyu Meng, and Zongben Xu. Cmw-net: Learning class-aware sample weighting mapping for robust deep learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 15(10):115211111 1521-11539, 2023.
[17] Zizhao Zhang and Tomas Pfister. Learning fast sample re-weighting without reward data. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 725-734, 2021.
[18] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[20] Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146 2016.
Wide residual networks. arXiv preprint
[21] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 2018.
[22] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint 20Xx:20006.05999, 2020.
[23] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 6514-6523, 2023.
[24] Xiaofeng Cao, Weiyang Liu, and Ivor W Tsang. Data-efficient learning via minimizing hyperspherical energy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(11):13422-133377 2023.
[25] Shuo Yang, The Cao,, Sheng Guo, Ruiheng Zhang, Ping Luo, Shengping Zhang, and Liqiang Nie. Mind the boundary: Coreset selection via reconstructing the decision boundary. In Forty-first International Conference on Machine Learning. 2024.
[26] Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Bicro: Noisy correspondence rectification for multi multi-modality data via bi-directional cross-modal similarity consistency. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 19892,19892, 2023.
[27] Ruiheng Zhang, Biwen Yang. Lixin Xu, Yan Huang, Xiaofeng Xu, Qi Zhang, Zhizhuo Jiang, and Yu Liu. A benchmark and frequency compression method for infrared few-shot object detection. IEEE Transactions on Geoscience and Remote Sensing, 2025.
[28] Xiaofeng Cao, Yaming Guo, Heng Tao Shen, Ivor W Tsang, and James T Kwok. Mentored learning: Improving generalization and convergence of student learner. Journal of Machine Learning Research, 25(325):1 45, 2024.
[29] Rui Huang, Shitong Shao, Zikai Zhou, Pukun Zhao, Hangyu Guo, Tian Ye, Lichen Bai, Shuo Yang, and Zeke Xie. Diffusion dataset condensation: Training your diffusion model faster with less data. arXiv preprini arXiv:2507.05914 2025.
[30] Ruiheng Zhang, Lixin Xu, Zhengyu Yu, Ye Shi, Chengpo Mu, and Min Xu. Deep-irtarget: An automatic target detector in infrared imagery using dual-domain feature extraction and allocation. IEEE Transactions on Multimedia, 24:1735-1749, 2021.
[31] Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent bayes-label transition matrix using deep neural network. In International Conference on Machine Learning, pages 25302-25315 PMLR, 2022.
[32] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894, PMLR, 2017.
12

---

[33] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242-2251. PMLR, 2019.
[34] Yongchan Kwon, Manuel A Rivas, and James Zou. Efficient computation and analysis of distributional shapley values. In International Conference on Artificial Intelligence and Statistics, pages 793-801. PMLR, 2021.
[35] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.33000 2020..
[36] Ki Nohyun, Hoyong Choi, and Hye Won Chung. Data valuation without training of model. In The Eleventh International Conference on Learning Representations, 2022.
[37] Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. In International Conference on Machine Learning, pages 24150-24176. PMLR, 2022.
[38] Rui Zhang and Shihua Zhang. Rethinking influence functions of neural networks in the wve-pparameterized regime. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9082-9090, 2022.
[39] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint 220.::2205,00329, 2022.
[40] Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory, 28:547-565, 1999.
[41] Sebastian Bordt and Ulrike von Luxburg. From shapley values to generalized additive models and back. In International Conference Artificial Intelligence and Statistics, pages 709-745. PMLR, 2023.
[42] Khin Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018
[43] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 33:19920-19930, 2020.
[44] Yuanyuan Chen, Boyang Li. Han Yu, Pengcheng Wu, and Chunyan Miao. Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7081-7089, 2021.
[45] Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey, Machine Learning, 113(5):2351-244 2024.
[46] Xiangxin Zhu, Carl Vondrick, Charless C Fowlkes, and Deva Ramanan. Do we need more training data? International Journal of Computer Vision, 119(1)::6 92, 2016.
[47] Zihui Zhang, Weisheng Dai, Hongtao Wen, and Bo Yang. Logosp: Local global grouping of superpoints for unsupervised semantic segmentation of 3d point clouds. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1374-13884, 2025.
[48] Jiahang Liu, Yunpeng Qi, Jiazhao Zhang, Minghan Li. Shaoan Wang, Kui Wu, Hanjing Ye, Hong Zhang, Zhibo Chen, Fangwei Zhong, et al. Trackvla+++: Unleashing reasoning and memory capabilities in vla models for embodied visual tracking. arXiv preprint arXiv:2510.077 2025.
[49] Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on vision- language models. Advances in Neural Information Processing Systems, 37:3122-3156,0 2024.
13

---

[50] Ruiheng Zhang, Zhe Cao. Yan Huang, Shuo Yang, Lixin Xu, and Min Xu. Visible-infrared person re-identification with real-world label noise. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
[51] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing autoo-ddistriuutinn examples via augmenting content and style. arXiv preprint arXiv:2207.03162 162, 2022.
[52] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion-bar based augmentation. Advances in neural information processing systems, 36:79024-79033 2023.
[53] Qilin Liao, Shuo Yang. Bo Zhao, Ping Luo, and Hengshuang Zhao. Bood: Houndary-based out-of-distribution data generation. arXiv preprint wxXiv2558.003350 2025.
[54] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv:2101. 06395, 2021.
[55] Dan Hendrycks, Norman MU, Ekin Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. 19w.. 027811 2019.
arXiv preprint arXiv: arXiv:
[56] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? arXiv preprint arXiv:2010.04819, 2020.
[57] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[58] Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
(59] Gianluigi Ciocca, Claudio Cusano, Francesca Gasparini, and Raimondo Schettini. Self-adaptive image cropping for small displays. IEEE Transactions on Consumer Electronics, 1322 1627, 2007.
[60] Ian J Goodfellow, Jean Pouget-Abris Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[61] Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Synthetic data augmentation using gan for improved liver lesion classification. In 2018 IEEE 15th international symposium biomedical imaging (ISBI 2018), pages 289-293. IEEE, 2018.
[62] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[63] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with rext-guided diffusion models. arXiv preprint arxiv: 2021.
[64] Haowei Zhu, Ling Yang, Jun-Hai Yong, Hongzhi Yin, Jiawei Jiang, Meng Xiao, Wentao Zhang, and Bin Wang. Distribution-awe -awar data expansion with diffusion models. arXiv preprint urXiv:2403.06744
2024.
[65] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 0/6/8, 2022.
[66] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8228-8238 2024.
14

---

[67] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances neural information processing systems, 34:21696-21707, 2021.
[68] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
[69] Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. arXiv preprint arXiv:241..0950,, 2024,
[70] Lichen Bai, Shitong Shao, Zikai Zhou, Zipeng Qi, Zhiqiang Xu, Haoyi Xiong, and Zeke Xie. Zigzag diffusion sampling: Diffusion models can sell-improve via self-reflection. In The Thirteenth International Conference on Learning Representations, volume 2, 2024.
[71] Donghon Ahn, Jiwon Kang, Sanghyun Lee, Jaewon Min, Minjae Kim, Wooseok Jang, Hyoung- won Cho, Sayak Paul, SeonHwa Kim, Eunju Cha, et al. A noise is worth diffusion guidance. arXiv preprint M...i:24/2.03300 2024.
[72] Teresa Yeo, Andrei Atanov, Harold Benoit, Aleksandr Alekseev, Ruchira Ray, Pooya Esmacil Akhoondi, and Amir Zamir. Controlled training data generation with diffusion models, arXiv preprint arXiv:2403. 15309, 2024.
[73] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recogmition, pages 248-255. leee, 2009.
[74] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pullern recognition workshop, pages 178-178. IEEE, 2004.
[75] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[76] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101-mining discriminative components with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 62, 2014, proceedings, part VI 13, pages 461. Springer, 2014.
[77] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[78] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. Proceedings the IEEE conference on computer vision and pattern recognition, pages 3606-361 2014.
[79] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arX1v:2501.12948, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm that the abstract and introduction accurately reflects the paper's contributions, as it clearly outlines the proposed UTILGEN framework, its adaptive opti- mization approach, and experimental validation across diverse datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed Appendix C
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide comprehensive implementation details in Appendix B and dataset specifications in Appendix IF) including methodological implementations and dataset charac- teristics, to ensure full reproducibility of our work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our study uses exclusively publicly accessible datasets and includes the complete implementation source code in the supplementary materials (provided as a 7IP archive).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptiips.cc/publcc/uuddee Coape///oeeSSuieieeeee seriissssonpoiioio
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide comprehensive implementation details, including data partitioning. hyperparameter settings, and additional relevant information in Appendix and Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We opted not report error bars because the results demonstrated consistent stability across multiple runs, reducing the necessity for such reporting. ensure robustness, each experiment was conducted using three different random seeds with results averaged accordingly. Additionally, extensive evaluations across diverse datasets further strengthen the reliability of our findings.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information on the computational resources used in this work, including the types and numbers of GPUs, GPU memory usage during execution, and overall running time. For more details, please refer to Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: pettps://neuripeee wwtpoouurppsccpppulleeeeeeeeeeee hwtp///wbb..t...ccccctieee...ee.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: We have carefully examined the NeurlPS Code of Ethics and affirm that all aspects of this research comply with its guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
19

---

We provide discussion of both positive and negative societal implications in Appendix
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This research does not involve releasing models or datasets, thus the discussion of release safeguards is not applicable to our work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets (e.g., ImageNet (73) and foundation models (e.g., Stable Diffusion (4) used in this work are appropriately cited the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code documentation is provided the supplementary materials.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs were used for language polishing (grammar and spelling corrections).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hccpps//naarriss.cc//oofeeeereee for what should or should not be described.
22

---

A Additional Visualizations of Synthetic Data
In this section, we present further visualizations of synthetic data generated by the UTILGEN. As illustrated in Figure 8 we compare synthetic samples produced by different augmentation strategies. including GIF (prioritizing diversity in visual features) and DataDream (emphasizing fidelity in visual presents representative samples from UTILGEN, demonstrating semantically consistent, visually realistic generations that preserve disrimmicrii features while maintaining diversity.
features). Figure [
Dataset: CUB
Dataset:
Dataset: PETs
Dataset: ImageNet
Dataset: DTD
Dataset:
Flowers
Caltech101
Reai Image
DataDream
Focusing on optimizing data visual features: Fidelity
GIF
Focusing optimizing data visual features: Diversity
UtiGen
Focusing optimizing the taslespecific utility of the data
Figure 8: Comparison of synthetic samples generated by GIF, DataDream, and UTILGEN.
23

---

B More Implementation Details
B.1 Implementation Details of Weight Network
The weight network Wo is designed lightweight MLP with single hidden layer. It takes the per-sample classification loss E; = C(f((; 0), M/) as input, where f(; 0) is the downstream classifier, and predicts normalized sample weights wi [0, 1] via sigmoid activation. These weights used to re-weight the corresponding per-sample losses during classifier training, thereby prioritizing examples deemed to have higher utility by the meta-learned network. The network architecture is defined as:
(c(C)) = oWW, ReLU((14 + b1) + b2)
(10)
where 4 E R is the scalar loss value for the i-th sample, W1 E R 1x and W₂ E R000 are weight matrices, and b1, by are bias terms. The ReLU activation introduces non-linearity, and the final sigmoid ensures the output lies in j0, 1]. Albeit simple, this network is universal approximator for continuous functions and can fit wide range of weighting functions. The training settings for both the classifier and the weight network, including optimizers, learning rates, and batch sizes, are summarized in Table
Table 8: Training settings for the classifier and weight network.
Component
Optimizer
Learning Rate
Batch Size
Classifier
SGD (momentum=0.9)
0.01
128
Weight Network
Adam
10-3
128
B.2 Implementation Details of MICO
Algorithm 3 Model-Level Generation Capability Optimization (MLCO)
Input: Initial diffusion model gw; class prompts {cI,..., cx]
Required: Batch size B; DPO learning rate y7 reference model guz; weight network Wo; downstream classifier fe: loss function L: max iteration /; selection ratio p..
1: for iteration = 1 to do
2: Generation & Evaluation
for each class E (L,..., K} do
4: for 1 to B do
5: at 4 9* (C),
Sample er ~ ~N(00
Generate with current model
7:
w + (o(((((()))))
Evaluate utility using classification loss input
8: end for
9:
end for
10: // Preference Construction
11:
for each class & do
12:
Sort {u(²) by wf descending
13:
Select top proportion of samples as DY, bottom proportion of samples as D
14:
Dpeet f- Dpeet U {(CA, x" ,/)))=))))))
_.) x E DE e )
15:
end for
Model Optimization 17: for each (ck, 30 .x') in preat do
18:
t/- - 1/V Lorollee Peet Ck2 2 )
Update via Eq
19:
end for
20:
end for
Output: Optimized diffusion model gē
The proposed Model-Level Generation Capability Optimization (MLCO) framework iteratively improves the diffusion model's generative ability based on utility-guided preferences. In each iteration, the process proceeds in three stages, and he full process is illustrated in Algorithm
24

---

Generation & Evaluation: The current diffusion model Iv generates synthetic images for each class prompt. Each generated sample is then evaluated by the trained weight network Wo obtain utility scores w.
Samples within each class are ranked by their utility scores. The top and bottom p proportions are selected to form preference pairs pref-
Preference Construction:
Model Optimization: Using Direct Preference Optimization (DPO), the model is updated to customize its generation capability based on utility-guided preferences over training data.
Our implementation of DPO is adapted from Diffusion-DPPO [66], which enables preference- based training of diffusion models for guiding generative outputs with uuility-informe preferences. The loss function Eq. [ Jsed in Line of Algorithm follows this framework. For the detailed mathematical derivation of Eq. []] please refer to the original Diffusion-DPP paper [66]. The key hyperparameters used our DPO training process are summarized Table 9
Table 9: Hyperparameters used in DPO training
Batch Size
Max Steps Per Class
Learning Rate
Gradient Accumulation Steps
Beta DPO
400
1 10
5000
B.3
Implementation Details of ILPO
The prompt-noise optimization process consists of two primary components: the optimization of prompt embeddings and the initial noise used during generation. The goal maximize the utility of each synthetic sample while ensuring semantic alignment with the target domain.
Prompt embedding optimization: The optimization process begins with textual inversion [65]
to establish class-specific prompt embeddings that align with target concepts. Specifically, we use (79] to select initializer token for textual inversion [65] for each class. After obtaining the class-specific prompt embeddings aligned with the target labels, these embeddings are refined through gradient- based optimization to maximize the utility score of the generated samples. The learning rate for prompt optimization is set to 0,001, and the optimization process runs for 400 epochs ensure convergence and meaningful results.
DeepSeck-RI-Distil--Weee
Table 10: Hyperparameters used in textual inversion
655
Batch Size
Learning Rate
Training Steps
Instance Images per Class
1 10-4
400
16
Noise optimization: For noise optimization, we leverage the discrepancy between denoising and inversion Classifier-Free Guidance (CFG) scales, which allows us to implicitly inject prompt semantic information into the noise vector. The denoising guidance scale is set to 5.5, while the inversion guidance scale is set to O.
The specific hyperparameters used for this optimization process summarized in Table I1
Table 11: Hyperparameters used in the prompt-noist optimization process.
Guidance Strength
Guidance Strength (Inversion)
Prompt Learning Rate
Prompt Learning Epochs
(Denoise)
0.001
400
5.5
B.4
Implementation Details of Image Generation
The image generation process is guided by the hyperparameters listed in Table Instead of using fixed prompts and random noise, both the class-specific text prompts and the initial noise vectors
25

---

are optimized via our proposed ILPO strategy to better align with high-utility regions in the data distribution. The Stable Diffusion v2.1 model, fine-tuned with MLCO, is employed for the generation process, using sampling method with 50 steps, the DDIM scheduler, and a guidance scale of 2.0. The images are generated at a resolution 512 512 pixels.
Table 12: Hyperparameters for Training Data Synthesis
Base Model
Sampling Steps Scheduler
Guidance Scale
Image Size
Stable Diffusion v2.1 (ML.CO-fine-tueed)
DDIM
2.0
512x512
B.5
Implementation details of model training
The downstream classifiers are trained on four standard architectures: ResNet-55[18), ResNeXt-50
[19. WideResNet-50 (20), and MobileNetV2 (21). All models trained with identical hyperparam-
eters. The training configuration uses SGD optimizer with momentum 0.9 and weight decay 5e-4. The learning rate starts at 0.01 with cosine decay schedule over 100 epochs. A fixed batch size of 256 used for all experiments, with standard data augmentation including random horizontal flips and crops. Each experiment is repeated three times with different random seeds ensure reliability.
Table 13: Hyperparameters for Downstream Classifier Training
Optimizer
Weight decay
Initial LR
Epochs
Batch size
SGD (momentum=0.9)
5e-4
0.01
100
256
C
Limitations
While demonstrating strong performance in enhancing synthetic data utility for downstream tasks, UTILGEN presents two noteworthy considerations: (1) The dual-level optimization framework incurs modest computational overhead compared to conventional augmentation methods; (2) Although effectively improving the utility of synthetic data downstream tasks, the approach remains contingent upon the base generative model's capability to produce viable initial samples. These considcrations do not substantially compromise overall performance but indicate potential avenues for future enhancement.
D Efficiency and Cost of Data Augmentation
To evaluate the computational efficiency of UTIL GEN, we compare it against three representative
generative augmentation methods: GIF [12], GAP (72). and DataDream [[]]. The comparison is conducted under a unified setup using the ImageNet- IK dataset, where each method generates 1000 synthetic images per class. We decompose the computational pipeline into three stages: (I) Model Optimization, (2) Policy Optimization, and (3) Image Generation. UTILGEN incorporates feedback-driver optimization components at both the model and instance levels, introducing moderate overhead that remains manageable.
Table 14 presents the runtime taken for each stage, while Table 15 shows the peak GPU memory usage during execution of each stage. All methods are evaluated on a multi-GPU server equipped with 8xA100 GPUs. Despite the feedback-hased optimization, our framework remains computationally efficient. Overall, UTILGEN strikes favorable balance between computational cost and data utility, demonstrating its scalability and practicality for real-world deployments.
E
Broader Impact
Our utility-driver augmentation approach facilitates more efficient model training while reducing reliance on real data, especially benefiting domains with limited or private datasets. By generating
26

---

Table 14: Computational cost (in hours) comparison on ImageNet- for generating 1000 images per class. Values are estimated averages.
Method
Model Optimization
Policy Optimization
Image Generation
GIF 12
~29. Ih
GAP 72
445.1
99.7h
DataDream M
~00.8 8h
40..h
UTI.GGEN (Ours)
~7.7h
25.1
~41.6
Table 15: GPU memory usage (peak usage GB) per stage. maximum workload per module on each GPU.
All values are measured during
Method
Model Optimization
Policy Optimization
Image Generation
GIF 12
25.9G
GAP
5.2G
44.5G
DataDream
19.5G
15.8G
UTILGEN (Ours)
20..5
44.4G
4,.3
synthetic training data, it enhances learning efficiency and lowers dependence on large- scale real datasets. Nonetheless, since the synthesis process is guided by small set of real images, the generated data may inadvertently inherit and amplify biases present in the original samples.
task-specific
Dataset Details
To evaluate UTILGGENs performance, utilize eight benchmark datasets spanning variety of classification tasks: coarse-grained classification, fine-grained classification, and texture classification.
The coarse-grained datasets include ImageNet-Ik-Subbee 73
1ot--eet--0-Subsee 73 and Caltech 101 77). ImageNet-Ik-Suteee [73] and ImageNet- 100-Suseet [73), both randomly sampled with 100 images per class. ImageNet-100-Subbee
is subset of 100 animal-related classes from ImageNet- IK. Caltech 101 74 consists of 101 object categories. For fine-graine grained classification, we use Oxford Pets (75). Food (76), Flowers 102 [68], and
(77). with Food101- Subset [6 being a curated subset of Food101 (76) containing 101 food categories. Other datasets follow their original training and validation setups. Texture classification evaluated using the DTD 78 dataset, which contains 47 texture categories. Detailed dataset statistics are provided in Table 16, summarizing the number of classes, training samples, and test samples for each dataset. important to note that datasets with a higher number of classes or fewer average samples per class present greater challenges in terms of classification and generalization.
Table 16: Statistics of the benchmark datasets
Dataset
Task Type
Classes
Training Data
Test Data
ImageNet-1Ke -eeeNNettkk-uaee 1
73
Coarse-grained object classification
1000
100,000
50,000
ImageNet- 1ee--eet--0--Suae 100-Subset
73
Coarse-grained object classification
100
10,000
5,000
Caltech 101
77
Coarse-graines object classification
101
3060
6084
Oxford Pets 75
Fine- tine-grained object classification
37
3680
3669
Food101 1ood101-Subse Subsett 77
Fine- Fine-grained object classification
101
10100
25,250
Flowers 102
68
Fine- anne-granee object classification
102
6,552
818
777
Fine- grained object classification
200
5,994
5,794
DTD 78
Texture classification
47
1880
1,880
27


Reviews:
[{"review_id": "u0zmU86sLK", "reviewer": "Reviewer_or1d", "summary": "The paper introduces a utility-centric data augmentation framework aimed at improving the task-specific effectiveness of synthetic data used in computer vision models. While existing data augmentation methods using generative models primarily focus on visual fidelity and diversity, UTILGEN shifts the paradigm to focus on task-specific utility—how useful synthetic data is for enhancing downstream task performance.\n\nTo achieve this, UTILGEN employs a meta-learned weight allocation network that efficiently evaluates the task-specific utility of each synthetic data sample without requiring full training cycles. These utility scores then guide a dual-level optimization process: 1. Model-Level Optimization: Adapts the generative model itself to the downstream task using methods like Direct Preference Optimization (DPO). \n2. Instance-Level Optimization: Fine-tunes generation policies (e.g., prompt embeddings, initial noise) to produce more useful data samples.\n\nTheir framework is evaluated across **eight** benchmark datasets and shows an average improvement of 3.49% in accuracy over state-of-the-art augmentation methods. Notably, a ResNet-50 trained solely on UTILGEN-generated synthetic data outperforms its real-data-trained counterpart on multiple tasks. The framework is also effective across different architectures (e.g., ResNeXt, WideResNet, MobileNet), demonstrating versatility and robustness.", "questions": "The 1-5 questions asked in the Weaknesses section, would certainly affect the evaluation scores.", "limitations": "The missing limitation related to *Limited Reusability of Synthetic Data* is discussed under the Weakness section above. Apart from this, there are no other major weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n1.  Paper is very well written and easy to understand.\n\n2. UTILGEN introduces a novel shift in data augmentation—from optimizing visual fidelity and diversity to maximizing task-specific utility. This addresses a major gap in generative augmentation by aligning synthetic data generation with actual model learning needs.\n\n3. The \"meta-learned weight allocation network\" estimates task-specific sample utility without full retraining. Combined with influence function analysis and PCA-based visualization, the framework demonstrates that it generates data with stronger positive impact and clearer alignment within high-utility region. Such analysis is missing in prior methods.\n\n4. UTILGEN’s synthetic data scales effectively—performance continues to improve from 1× to 5× synthetic data, with increasing positive influence on downstream models. \n\n5. The experiments with model variants is extensive, making the results robust and effective across different multiple architectures (ResNet, ResNeXt, WideResNet, MobileNetV2).\n\n**Weaknesses**:\n\n1. The meta-learned weight network provides an efficient way to estimate task-specific utility, but its internal logic remains unclear. While data influence and distribution analyses validate its effectiveness, understanding why certain samples are rated as high-utility is not directly interpretable to user? More insights on this will aid the method's effectiveness and user's understanding.\n\n2. While UTILGEN introduces a task-aware, feedback-driven approach to data generation, it does not compare against DisCL [1], a method that also incorporates task-adaptive feedback—albeit through a curriculum learning framework. Both methods share the goal of improving downstream performance by guiding data synthesis based on task-specific signals. However, DisCL is absent from both the related work discussion and experimental baselines. Including such a comparison is important to properly position UTILGEN within the broader landscape of task-adaptive data generation methods, and to validate its generality and superiority over other task-aware strategies. \n\n3. Although UTILGEN avoids full retraining cycles via its meta-learned utility predictor, the overall framework—including dual-level optimization (model- and instance-level) and task feedback loops—remains computationally intensive. The cost-benefit tradeoff compared to collecting more *labeled real data* **isn’t fully analyzed** in the paper. \n\n4. Their results shows that synthetic data alone can match or surpass real-data-trained models on multiple benchmarks.\nHowever, This seems to differ from findings in prior work such as DataDream [2], where synthetic data is shown to be most effective when mixed with real data. Could the authors elaborate on what aspects of UTILGEN (e.g., the utility-guided generation process) you believe contribute most to this breakthrough in synthetic-only training performance?\n\n5. *Limited Reusability of Synthetic Data*: The framework still requires fine-tuning for each downstream task, including model-level generation optimization and instance-level controls. This limits out-of-the-box reusability of synthetic data across tasks. How would authors address or acknowledge this limitation ? Also Can this framework be extended to few shot setting ?\n\n**References**\n\n[1] Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion. \n\n[2] DataDream: Few-shot Guided Dataset Generation", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thoughtful feedback and insightful questions, which have helped identify key areas for improving our paper's clarity and academic rigor. After careful consideration of your suggestions, we provide the following detailed responses.\n\n>  **Q1. Concern on the interpretability of the utility weight network and why certain samples are rated as high-utility.(Weakness 1)**  \n\nThank you for highlighting the need for further clarification regarding why certain samples are rated as high-utility. As mentioned in our paper, the weights learned by our network effectively reflect sample utility for downstream tasks. This behavior is theoretically grounded in the Meta-Weight-Net[1] framework, where the updating equation for Weight-Net parameters can be explained by that **samples that better comply with the knowledge derived from validation data (i.e., those that help improve the classification model's performance on downstream tasks) will have their weights increased**, while harmful samples are down-weighted. Although we cannot show the full derivation here due to space constraints, Meta-Weight-Net[1] provides rigorous proofs (Section 2.3 and Appendices B–C). Essentially, our weight allocation network inherits this property, where higher weights reliably indicate samples that enhance classification performance, thus guiding effective utility optimization. We will include further explanation in the revised version to improve clarity.\n\n> **Q2. Suggest including DisCL[2] as a task-adaptive baseline for comparison.(Weakness 2)**\n\nThank you for the valuable suggestion. We have now included a comparison between UtilGen and DisCL[2] on the ImageNet-100 dataset. In addition, our main paper also presents a comparison with GAP[3], a feedback-driven approach that focuses on generating hard samples for downstream tasks. This unified comparison allows us to evaluate three different feedback-driven generation strategies. The evaluation covers two settings: synthetic data only, and a mixed setting combining synthetic and real data. The results confirm the superiority of our utility-guided data generation method compared to other baselines that are also guided by downstream task feedback.\n\n| Method     | Setting                  | ImageNet-100 Accuracy (%) |\n|------|:--------:|:-----------:|\n| GAP[3]        | Synthetic Only (5×)      | 30.94                     |\n| DisCL[2]  | Synthetic Only (5×)      | 37.82                     |\n| UtilGen    | Synthetic Only (5×)      | 40.86 (**+3.04**)         |\n| GAP[3]        | Synthetic (5×) + Real    | 53.14                     |\n| DisCL[2]  | Synthetic (5×) + Real    | 56.43                     |\n| UtilGen    | Synthetic (5×) + Real    | 61.14 (**+3.71**)         |\n\nWe will include additional experimental comparisons and discussion with DisCL[2] in the revised manuscript to better position UtilGen among task-adaptive generation methods.\n\n> **Q3. Analysis of cost-benefit tradeoff compared to collecting more labeled real data.(Weakness 3)**\n\nWe thank the reviewer for raising this important point regarding the cost-effectiveness of our approach. Compared to manually collecting and annotating real data, UtilGen significantly reduces both time and cost for dataset expansion. Specifically, manually annotating 10,000 images (e.g., collecting 100 images for each of 100 classes), according to Masterpiece Group, would typically take around 2 weeks and cost approximately \\$800. In contrast, generating the same amount of data with UtilGen takes only about **0.94 hours** and costs roughly **\\$20** for renting 8 V100 GPUs according to Google Cloud. Furthermore, on the ImageNet-100-Subset dataset (which contains 100 classes with 100 real images per class), we achieve better performance than the real-data baseline by using only 5× synthetic data. Generating this 5× synthetic dataset requires approximately **4.70 hours**, costing around **\\$100** for renting 8 V100 GPUs.\n\n| Method        | Images | Time Required | Cost  | ImageNet-100 Accuracy(%)  |\n|---------|:-------:|:-------------:|:----------:|:--------:|\n| Manual Annotation | 10,000  | ~2 weeks      | \\$800       | 38.58         |\n| UtilGen (1×)      | 10,000  | ~0.94 hours    | \\$20        | 18.04       |\n| UtilGen (5×)      | 50,000  | ~4.70 hours    | \\$100        | **40.86** |\n\n> **Q4. What enables UtilGen to achieve strong performance in the synthetic-only setting, unlike prior work?(Weakness 4)**\n\nWe think this breakthrough in synthetic-only training stems from  shifting the optimization objective of synthetic data toward enhancing utility for downstream tasks, thereby **aligning the generative model’s optimization objective with the downstream task goals**.  \n\nPrevious methods primarily focus on enhancing the visual quality of synthetic images; however, there often exists a gap between generating visually high-quality data and producing data that effectively improves downstream model performance. In particular, many approaches struggle to directly optimize the utility of synthetic data for target tasks.\n\nOn the other hand, efforts that concentrate on **optimizing intrinsic data attributes may introduce new limitations**. For example, DataDream[4] aims to enhance image fidelity and better align synthetic samples with the real data distribution. While effective in certain respects, this process can inadvertently reduce the diversity of the generated data. To quantify this, we compare the intra-class diversity of synthetic data generated by Stable Diffusion v2.1, DataDream[4], and UtilGen on ImageNet-100.The diversity metric is defined as the mean pairwise cosine distance between CLIP features within each class, averaged across all classes. Higher values indicate greater intra-class diversity.\n\n\n| Method                  | Mean Intra-Class Diversity ↑ |\n|------------|:-------------:|\n| Stable Diffusion v2.1  | 0.5815                      |\n| DataDream[4]          | 0.5238                       |\n| UtilGen (Ours)         | 0.6054                       |\n\n\n> **Q5. Limited reusability of synthetic data across tasks.(Weakness 5)**\n\nThank you for raising this important point. While we evaluated UtilGen's generalization across network architectures, we acknowledge that the paper did not explicitly analyze data reusability across broader task settings. Below, we offer additional analysis and discussion regarding this aspect.\n\nIf “different tasks” refers to directly using the generated data out-of-the-box as training data for different downstream models (e.g., other classifiers), we clarify that UtilGen-generated data demonstrates strong **cross-architecture transferability**. As evidenced by the table below, even if the utility predictor(i.e., Weight-Net) is trained using ResNet-50, the resulting data still yields strong performance when used with MobileNet and CLIP—suggesting that high-utility samples identified by the weight network are not tightly coupled to a specific architecture.\n\n| Classifier Used to Train Weight-Net | Downstream Model Using Synthetic Data |   Method   | ImageNet-100 Acc (%) |\n|:----------|:------:|:----------:|:---------:|\n| --                                  | WideResNet                            | DataDream[4]  |        31.76        |\n| ResNet-50                           | WideResNet                            |  UtilGen   |        36.40 **(+4.64)**        |\n| --                                  | CLIP                                  | DataDream[4]  |        71.42         |\n| ResNet-50                           | CLIP                                  |  UtilGen   |        72.14 **(+0.72)**        |\n\nHowever, when task differences involve changes in dataset or domain, UtilGen does require re-optimization to align with new objectives. We recognize this as a limitation, though it does not undermine our main contribution—demonstrating that utility-guided generation leads to highly effective synthetic data for the target task. We will dedicate our future work to making UtilGen more generalizable across tasks.\n\n> **Q6. Can this framework be extended to few-shot settings? (Weakness 5)**  \n\nWe appreciate this insightful question regarding the framework's adaptability to low-resource scenarios. As designed, **UtilGen naturally supports few-shot learning**, requiring only minimal labeled data while maintaining competitive performance. This capability is empirically validated in our comparative experiments with DataDream[4] on the ImageNet-100, where UtilGen demonstrates superior effectiveness in data-scarce regimes. Thank you for raising this important point, and we will include additional analysis of few-shot scenarios in our revised manuscript to further strengthen this discussion.\n\n**Experimental Setup**:  \n- Half of the images are simultaneously used for:  \n  - The training set for Weight-Net training  \n  - Textual inversion optimization  \n- The remaining half serves as the validation set for Weight-Net training.  \n\n \n| Method       | 2-shot (%) | 4-shot (%) | 8-shot (%) | 16-shot (%) |  \n|:-------|:--------:|:--------:|:-------:|:--------:|  \n| DataDream[4] | 21.70      | 27.18      | 33.44      | 35.48       |  \n| **UtilGen**  | **22.32**  | **29.16**  | **36.82**  | **39.62**   | \n\n**References**\n\n[1] Meta-Weight-Net: Learning an Explicit Mapping  For Sample Weighting\n\n[2] Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion\n\n[3] Controlled Training Data Generation with Diffusion Models\n\n[4] DataDream: Few-shot Guided Dataset Generation\n\nThank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.\n\nSincerely,\n\nThe Authors"}, {"review_id": "gBHCKWvw6t", "reviewer": "Reviewer_2gVY", "summary": "This paper presents a data augmentation technique that uses synthetic samples from a diffusion model. The authors introduce a task dependent objective for sample generation. First they formulate a bilevel optimization problem to train a simple MLP that assigns weights to samples in the cross entropy loss, with the MLP computing a utility gain for each sample. Next they leverage this utility network to generate negative and positive samples for direct preference alignment. They also propose model level and instance level tuning of class specific prompt embeddings and initial noise for each generated sample based on the utility MLP. Experimental results show improved performance when a target model is trained on synthetic data alone and on a mix of real and synthetic data using the proposed framework.", "questions": "I also have the following questions and requests:\n\n1. I am curious about the performance gap of UtilGen compared with other baselines under different synthesis budgets. How does the gap change as more data is augmented?\n\n2. Why not first perform textual inversion to adapt the diffusion model to each class and then use the synthesized samples via the textual inversion (concatenated to the real data) to train the utility MLP in the first step? This would reduce distribution shift for the MLP when later inferred for diffusion alignment and DPO labeling.\n\n3. I would appreciate if the authors could explain how sample weights relate to the model’s cross entropy loss. Is there a trend between sample weights and the uncertainty of the $ \\theta $ model, and if so how does this trend evolve across different optimization stage of $\\theta$?\n\n4. Following on the previous point, although it may be outside the current method’s scope and that is absolutely fine if it is not addressed because of time constraints, it would be interesting to evaluate performance when the utility MLP is trained iteratively:\n\n   1. First augment the dataset by a factor of 1×.\n\n   2. Fine tune the MLP to produce updated sample weights after training on this augmented set.\n\n   3. Synthesize new samples using the improved MLP to achieve an 2× augmentation factor and so on to 3x and beyond...\n\n   This experiment would reveal whether adapting the MLP for weighting at different training stages yields additional gains.\n\n\nI thank the authors for the good work and I would be willing to raise my score upon addressing the questions.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n• The paper is well structured; topics flow logically.  \n• Formatting is clean; figures and tables are of high quality.  \n• The proposed method is novel and improves over prior work.  \n• Ablations and experimental design follow a logical sequence and address readers’ questions as they arise.  \n\n**Weaknesses**\n\n1. The implementation details require further clarification on several points of the method:\n\n   a. How is the bilevel optimization solved? Obtaining $\\theta^*(\\phi)$ requires gradient unrolling to backpropagate through $\\phi$, which can demand substantial memory.  \n\n   b. How is the prompt optimized during the denoising stage? Diffusion models involve multiple denoising steps and backpropagating through prompt parameters may incur high computational cost.  \n\n2. The authors claim that UtilGen enhances diversity but the connection between the utility MLP and diversity promotion is not clear. It would be useful to report a diversity metric such as pairwise distance in feature space (or any other) to quantify the diversity of synthesized samples.", "was_revised": true, "final_justification": "The concerns regarding the lack of additional experimental details on solving the bilevel optimization problem and textual inversion have been resolved. The authors have also provided additional ablations that further clarify the effectiveness of their method, including comparisons against baselines under different data regimes, mixing text-inverted synthetic data with real data for training the MLP, and a diversity analysis.\n\nI have raised my score accordingly.", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thoughtful feedback and insightful questions, which have helped identify key areas for improving our paper's clarity and academic rigor. After careful consideration of your suggestions, we provide the following detailed responses.\n\n>  **Q1. How is the bilevel optimization solved?(Weakness 1.a)**\n\nWe implement the bi-level optimization through **one-step gradient unrolling**, where the classifier performs a single gradient update step on the training data, and then the weight network is updated based on the validation loss. While gradient unrolling can be memory-intensive, in our setup it remains computationally tractable due to the lightweight MLP architecture of the weight network and the limited unrolling steps. This approach effectively avoids excessive computational and time costs, making the optimization practical and efficient.\nBelow, we report the peak GPU memory usage during training with and without the weight network on ResNet-50 as the classifier:\n\n| Classifier  | Weight Network | Peak Memory (GB) |\n|-------------|:---------:|:---------:|\n| ResNet-50   | ✗              | 14.8              |\n| ResNet-50   | ✓              | 22.9              |\n\nAs shown, incorporating the weight network introduces only a modest increase in memory consumption, demonstrating the practicality of our bilevel optimization implementation.\n\n>  **Q2. How is the prompt optimized during the denoising stage?(Weakness 1.b)**\n\nWe thank the reviewer for raising this important point. In UtilGen, we optimize the prompt embeddings by backpropagating through the denoising process of the diffusion model. This involves computing gradients through denoising steps, which can indeed incur additional computational overhead. We acknowledge this as a limitation of our current design. However, in practice, **the overhead remains manageable because we only optimize one prompt per class**, rather than per instance. This significantly reduces the total optimization cost and makes the method practically feasible.\n\nTo evaluate the computational efficiency, we compare our method UtilGen with GAP[1] in terms of memory and runtime for prompt optimization. A broader comparison of UtilGen and all other baselines in terms of GPU memory usage and runtime is provided in Appendix D.  In future work, we plan to explore more efficient prompt optimization strategies to further reduce the computational burden while preserving performance.\n\n| Method     | GPU Memory (GB) | Runtime (GPU hours) | ImageNet-100 Acc (%) |\n|------------|:---------:|:---------:|:---------:|\n| GAP[1]        | 5.2             | 36.1             | 30.94    | \n| UtilGen    | 4.4             | 20.1             | 40.86   |\n\n>  **Q3. Diversity analysis of synthetic data.(Weakness 2)**\n\nWe thank the reviewer for the valuable suggestion. UtilGen maintains data diversity during the optimization of data generation. To quantitatively evaluate the diversity of synthesized samples, we compare the **mean intra-class diversity** of synthetic data produced by Stable Diffusion v2.1, DataDream[3], and UtilGen on ImageNet-100.\n\nThe diversity metric is defined as the mean pairwise cosine distance between CLIP (ViT-L/14) features within each class, averaged across all classes. Higher values indicate greater intra-class diversity, which typically benefits model generalization.\n\n| Method                 | Mean Intra-Class Diversity ↑ |\n|------------------------|:----------------------------:|\n| Stable Diffusion v2.1   | 0.5815                       |\n| DataDream [3]           | 0.5238                       |\n| UtilGen         | 0.6054                       |\n\nThese results demonstrate that UtilGen produces diverse samples within classes compared to the baselines, supporting the claim that our utility-guided generation process also does not harm or suppress diversity, which is beneficial for downstream tasks.\n\n>  **Q4. How does the performance gap between UtilGen and other baselines evolve under different synthesis budgets?(Question 1)**\n\nWe thank the reviewer for the insightful question. We conducted a controlled comparison of UtilGen and four baselines (SD v2.1, GIF[2], GAP[1], and DataDream[3]) under varying synthesis budgets (i.e., the ratio of synthetic to real data). As shown in the table below, UtilGen consistently outperforms all baselines, and the performance gap becomes more pronounced as the synthesis budget increases.\n\nThis trend suggests that **UtilGen better scales with larger amounts of generated data**, owing to its utility-centric data augmentation.\n\n| Synthesis Budget (× real data) | SD v2.1 | GIF[2]  | GAP[1]  | DataDream[3] | **UtilGen (Ours)** |\n|---------------------------------|:---------:|:------:|:------:|:-----------:|:--------------------:|\n| 1×                            |  12.18   | 13.44 | 15.02 | 17.12      | 18.04 **(+0.92)**           |\n| 3×                            |  20.20 | 21.90 | 21.16 | 25.26      | 28.52 **(+3.26)**           |\n| 5×                            |  27.96   | 31.94 | 30.94 | 35.48      | 40.86 **(+5.38)**           |\n\n\n\n\n>  **Q5. Why not perform textual inversion before utility learning to reduce distribution shift?(Question 2)**\n\nWe thank the reviewer for the insightful suggestion. In our initial design, we trained the utility MLP using synthetic samples directly generated from the pre-trained SD v2.1 model, without applying textual inversion. This was motivated by the observation that such raw generations often contain low-utility samples (e.g., off-distribution or task-irrelevant data), which we believed would help the utility MLP better distinguish varying levels of usefulness.  \n\nFollowing the reviewer’s suggestion, we conducted additional experiments using class-adapted samples generated via textual inversion to train the utility MLP. These adapted samples reduced the distribution shift encountered during downstream diffusion alignment and DPO labeling. Our experiments confirmed that using these samples indeed improves the effectiveness of the utility MLP.  \n\nMoreover, we also found that **combining both types of data—raw samples from SD v2.1 and class-adapted samples from textual inversion yields the best overall performance**. This hybrid training strategy not only mitigates distribution mismatch but also enhances the MLP’s ability to assess utility across a broader quality spectrum.\n\n| Training Strategy for Utility MLP     | ImageNet-100  Accuracy (%) |\n|:------------------------------------- |:-------------------------------:|\n| Raw SD v2.1 samples only + real data              |              40.86              |\n| Textual inversion samples only + real data       |              41.58              |\n| Raw + Textual inversion (hybrid) + real data     |            **41.94**            |\n\n>  **Q6. What is the relationship between sample weights and model uncertainty or loss?(Question 3)**\n\nIn our analysis, we observe a nonlinear relationship between the sample weights predicted by the utility MLP and the model’s cross-entropy loss. While we cannot include plots here, the general trend roughly forms an **“inverted U-shaped” curve**, where the weights tend to increase with the loss when the loss is relatively small, but starts to decrease as the loss becomes large.\n\nThis pattern aligns well with intuition: samples with very low loss are often easy or redundant, thus contributing less utility to model performance; samples with very high loss are likely noisy or exhibit significant distribution shifts, and are therefore also assigned lower weights.\n\nRegarding the evolution of this relationship across different optimization stages, our experiments indicate that the correlation between sample weights and cross-entropy loss gradually stabilizes to this nonlinear “inverted U-shaped” trend as training progresses.\n\n\n>  **Q7. Would iterative utility learning improve performance?(Question 4)**\n\nWe thank the reviewer for this insightful suggestion. We have conducted experiments to evaluate the iterative training of the utility MLP as proposed. Specifically, we compared two settings:\n\n1. **Direct 5× augmentation:** generating all synthetic samples at once using a single utility MLP trained initially.\n\n2. **Iterative augmentation:** first generating 1× synthetic data, fine-tuning the utility MLP based on the augmented set, then generating an additional 1× data with the improved MLP, and repeating this process until reaching 5× augmentation.\n\nOur results show that **the iterative approach achieves better performance compared to direct generation**, confirming that adapting the utility MLP at different training stages can bring additional gains. However, this improvement comes at the cost of increased computational overhead due to repeated MLP fine-tuning and data generation steps.\n\n| Augmentation Strategy     | Accuracy (%) |\n|:--------------------------|:------------:|\n| Direct 5× Augmentation    |     40.86    |\n| Iterative (1× → 2× → 5×) |  43.10 **(+2.24)**   |\n\n**References**\n\n[1] Controlled Training Data Generation with Diffusion Models\n\n[2] Expanding Small-Scale Datasets with Guided Imagination\n\n[3] DataDream: Few-shot Guided Dataset Generation\n\nThank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.\n\nSincerely,\n\nThe Authors"}, {"review_id": "SIGd5uFVSl", "reviewer": "Reviewer_4Eig", "summary": "This paper addresses the problem of designing data-generation models that are well aligned with a downstream task. A key element of this setup is a weight network, which learns to define weights associated with data samples, reflective of their utility for a given task, e.g., classification. Once this weight network is learned, it is used to refine the synthetic data generation, such that it yields high-weight data. Emphasis is placed on diffusion models. Additionally, the noise generation and text-prompt embeddings are also optimized, again leveraging the weight network.", "questions": "1. I found the discussion in Sec 3.1 not very clear, particularly Eqs (2) and (3). In (3), you are optimizing \\phi, it seems, parameters associated with the weight network. It seems therefore that you are optimizing \\omega_i. These weights impact the learning of \\theta in (2). Please provide some clarification of what you are saying mathematically here, as it is not very clear. I was helped by Fig 2, which at least was able to give me the idea of what you are trying to say.\n\n2. Typos:\n\na) Line 114: \"preferences preferences\"\nb) Line 119: \"during training process\"; missing a \"the\"\nc) Line 146: \"MLPO\"; I think you mean MLCO", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses an important problem and provides an effective procedure\n2. The demonstration figures are helpful in giving intuition (Fig 1) and in providing a concise summary of the approach (Fig 2)\n\nWeaknesses:\n\n1. Aspects of the paper are not very clearly written (Fig 2 was important for understanding the idea, as some of the text description could have been more clearly written)\n2. It is said that an advantage of this approach is that it doesn't need classifier-based methods to evaluate the quality of the generated data. However, the method used to learn the weight network relies heavily on training aligned with a downstream task, e.g., classification. There is a need for sufficient training and validation data to make this model (weight network) effective. While it is true that the later stages of the setup do not need a classifier, the entire approach rests on the quality of the weight network (stage 1), and a lot of data may be needed to learn that well.", "was_revised": true, "final_justification": "There were technical issues in the original submission that were not clear as written, and this led to excessive reliance on figures by the reader. The authors did a nice job in their rebuttal of addressing these concerns, and I am confident that they will address these issues in the revision. This is a borderline paper, but I recommend acceptance.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful feedback and insightful questions. Your comments have helped us to identify areas where we can improve the clarity and rigor of our paper. We have carefully considered your suggestions and provide our detailed responses below.\n\n>  **Q1. Concern regarding the claim of not requiring a classifier-based method to evaluate data utility, and the potential reliance on training and validation data for utility learning (Weakness 2)**\n \nWe appreciate these insightful questions and would like to provide further clarification.\n\n**- Evaluation without exhaustive classifier-based re-training**: We appreciate the reviewer’s comment and would like to clarify that our claim of not requiring classifier-based methods refers to **avoiding exhaustive re-training of downstream classifiers**. Instead, it uses a weight network as a surrogate utility estimator, which enables efficient evaluation of synthetic samples based on their estimated contribution to downstream performance. This avoids the computational burden associated with repeatedly training classifiers on different subsets of generated data.\n\n**- Training data requirements of the weight network**: We thank the reviewer for raising this important point. While the weight network is indeed trained on a large number of samples, we would like to clarify that **this does not require a large-scale real labeled dataset**. In fact, **UtilGen naturally supports few-shot learning**, requiring only minimal labeled data while maintaining competitive performance. The remaining training data for the weight network consists entirely of synthetic samples. This capability is empirically validated in our comparative experiments with DataDream[4] on ImageNet-100, where UtilGen demonstrates superior effectiveness in data-scarce regimes.\n\n**Experimental Setup**:  \n- Half of the images are simultaneously used for:  \n  - The training set for Weight-Net training  \n  - Textual inversion optimization  \n- The remaining half serves as the validation set for Weight-Net training.  \n\n| Method    | 2-shot (%) | 4-shot (%) | 8-shot (%) | 16-shot (%) |\n|:---------:|:----------:|:----------:|:----------:|:-----------:|\n| DataDream[1] | 21.70     | 27.18      | 33.44      | 35.48      |\n| UtilGen   | 22.32      | 29.16      | 36.82      | 39.62       |\n\n> **Q2. The Clarity of Section 3.1 and Equations (Weakness 1 & Question 1)**\n\nWe thank the reviewer for the helpful suggestion, and we will provide a more transparent and detailed description of the equations in Section 3.1 to improve clarity and understanding. Below we provide a step-by-step clarification of the interaction between the classifier parameters $\\theta$, the weight network parameters $\\phi$, and the computed weights $\\omega_i$.\n\n---\n\n### **Bi-Level Optimization Procedure**\n\nOur method employs a bi-level optimization framework with two alternating stages. The goal is to meta-learn a weight network $\\mathcal{W}_\\phi$ that assigns utility scores $\\omega_i \\in [0, 1]$ to data points, such that samples with higher utility contribute more effectively to downstream generalization.\n\n**Stage 1: Inner Optimization — Classifier Training**\n\nGiven a current weight network $\\mathcal{W}_\\phi$, we first compute the weight $\\omega_i$ for each training example using the weight network. \n\nThe classifier parameters $\\theta$ are then optimized on the merged training set using a weighted loss:\n\n$$\n\\theta^*(\\phi) = \\arg\\min_{\\theta} \\frac{1}{N} \\sum_{i=1}^{N} \\omega_i \\cdot \\mathcal{L}(f(x_i; \\theta), y_i)\n\\quad \\text{(Eq. 2)}\n$$\n\nNote that $\\theta^*$ is a function of $\\phi$ because the weights $\\omega_i$ used in training depend on $\\mathcal{W}_\\phi$. This shows that the optimal classifier $\\theta^{\\ast}(\\phi)$ is implicitly determined by the weight network parameters $\\phi$ via the computed weights.\n\n**Stage 2: Outer Optimization — Weight Network Update**\n\nOnce the classifier has been trained with the current weights $\\omega_i$, we evaluate its generalization on a validation set. The goal of the outer optimization is to update $\\phi$ such that the classifier trained with weighted loss performs well on validation set:\n\n$$\n\\phi^{\\ast}(\\theta) = \\arg\\min_{\\phi} \\frac{1}{M} \\sum_{j=1}^{M} \\mathcal{L}(f(x_j; \\theta^{\\ast}(\\phi)), y_j)\n\\quad \\text{(Eq. 3)}\n$$\nThis outer loop updates $\\phi$ so that the next round of training with newly computed weights $\\omega_i$ better guide the classifier training, leading to improved downstream generalization.\n\nThis process is repeated iteratively: the weight network is progressively refined to better estimate the utility of each sample, thereby guiding the classifier to train more effectively using weighted loss in the next iteration. Over time, the weight network progressively learns to assign higher weights to training samples that contribute more significantly to improved validation performance. In other words, the learned weights act as a surrogate for each sample's utility with respect to downstream generalization.\n\n---\n\n### **Intuition**\n\nIn essence, we are **meta-learning** the weight network to simulate the effect of each training sample on validation performance:\n\n- If increasing the weight of sample $x_i$ leads to a lower validation loss, $\\mathcal{W}_\\phi$ is encouraged to assign it a higher weight.\n- Conversely, samples with low or negative impact on generalization are down-weighted.\n\nThis interaction forms a feedback loop between the classifier training (inner loop) and the weight assignment (outer loop), enabling the weight network to serve as a proxy utility estimator for guiding data generation in later stages.\n\nWe appreciate the reviewer’s reference to Fig. 2 — we will further revise Section 3.1 to explicitly describe these dependencies and clarify the semantics of the equations involved.\n\n\n>  **Q3. Typos and Wording Issues.(Question 2)**\n\nWe thank the reviewer for pointing out the typos and wording issues. We have carefully revised the manuscript to correct these and other textual errors for improved clarity and accuracy.\n\n**References**\n\n[1] DataDream: Few-shot Guided Dataset Generation\n\nThank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.\n\nSincerely,\n\nThe Authors"}, {"review_id": "KvMO8X7Q6F", "reviewer": "Reviewer_TgCV", "summary": "While most existing generative data augmentation methods prioritize the generation of high-fidelity and diverse images, this paper argues that practical tasks and model architectures often necessitate tailored training data distributions rather than generic visual quality. To address this, the authors propose Utility-Generated Data Augmentation (UtilGen), a framework that explicitly optimizes synthetic data to enhance task-specific performance, rather than merely adhering to visual standards. UtilGen introduces three key components: 1) Task-Oriented Data Valuation, which is a meta-learned weight allocation network that quantifies the task-specific utility of synthetic data. 2) Model-Level Generation Capability Optimization, which finetune the generator model (stable diffusion) using  DPO. 3) Instance-Level Generation Policy Optimization, which optimizes prompt embedding and input noise to maximize the utility weights of generated images. Extensive experiments across 8 datasets demonstrate that UtilGen outperforms prior state-of-the-art (SOTA) methods by an average of 3.49% in performance, validating its effectiveness in aligning synthetic data with downstream task requirements", "questions": "1.\tHow sensitive is the performance of UtilGen to the choice of the initial classifier architecture and training/val data selection?\n\n2.\tAre there any scenarios or types of datasets where use of UtilGen might not be effective?\n\n3.\tThe framework seems to have several hyperparameters for each components. How sensitive is the method to the choice of these hyperparameters, and do you have any guidelines for setting them?\n\n4.\tHow does this approach perform when we use pretrained vision transformer models as the classifier?", "limitations": "Yes, listed in appendix", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n\n•\tThe paper's main strength is its proposal of a utility-centric paradigm for generative data augmentation. This is a departure from the conventional focus on visual fidelity and diversity, offering a more targeted and more effective approach to data augmentation.\n\n•\tA meta-learned weight allocation network provides an efficient, retraining-free solution for task-oriented data valuation.\n\n•\tModel-level and instance-level optimization provides a comprehensive framework for tailoring both the generative model and the individual generated samples for maximizing utility weights.\n\n•\tPaper provides rigorous experimental results on various datasets, showcasing an average of 3.49% performance improvement compared to previous SOTA approach\n\n•\tPaper provides all necessary ablation study and results which provide valuable insights into why the proposed method is effective.\n\nWeakness:\n\n•\tThe proposed framework's complexity arises from its integration of multiple modules and training stages (e.g., TODV, MLCO, ILPO, DPO, textual inversion), which may pose challenges for practical implementation\n\n•\tWhile the paper emphasizes the efficiency of its utility evaluation method, the overall framework, e DPO fine-tuning and instance-level optimization, could be computationally intensive. The computation cost is given in the appendix.\n\n•\tThe performance of UtilGen is likely dependent on the quality of the initial classifier and the representativeness of its feature, as the utility weightage network uses the feature from the classifier model. The paper could have explored the sensitivity of overall method on choice of different classifier such as CNNs vs pretrained transformer models.\n\n•\tPaper could also compare their approach against simple augmentation or self-supervised pretrained models.", "was_revised": true, "final_justification": "I appreciate the author's response to all my questions. After reviewing the rebuttal, I maintain that my original rating remains appropriate.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful feedback and insightful questions. Your comments have helped us to identify areas where we can improve the clarity and rigor of our paper. We have carefully considered your suggestions and provide our detailed responses below.\n\n>  **Q1. The proposed framework is complex due to the integration of multiple modules.(Weakness 1)**\n\nWe thank the reviewer for this thoughtful concern. Each module in our framework is designed with clear modularity, allowing **independent development and evaluation**. This facilitates practical adoption while enabling the components to work synergistically for the substantial performance gains observed in our experiments.\n\n>  **Q2. Efficiency concerns regarding the overall computational cost of the framework.(Weakness 2)**\n\nWe thank the reviewer for highlighting this important aspect. While UtilGen integrates several components to improve sample quality and task alignment, we would like to clarify that the overall framework remains computationally reasonable. As detailed in Appendix D, UtilGen exhibits favorable GPU memory usage and runtime compared to existing baselines. **We believe these costs are justified by the substantial accuracy improvements, and we plan to further explore efficiency optimizations in future work.**\n\n**Table: Computational cost (in hours) comparison on ImageNet-1K for generating 1000 images per class**\n\n| Method              | Model Optimization | Policy Optimization | Image Generation |\n|:--------------------|:------------------:|:-------------------:|:----------------:|\n| GIF[2]                 | --                 | --                  | ~229.1h          |\n| GAP[1]                 | --                 | ~45.1h              | ~93.7h           |\n| DataDream[1]           | ~10.8h             | --                  | ~40.2h           |\n| UtilGen       | ~7.7h              | ~25.1h              | ~41.6h           |\n\n**Table: GPU memory usage (peak usage in GB) per stage**\n\n| Method              | Model Optimization | Policy Optimization | Image Generation |\n|:--------------------|:------------------:|:-------------------:|:----------------:|\n| GIF[2]                 | --                 | --                  | ~25.9G           |\n| GAP[1]                 | --                 | ~5.2G               | ~4.5G            |\n| DataDream[3]           | ~19.5G             | --                  | ~15.8G           |\n| UtilGen     | ~20.5G             | ~4.4G               | ~4.3G            |\n\n\n\n>  **Q3. The sensitivity of overall method on choice of different classifier such as CNNs vs pretrained transformer models.(Weakness 3)**\n\nWe thank the reviewer for this valuable point. To investigate the sensitivity of UtilGen to the choice of classifier, we conducted experiments on ImageNet-100 using various architectures, including CNNs (ResNeXt, WideResNet, MobileNet) and pretrained transformer models (CLIP). As shown in the table below, UtilGen consistently improves downstream performance across different classifiers, **demonstrating robustness to the classifier choice**. This validates that our weight network can effectively leverage diverse model architectures without being overly dependent on a specific backbone.\n\n| Method      | ResNeXt-50 | WideResNet-50 | MobileNetV2 | CLIP   |\n|-------------|:----------:|:-------------:|:-----------:|:------:|\n| GIF         | 27.54%     | 27.84%        | 31.24%      | 69.24% |\n| GAP         | 27.66%     | 27.76%        | 32.72%      | 69.62% |\n| DataDream   | 31.24%     | 31.76%        | 35.48%      | 71.14% |\n| **UtilGen** | **37.50%** | **37.78%**     | **40.56%**  | **72.18%** |\n\n>  **Q4. Compare the approach against simple augmentation or self-supervised pretrained models.(Weakness 4)**\n\nWe thank the reviewer for the helpful suggestion. We compare UtilGen with Cutout[4] and CLIP on ImageNet-100 using a 10× enlarged training set that combines both real and synthetic data. For fair comparison, Cutout[4] and UtilGen are trained using ResNet-50. UtilGen significantly outperforms Cutout[4] and even slightly surpasses CLIP in the zero-shot setting, while fine-tuning CLIP on the same augmented dataset yields the best performance. These results **highlight the effectiveness of our approach compared to simple augmentation and self-supervised pretrained models**. \n\n| Method                                              | ImageNet-100 Accuracy (\\%) |\n|-----------------------------------------------------|:---------------------------:|\n| Cutout[4] (ResNet-50)                              |           40.32            |\n| CLIP (Zero-Shot)                                    |           69.12            |\n| UtilGen (ResNet-50)                        |           69.74            |\n| CLIP (Fine-tuned on real + synthetic data, 10×)     |         74.68          |\n\n\n>  **Q5. Sensitivity to the choice of initial classifier architecture and train/val data selection.(Question 1)**\n\nAs noted in **Q3**, UtilGen is not sensitive to the initial classifier architecture, showing consistent gains across different backbones. It is also robust to variations in training/validation data selection: for training the weight network, we randomly sampled a few real images per class, and observed consistent improvements across different random splits. This stability indicates that UtilGen is not sensitive to the choice of train/val data.\n\n\n>  **Q6. Potential scenarios or dataset types where UtilGen may not perform well.(Question 2)**\n\nSince UtilGen relies on a few real images for guidance, **very low-quality inputs** (e.g., noisy or low-resolution) may reduce generation quality by affecting textual inversion. Nonetheless, this has minimal impact on overall performance and does not compromise our main contributions. This also suggests a valuable direction for future work—improving robustness to low-quality real data.\n\n>  **Q7. Sensitivity to hyperparameter choices and guidelines for tuning.(Question 3)**\n\nWe appreciate the reviewer’s question. The implementation details and hyperparameter configurations for each component of our framework are provided in Appendix B. \n\nAmong all components, the **most sensitive hyperparameter** is the **guidance scale** used in the diffusion model for image generation. We conducted experiments to study its impact:\n\n| Guidance Scale | ImageNet-100 Accuracy (%)| \n|----------------|:------------------:|\n| 5              | 37.20                |\n| 2              | 40.86                | \n\n\n>  **Q8. Performance when using pretrained vision transformer models as the classifier.(Question 4)**\n\nWe evaluated UtilGen using CLIP as the downstream classifier and compared it with DataDream under the same setting on the ImageNet-100 dataset. Both methods use only 5× synthetic data for training. UtilGen consistently achieves better performance:\n\n| Method                 | ImageNet-100 Accuracy (%) |\n|------------------------|:------------------------:|\n| DataDream (CLIP)       |           71.14          |\n| UtilGen (CLIP)         |       **72.18 (+1.04)**  |\n\nThis shows that UtilGen remains effective even when using pretrained transformer-based classifiers.\n\n**References**\n\n[1] Controlled Training Data Generation with Diffusion Models\n\n[2] Expanding Small-Scale Datasets with Guided Imagination\n\n[3] DataDream: Few-shot Guided Dataset Generation\n\n[4] Improved regularization of convolutional neural networks with cutout.\n\nThank you once again for your valuable feedback, which has helped us to significantly improve the scientific rigor of our paper. We will incorporate the corresponding revisions and clarifications into the updated manuscript. If you have any further questions or concerns, we would be glad to address them.\n\nSincerely,\n\nThe Authors"}]

Author rebuttal:

Meta-review: In this paper, the authors present a data augmentation technique that optimizes sample generation to maximize downstream task performance through a bilevel optimization problem.  The method is based on model-level optimization and instance-level optimization.  Their technique is extensively evaluated and shows consistent improvements over SOTA on a variety of tasks and architectures.  The paper has significant novelty and has the potential to lead to a shift in the philosophy of building data augmentation algorithms.
Decision: Accept (poster)