Increasing the Utility of Synthetic Images through Chamfer Guidance
Authors: Nicola Dall'Asen
Xiaofeng Zhang
Reyhane Askari-Hemmat
Melissa Hall
Jakob Verbeek
Adriana Romero-Soriano
Michal Drozdzal
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
synthetic data diversity
guidance-based generation
Primary area: applications

Abstract: Conditional image generative models hold considerable promise to produce infinite amounts of synthetic training data. 
Yet, recent progress in generation quality has come at the expense of generation diversity, limiting the utility of these models as a source of synthetic training data.
Although guidance-based approaches have been introduced to improve the utility of generated data by focusing on quality or diversity,
the (implicit or explicit) utility functions oftentimes disregard the potential distribution shift between synthetic and real data. In this work, we introduce Chamfer Guidance: a training-free guidance approach which leverages a handful of real exemplar  images to characterize the quality and diversity of synthetic data. We show that by leveraging the proposed Chamfer Guidance, we can boost the diversity of the generations w.r.t. a dataset of real images while maintaining or improving the generation quality on ImageNet-1k and standard geo-diversity benchmarks. Our approach achieves state-of-the-art few-shot performance with as little as 2 exemplar real images, obtaining 96.4% in terms of precision, and 86.4% in terms of distributional coverage, which increase to 97.5% and 92.7%, respectively, when using 32 real images. We showcase the benefits of the Chamfer Guidance generation by training downstream image classifiers on synthetic data, achieving accuracy boost of up to 15% for in-distribution over the baselines, and up to 16\% in out-of-distribution.
Furthermore, our approach does not require using the unconditional model, and thus obtains a 31\% reduction in FLOPs w.r.t. classifier-free-guidance-based approaches at sampling time.

Increasing the Utility of Synthetic Images through Chamfer Guidance
Nicola Dall' Asll'Asse!!a 1:2+
Xiaofeng Zhang 3.4.f
Reyhane Askari-Hemmat"I
Melissa Hall
Jakob Verbeek
Adriana Romero-Soriano-44s:
Michal Drozdzal
University of Trento
University of Pisa
Mila Québec Al Institute
#FAIR at Meta
Université de Montréal
*McGill University
"Canada CIFAR Al chair
Abstract
Conditional image generative models hold considerable promise to produce infinite amounts of synthetic training data. Yet, recent progress in generation quality has come the expense of generation diversity, limiting the utility of these models as a source of synthetic training data. Although guidance-basee approaches have been introduced improve the utility of generated data by focusing on quality or diversity, the (implicit or explicit) utility functions oftentimes disregard the potential distribution shift between synthetic and real data. In this work, we introduce Chamfer Guidance: training-free guidance approach which leverages handful of real exemplar images to characterize the quality and diversity synthetic data. We show that by leveraging the proposed Chamfer Guidance, we can boost the diversity of the generations w.r./. a dataset of real images while maintaining or improving the generation quality on ImageNet- Ik and standard geo-diversity benchmarks. Our approach achieves performance with as little as 2 exemplar real images, obtaining 96.4% in terms of precision, and 86.4% in terms distributional coverage, which increase to 97.5% and 92.7%, respectively, when using 32 real images. We showcase the benefits the Chamfer Guidance generation by training downstream image classifiers on synthetic data, achieving accuracy boost of up to 15% for in-distribution over the baselines, and up to 16% in out-of -itt---diitribttion. Furthermore, our approach does not require using the unconditional model, and thus obtains 31% reduction in approaches at sampling time.
satte-of-the-ar few-shot
FL.OPs w.r.f.
1 Introduction
In the last few years, conditional image generative models [31, 47, 55] have demonstrated extraor- dinary capabilities, producing highly realistic images from both textual descriptions and class labels. These models have rapidly evolved from experimental research tools to widely accessible applications, enabling creative expression for users across various domains. With this acceleration, researchers have also started exploring the use-cases of conditional image generative models as synthetic training data for downstream machine learning models [6]. However, recent research has revealed that as models grow in size and capability, they tend to produce images of higher quality, aligned with human pref- erence, but with diminished diversity, hindering their utility as synthetic training data generators [5].
Learning bassidg-based approaches and guidance-bart based sampling techniques [4, 25, 26] have been intro- duced in the literature mitigate the shortcomings of vanilla synthetic data. On the one hand,
Work done during internship FAIR, Meta. Currently Fondazione Bruno Kessler. cqual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Redereaceffre free
Trahisghaerd
Our Chariir Gridance k-12
Vanila Ere-tazing, Charser fize tapei 2-12
2.0.3, 0.0
Figure 1: Our Chamfer Guidance addresses key limitations of existing image generation approaches, producing high-quality and diverse outputs. Base models (here LDM3,sM) necessitate high CFG scales to achieve prompt adherence and quality, at the expense of diversity. Reference-fre free methods can introduce ungrounded diversity, failing to capture the underlying data distribution. While training- based solutions effectively narrow the fidelity gap with the reference distribution, they suffer from low subject diversity, particularly in backgrounds. Our Chamfer Guidance achieves superior image quality without using CFG, substantially improving grounded coverage (C) and aligning the generated images more precisely (P) with the reference distribution. Best viewed zoomed in.
learning-basee based approaches [21, 48] aim to improve the overall utility of synthetic data by reducing the distribution shift between generated samples and real samples. These techniques leverage real data samples to train or fine-tune parts of existing conditional image generative models. On the other hand, guidance-bart based sampling methods define reward functions to guide the process of generating synthetic data. These reward functions characterize the utility of the generated samples in terms of generation quality [64] and diversity [25]. However, in-depth analyses of conditional image generative models have highlighted tension between these two desirable properties [5]. This tension becomes more apparent when defining generation diversity as the variation among generated samples without reference any target distribution. Contextualizing the desired diversity through real exemplar data holds the promise of limiting potentially disadvantageous variations in synthetic data.
Using exemplar data at inference time to guide the generation process is reminiscent of few-shot in-context learning (ICL) approaches in large language models (LLMs), where a small number of examples are used to adapt new tasks without requiring model fine-tuning. Yet, in conditional image generation, most few-shot approaches require partial model training or fine-tuning [21, 48), and the use of exemplar data inference time remains underexplored. Following this line of research, Contextualized Vendi Score Guidance (c-VSG) [25] proposed to leverage a handful of contextualizing real images to ground the diversity of generations that of the real data. However, contrary to ICL ap- proaches [1] J. increasing the number of examples did not show to improve e- CSG's performance [25].
In this work. we introduce new training- free guidance method to increase the utility of synthetic data. The method leverages exemplar real data to characterize the desired quality and diversity of synthetic data and scales gracefully with the number of exemplar real images. We define utility of synthetic data through the Chamfer distance [63] between generated and real data, and use this utility formulation as reward guidance to improve off-the-shelf state-of-the-ans conditional image generative models. We demonstrate that our Chamfer Guidance approach increases the diversity of generated images compared to previous methods, while maintaining improving generation quality. We validate these results on ImageNet- k [13] and standard geo-diversity benchmarks [22, 44]. On ImageNet- k, we reach a distributional coverage of 91.2% and 92.7% while reaching a precision of 95% and 97.5% for LDM,s [47] and L.MMa.M [18), respectively. To further illustrate the benefits of our approach, we contrast Chamfer Guidance with alternative approaches in Figure On the geodiversity task, we observe an improvement w.r.f. prior art of 6.5% and 5.7% in terms of average Fi and worst-region Fi. respectively. To validate the utility of our Chamfer Guidance, we train classifier for the ImageNet-1k 1k [13] dataset on synthetic data, and we demonstrate that our guidance can boost downstream accuracy of up to 15% over the classifier-free guidance sampling. We further investigate out-of -il-o---iittitttinn generalization, and our guidance outperforms the classifier-free guidance sampling on several ImageNet variants, and it can obtain a boost of up to 16% on ImageNet-Sketch [62] using L.M Finally, for LDM our approach does not require unconditional model to obtain state-of-the-art quality and diversity, obtaining a 31% % reduction in FL.OPs w.r.f. CFG-based approaches during sampling. With this work, we advance the synthetic training data field and show
2

---

that by designing appropriate guidance functions we can unlock the full potential of conditional image generative models and better exploit the knowledge encapsulated in them.
2 Related work
While most modern text-to-image diffusion models utilize classifier-free guidance [30], there has been increasing exploration of alternative guidance methods to further improve the sampling process. A subset of works focus improving image quality with methods including modifications to classifier- free guidance to reduce over-saturation, such as Adaptive Projected Guidance (APG) [50], utilization of blurring [32, 33] or perturbations [2], and employment of smaller, less-trained version of the model for guidance [35]. Other works explore guidance methods to improve the diversity of generated images, e.g., Condition-Annen -Aneeleed Diffusion Sampler (CADS) [49] anneals the conditioning signal by incorporating scheduled, monotonically decreasing Gaussian noise into the conditioning vector during the diffusion chain to increase diversity of the generated samples, while Limited Interval guidance [37] removes the conditioning signal in a certain interval to improve ungrounded diversity. Particle Guidance [11] overcomes the common assumption of independent samples in a batched generation, and proposes an extension of sampling where a joint-particle vioe-evolving potential enforces diversity. c-VSG, which guides the denoising process increase the diversity of a sample compared to images previously generated with the same prompt [25]. Still other works focus on safety-related guidance by reducing harmful representations, such as using classifier guidance to guide generations away from inappropriate content [9, 51] or "forgetting" concepts by zeroing out their cross-attention scores [65]. However, these methods tend focus on generation quality/diversity/ssiiee
as ends in themselves, rather than as means to increasing synthetic image corpus' utility in training downstream models.
Other recent works have begun to explore use of guidance methods to augment real image training datasets with synthetic samples for use in downstream tasks, such using the loss and entropy of the downstream classifer to guide the synthetic image generator [26], promoting sample diversity to supplement small-scale datasets [66], and leveraging textual inversion [21] to generate images for the use of long-tail image classification [53]. There has also been increasing study into the use of guidance to help in model self- self-imprrvement, such as using generated samples as negative guidance [3]. Our work complements these methods by utilizing features
to avoid mode collapse/degradates from a small set of real images in the guidance process to improve the utility of generated samples.
Finally, in lieu of guidance approaches to improve the generations of diffusion models, a standard practice is to perform fine-tuning. These methods include fine-tuning directly additional data sources that better capture the desired qualities of the generations [12, 17, 18, 47) or utilizing reward models incorporate preferences of model outputs [10, 15, 64]. For example, it is possible to directly fine-tune latent diffusion models by integrating human- -reeeeeeeeeeeenee feedback during the denoising process using Reward Feedback Learning (ReFL) [64]. Instead of the conventional generate- then- filter approach, ReFL, leverages reward model to provide gradient feedback. A key insight is that the quality of generated images becomes discernible only in the later denoising steps. At early timesteps, the reward scores are uniformly low, but once a sufficient number of steps have been taken (e.g.. after 30 out of 40 steps). the reward model produces distinguishable scores. To balance stability and effective feedback, random timestep from a designated (late) interval is chosen for fine-tuning. Although this approach has been proven successful for quality fine-tuning, it is not yet tested whether this approach would increase diversity in generation given the proper reward. In this work, we study how strong diversity and quality can be achieved by guidance without the need to perform fine-tuning.
3 Method
In this section, we introduce a training-free approach for diffusion and flow models, which leverages real data samples to characterize the desired quality and diversity of the synthetic data. Our method leverages the Chamfer distance mechanism to match sets of real and generated samples.
3.1 Preliminaries
Diffusion Models. Diffusion models [31, 55) are generative models that transform a unit Gaussian prior into a data distribution via iterative denoising. They consist of a forward and a backward
3

---

process. The forward process progressively adds noise to data over T timesteps following Markovian process q( - (z 1,, where & E (0, 1) is variance schedule. The backward process learns to reverse this by denoising sample to 24-1, modeled as
po(2;-1 [ (1 no (14, 1), Eg (x+, t)). Ho al. [31] show how we can equivalently train a denoising neural network (0 to predict the noise using the following objective:
Simple = Eg,ra.c [|| co (vārso + v1 - Ccc, 1) t)]
Sampling starts with ør (O, I). and the learned reverse process iteratively applied to obtain zo. i.e., It-1 denoise( Eq) == Jar (Iz vi-aa eo (21, + ote, with E (O, I), and
a weighting parameter [31].
Flow matching. Flow Matching (FM) is generative modeling approach that offers deterministic alternative to diffusion models by learning continuous transformations between probability distribu- tions. While diffusion models rely on stochastic processes that gradually add and then remove noise through stochastic differential equation (SDE), FM employs first-order ordinary differential equations (ODEs) to define vector field guiding the transformation from simple base distribution the target data distribution. This deterministic nature results in faster inference and reduced computational costs. Moreover, FM work with optimal transport interpolations, which can lead more direct and stable mappings compared to the often curved trajectories in diffusion models [39].
CFG. Classifier -free guidance (CFG) [30] enhances controllability in diffusion models by combining conditional and unconditional outputs during sampling as follows:
Eguidance (Zz, t, (1 + ( c) wes (2,, t),
where w adjusts guidance strength, eØ cx L, c) is the noise prediction conditioned on e (commonly class label or text prompt), and eø (2re, is the unconditional counterpart. This extrapolation trades diversity for fidelity without requiring auxiliary classifiers.
Classffier-guidance. Another approach to control the generation is classifier classifier-guidance [14], which uses a pretrained classifier (y)z) to modify the unconditional score function Vz, log (1z). yielding the following modified score function for conditional generation:
sa log bogpo(Ce (x
V s, log (o + +VV,, log DogP&Co (zre)
where y is a scaling factor that controls the strength of the guidance signal. Classifier guidance can also be used top of the CFG score prediction from Equation (2).
DDIM. sampling time, using the insights of DDIM [56], we can obtain an approximate denoised image for each timestep t,
ž DDIMApprox(2:) := St - VI (nco(Is)
Vār
Access to the approximated denoised sample is useful and necessary to apply guidance on the denoising process at inference time without modifying the diffusion model network Eq [25].
3.2 Chamfer distance for image distribution matching
The Chamfer distance [7] is metric used to quantify the similarity between two sets of points by measuring the average closest-point distance between each point one set the other and vice versa. It has been widely used in 3D computer vision and computer graphics to register point clouds [58].
In this work, represent image distributions sets of points and leverage the Chamfer distance for distribution matching. Given set of real images X' and set of generated images y, our goal is to encourage the generated set of images to be as close possible set of real target images. We compute the Chamfer distance between these sets of images as:
CChamfer (aiimnee((10) (X,
1 ExEx mingey ||> 지
y|
+
Evey mingex ||x
w/1²
(5)
The distance consists of two terms: The first term matches each real sample with closest generated counterpart. This term is reminiscent to diversity metrics used to evaluate synthetic data [36, 41],

---

and may be viewed as a particular instance of Implicit Maximum Likelihood Estimation [38], which inherently resists mode collapse by encouraging coverage of the entire data distribution. The second term matches each generated sample with its closest real counterpart. This term is reminiscent to fidelity metrics used to evaluate synthetic data [36, 41].
Representation space projection. Before computing the Chamfer distance, we project both real and synthetic images into a representation space that induces more semantically meaningful distances than a naive l2 metric in RGB pixel-space. We use DINOv2 [42], which captures semantic information through supervueeee learning. Unlike CLIP's contrastive learning with text-image pairs and Inception's 's supervised ImageNet training, DINOv2's ' self-supervised learning approach provides a feature representation that balances between focusing on important objects and holistic image structure [57]. Moreover, DINOv2's 's features have been shown to capture human- perceived similarity better [24] than representations such as CLIPP [43] and Inceptionv3 [59].
3.3 Inference-time guidance
To improve the alignment between the generated and real data distributions at inference time, without requiring any fine-tuning, we introduce a guidance mechanism based on the Chamfer distance. This builds on the general framework of guidance in diffusion models, where external signals are used to steer the generative process toward a desired objective [4, 14, 25, 26, 52].
Chamfer Guidance. We introduce Chamfer Guidance, where the external signal is derived from the Chamfer distance between batch of generated samples and batch of real samples. Let A' denote set of features extracted from real images, and let & be the denoised approximation of a batch of generated samples Fy at time step & (see Equation (4)). The guidance score becomes:
Vx, log po(zz X) Zz, log (z) VV CChamfer &oo,),
(6)
where Lchamter is the Chamfer distance between the real sample set X' and the current batch of denoised samples, as defined in Equation (5). The negative sign in Equation (6) reflects the goal of minimizing the Chamfer distance, thereby encouraging the generated samples to be close to the real data (fidelity) and to cover its modes (diversity).
Efficient approximation. Evaluating the Chamfer distance across the full reverse diffusion trajectory computationally expensive. To mitigate this, similar [4, 26], we adopt the DDIM approximation in Equation (4), which provides a first-order estimate of the final denoised sample at each time step t. This allows us compute a differentiable reward signal and its gradient during the intermediate steps of the diffusion process without completing all T reverse steps.
also be interpreted in the context of reward- based generation methods, such as ReFL. In our experiments we find, however, that applying Chamfer-based guidance inference time leads to stronger distributional alignment than using the same reward function for fine-tuning.
Chamfer reward signal. Chamfer Guidance
4 Experiments
4.1
Experimental setup
We consider three primary experimental settings. The first is object-centric setup, where the focus lies on guiding or evaluating the generation of specific object categories. This setting allows us to assess the capability of conditional diffusion models in producing accurate and semantically consistent visual representations of well-defined object classes. The second setting addresses geographic representation and targets the well-documented limitations of T2I models in handling geographically grounded content. In this setting, we aim to evaluate how well the models capture regional diversity and mitigate the visually biased depiction of locations [23, 25J. Finally, we evaluate the downstream utility of the data generated by our Chamfer Guidance by training an image classifier purely synthetic data.
Datasets. We utilize three publicly available datasets. For the object-centric setting, we use ImageNet- 1k [13], a large-scale image classification dataset containing over one million images across 1,000 object categories. For the geodiversity representation, we use GeoDE [44] and DollarStreet [22]. GeoDE contains curated set of images annotated with geographic provenance and object labels.
5

---

DollarStreet includes photographs of everyday household items from around the world, labeled by country and income level, thus enabling fine-grained analysis of regional visual representations, in line with prior works [23-25]. For downstream utility, we employ ImageNet-1K 1k [13]. We further employ ImageNet-V. V2 [46], ImageNet-Sketch [62], ImageNet-R [27], and ImageNet-A [28] to measure out-of-distribution generalization.
Models. We consider two state-of-the-art T2I models, in particular latent diffusion models (LDMs), LDM15 [47], and 5DM [18], which we use in a class-conditional and texl-conditional way.
Implementation Details. All experiments are implemented using the diffusers library [61], using the default samplers with 40 denoising steps. For the latent projection of our Chamfer Guidance, we primarily use the DINOv2 [42] (ViT-L) feature space, which offers strong semantic representations suited for fine-grained perceptual alignment and whose features have been shown to correlate better with human judgement of similarity (24). To enable comparison with prior studies [25], we also report results using CLIP embeddings [43] for the geographic diversity scenario. Evaluation in the object- centric setting is conducted using the dgm-eval library [57], while the geographic representation scenario utilizes the publicly available evaluation code from DIG-In [23]. The Chamfer distance implementation is from PyTorch3D library [45]. As in c-VSG [25], we set the inference-time guidance frequency to Greq 5, i.e., we apply Chamfer Guidance once every five denoising steps.
4.2 Object-centric scenario: ImageNet-Ik
Baselines. We employ reference-frre,, and reference-based baselines, where reference- based baselines use a few real data samples as reference. As reference-free baselines, we test the default capabilities of LDMs with different CFG [30] values, as well as advanced guidance techniques such as APG [50], CADS [49], Limited Interval guidance [37], and Particle Guidance [11]. Regarding reference-based
approaches, for LDM we test c- C-V [25], which employs a memory bank of prior outputs and real-world exemplar images to guide the generation process, balancing ungrounded and grounded diversity through the use of two Vendi Scores [19]. Additionaly, we employ two training- solutions, vanilla fine-tuning and reward-based fine-tuning [64]. For vanilla fine-tuning, we fine-tune the model through the standard denoising loss in Equation (1), using the set of real reference images X' fine-tuning data. Drawing inspiration from ImageReward [64] and their ReFL algorithm, we implement a reward-baser based fine-tuning approach which leverages a reward function derived from the negative Chamfer distance, i.e., r -Chamfeer As such, this approach requires a few real data samples to encourage the model to generate samples that minimize the distributional distance to the real data. Unlike traditional fine-tur fine-tuning which focuses on individual samples, this approach explicitly optimizes for distribution-level properties.
Metrics. To comprehensively assess both image quality and diversity aspects, we report Precision and Recall [36], Density and Coverage [41], Fj measured as the harmonic mean of Precision to Coverage, as well as Fréchet Distance (FD) [29]. Note that these metrics are grounded on a reference
dataset. We use DINOv2 features to compute all the metries as recent literature [24, 57J found that DINOv2 space provides more accurate estimations of perceptual similarity. In addition, we report the standard FD using Inception V3 features.
Number of real images. We investigate the effect of increasing the number of real images used in both fine-tuning and inference-time guidance approaches. This allows us to establish the data efficiency and scaling properties of each method. For the number of real images we consider h E [1, 2, 4, 8, 16, 32].
Implementation details. For all the experiments with i < 8, training and inference. We use multiple GPUs for k = [16, 32]. Vanilla fine-tuning and Chamfer fine-tuning solutions are trained for at most 5,000 steps with checkpoints taken every 1,000 steps. We use a constant learning rate of 10 across all experiments. For LDM,,, we fine-tune the entire U-Net backbone, while for LDM M.5M we employ LoRA [34] fine-tuning with a rank applied on the key, query, value, and output layers of attention modules. For ReFL hyperparameters, we use the
a single H100 GPU to perform
official implementation and set À 10 33 T 40, Ti = 30, T2 39.
4.2.1 Quantitative Results
We show the quantitative evaluation on ImageNet- 1k dataset for LDM1,s and LDM in Tables la and 1b, respectively. For each k we report the metries corresponding to the best Fj score w.r. the validation set of each dataset. Our analysis reveals that existing reference-free solutions are unable to

---

Table 1: Quantitative results on ImageNet- Ik using LDM1, and LDM Our Chamfer guidance consistently achieves sate-oo-the--ase fidelity, diversity, and image quality compared to reference- free and training-based approaches across both models.
(a) Results using LDM
Method
Fi C
Precision f Coverage
Density
Recall 1
FDD
FID
LDM13
0.5U7
0.723
0.391
0.656
441.244
31.289
LDM
7.5
0. 7.709
0.862
0.603
0.775
0.115
218.
218.731
16.116
APG [50]
0.723
0.626
217.937
13.399
CADS [49]
4,0
0.718
0.850
0.621
0.713
0.546
217.959
13.131
Limited Interval [37]
0.7U8
0.613
11.405
Particlo Guidance [1]]
4,0
0.719
0.816
0.625
0.711
0.544
222/261
11.516
VVG [25]
2.0
0,660
0.788
0.568
0.632
0.738
236.337
10.742
Vanilla fine-tuning tee-tunnng
7.5
0.733
0.887
0.635
0.797
0.495
220.089
14.869
Vanilla fine tuning
7.5
0.741
0.889
0.646
0.796
0.503
214.045
14.723
Chamfer fine tuning
2.0
0.759
0.886
0.064
0.868
0.420
211.714
15.464
Chamfer fine-tuning
32
2.0
0. 0.706
0.898
0.068
0.875
0.404
200. 999
15.492
Chamfer Guidance (Ours)
2.0
0.586
0.947
0.833
1.108
0.450
156. 179 13-670
Chamfer Guidance (Ours)
32
2.0
0,931
0,950
0.912
1.213
0.649
113.301
8,935
(b) Results using 5DM
Method
k:
Fi (P, C
Precision
Coverage
Density f Recall
FDD J
FTD
LDMs.M
1.0
0.599
0.752
0.498
0.560
0.667
314.733
17. 268
LDM
2.0
0.727
0.872
0.623
0.797
0.502
231,890 890 15.673
APG [50]
2.0
0.723
0
0.625
0.783
0.4155
237 623 15.
CADS [49]
2.0
0.717
0.851
0.620
0.749
0.518
238.125
15.139
Vanilla line-tuning
2 4.0
0.727
0.861
0.628
0.799
0.433
238. 836 18.539
Vanilla fine tuning
32 4.0
0.728
0.855
0.635
0.789
0.472
232.911
18.098
Chamfer line tuning
2 2.0
0.741
0.906
0.626
0.906
0.255
284.654
5:38
Chamfer line tuning
32 2.0
0.731
0.809
0.616
0.890
0.276
286.706
708
22, 22.061
Chamfer Guidance (Ours)
2 1.0
0.912
0.964
0.864
1.245
0.466
134.3305
8.878
Chamfer Guidance (Ours)
32 1.0
0.950
0.975
0.927
1.366
0.350
121,403
9.606
LDM, Vanilla fine lanin
LDM, Chamder fane using
LDM, Chanfer Guilance (Dun)
LDM Vanilla fine uuing
L0M Chanter saning
LD Chander (Ouns)
Precisiin
0.85
0000
0.73
0.55
0:50
120
Grounding samples Class
Grounding sumples/ Class
Grounding samples
Figure 2: Effect of the number of real reference samples & on LDM and LDM for ImageNet- 1k. We can see that only our Chamfer Guidance can effectively leverage the the increased number of reference images, consistently obtaining favorable trends across Coverage, Precision, and FDD.
deliver satisfactory outcomes, offering only marginal improvements over the base LDM performance in terms of diversity. c-VSG achieves the best recall, at the expense of coverage, indicating the generation of outlier images. Furthermore, we observe that neither vanilla fine-tuning nor Chamfer fine-tuning improve their performance when trained with an increased number of available training samples. Interestingly, on LDM3.5M our Chamfer Guidance obtains statc-of-the-at -frtttrrrt results in fidelity and diversity without using CFG, i.e., w 1.0. This brings a significant computational reduction w.r.t CFG-based approaches of = 31% for the case k 32. The detailed computation of efficiency presented in the Appendix. Figure 2 shows how our Chamfer Guidance effectively increases the diversity when higher number of real samples become available. Meanwhile, training- approaches only marginally increase precision when learning higher number of real samples k; while not improving the image quality as indicated by the FDD metrie. Our results on Chamfer Guidance are in line with recent trends in LLMs, where dest-time compute can be more effective than training or fine-tuning-ue compute [54].
7

---

Table 2: Comparison on the GeoDE dataset under the geographical representation benchmark of c-VSG, with model selection on Fi (P, C). Metrics are computed in Inception space. i indicates re-implemented results. AF: Africa, WAS: West Asia, AM: Americas, EU: Europe. "label" is region label, "desc" is text description, and "img" is exemplar images. Our Chamfer Guidance achieves state-of-the-ar in terms of Fi, and coverage scales with an increased amount of available samples.
F, O
Precision
Coverage
CLIPScore
Method
Ref. Info
Worst-Reg.
Worst-Reg.
Arg.
Worst-Reg,
Woest-Reg.
Arg.
Worst-Nog.
LDM,,
0.412
0.346
0.878
0.574
0.319
0:251
0.238
Synoeyms
AF
0.334
0.297
0.350
0.206
0.378
0.297
0:215
0.200
V.SF (25)
AP
0.353
0.312
0.349
0.307
0.357
0.317
0.180
0.191
Paraphmaing
0.253
0:231
Semattic Guidance"
label
AP
0.412
0.344
0.456
0.376
0.576
0.317
0.261
0.249
PG CLIP (Loss)"
label
AP
0:418
0.201
0.443
0.422
0.387
0.385
0.245
0.244
CLPP (inropy)
label
AP
0.414
0.357
0.437
0.383
0.383
0.327
0.238
0.230
Toxtual lnvessoo (21]"
img
0.300
0.4018
0.250
012336
02212
0.214
v [25] (CLIP)
img
2
AM
0.435
0.412
0.424
0.4018
0.446
0.416
0.264
0.254
<-VSC [25] (CLIP)
AF
0.412
0.357
0.425
0.382
0.358
0.335
0.253
0.253
Chamfer Guidance rOus, CLIP)
0.455
0.461
0.525
0.487
0.468
0.457
0.265
0.263
Chamfer Guidance rOus, CLIP)
AM
0.402
0.457
0.513
0.430
0,475
0.468
0.267
0.266
Chunfer Guidunee (Ouss 2FNDv2;
img
2
AP
0.188
0.456
0.539
0.513
0.446
0.411
0,257
0.348
Chamfer Guidance (Ous, DENOx2)
img
AP
0.500
0.469
0,549
0.512
0.450
0.432
0.257
0.249
4.3 Geographic Diversity
We conduct a thorough comparative analysis of our approach against existing geographic diversity enhancement techniques, particularly focusing on the contextualized Vendi Score Guidance (c- VSG method [25], which is the closest in spirit to our approach. This comparison reveals several key advantages of our Chamfer Guidance.
Datasets and evaluation metrics. We follow the same evaluation protocol of c- VSG, but we employ the Fi score between precision and coverage metrics instead of precision and recall. This is avoid inflated results due to generated outliers. We use Inception-based metrics and prompt the models with {objen ject] in {region} for fair comparison with V-V. We report results computed in DINOV2 space in the Appendix. When reporting results for "worst region",, we follow the original protocol first identify the worst region in terms of the Fi score, and then report the other metrics corresponding to the same region. This may lead to cases where the worst region has a higher value for the metric w.r.f. the average.
Implementation details. We use LDM1, s as in c- C-VSG and apply the same filtering on the GeoDE dataset as described in prior work [23, 25]. We choose the DINOv2 [42] latent space to compute the Chamfer distance, but we also report results using CLIP [43] for fair comparison with c-VSG.
4.3.1 Quantitative Results
To enable a direct comparison, our experiments replicate the settings of c- c-V [25]. Table presents comparative analysis of our Chamfer Guidance against c-VSG on the GeoDE dataset, using recomputed baselines to use the revised F1 score. Details about the baselines are presented in the Appendix. Notably, our optimal approach (DINOv2 the feature extractor, and 4 grounding samples) achieves a nearly 7% improvement in Fi score over VSV. We also observe a consistent scaling in coverage, exhibiting the same trend as in the object-centric scenario. This indicates that the observed behavior persists even in the more challenging context of geographical diversity. Furthermore, our method substantially enhances worst-region coverage by up to 4.9%, demonstrating its efficacy in mitigating regional misrepresentation. As expected, the highest image-text alignment, measured by CLIPScore, is obtained when employing the CLIP latent space for projecting images during Chamfer distance computation. This methodology effectively guides synthetic images towards the CLIP subspace corresponding to the target object. Then, we evaluate our approach on DollarStreet, with results presented in the Appendix, where we demonstrate a 3.5% improvement in the Fi score compared to c-VSG, due to the enhanced quality and diversity of the generated data. In summary, we can draw the following conclusions from these results: Chamfer Guidance enhances the quality (precision) and diversity (coverage) of generated data, and demonstrates superior maate-text alignment. Unlike c- C-VSG it exhibits favorable scaling with increased real data volume. We show qualitative results in the Appendix, where our Chamfer Guidance showcases increased quality, less saturation and more diversity.
Our analysis reveals two additional significant advantages of our approach: (1) Unlike e- CSV which necessitates storing intermediate results to continuously update the Vendi Score computation, our approach operates with significantly lower memory overhead. (2) Our method eliminates the need to balance between diversity term and contextualization term, removing hyperparameter
8

---

from the optimization process. This simplification makes our approach more accessible and easier to tune, reducing the complexity of the diversity enhancement pipeline.
Table 3:
Validation accuracy on real images of classifiers trained with real data, synthetic data, and a mix of synthetic and training data for ImagetNet-IK, All models are trained for 200k iterations.
(a) Real data only.
Real images
IN1K
IN-42
1N-Sk
IN-R
IN-A
3.94
0.21
32k
34.05
25.38
4.17
5.04
0.53
1.3M
82.60
70.90
32.50
41.60
29.
(b) Synthetic only and mixed data. Real and syntheric images refer to the number of samples used to train the classifter.
Real images
Syn. imagges
Guidance
LDMrs
LDM
INIK
IN-v2
IN-Sk
IN-R
IN-A
INIK
IN-12
IN-SK
IN-R
IN-A
47.67
40.33
20.49
17.40
1.45
37.83
34.07
17.60
11.53
0.88
0
1.3M
Chamfer 2
32.88
45.37
28.07
19:00
1.71
52.14
44.27
33.47
20.26
1.93
Chamfer k-32
54,91
28.08
19.78
5.11
53.66
45.46
34.44
20.67
5.28
2k
1.3M
Chumiee km2
48.47
53.57
41.07
46.42
21.21
29,48
21.25
16:96
1.57
1,65
52.95
40.80
31.72
45,26
20.03
33,52
12.31
20.51
1.99
1.32
32k
1.3M
39.07
49.77
25.04
20.0
2.44
65.65
45.65
21, 31.64
14.97
1.54
Chamferr k=32
63.81
53.84
32.34
22.40
2.72
62.61
52.58
34,49
21.85
2.36
4.4 Training downstream image classifiers on synthetic data
We complement our analysis by evaluating the downstream utility of our Chamfer guidance in the "static" ImageNet-1k setup introduced in [4], and generate dataset of 1, 300, 000 synthetic images using our approach, with different & real exemplar images from the training set. Each synthetic image is generated with simple prompt (class name). We report the accuracy of ViT-B [16] classifier trained on this synthetic data and tested on real validation data.
We compare the performance of LDM1, and L.MMa.MM against a classifier trained on limited real images (k times the number of classes), with results shown in Table 3. We initially train the classifier on only limited real images and present the results in Table 3a to serve reference. Our findings demonstrate that this limited real data alone fails to achieve satisfactory performance. Next, we leverage the available real data per class to generate synthetic images using our Chamfer distance approach, presenting these results in Table 3b. We study the effect of images generated without and with our Chamfer guidance, and we show that our synthetic-only approach using Chamfer guidance achieves substantial improvements over the base LDM generations, with gains of +12.01 and +15.83 accuracy, respectively. Finally, we find that combining real and synthetic data yields optimal results, achieving up to 63.81% and 62.61% accuracy when using only 32 real images per class alongside our generated synthetic data using LDM1, and LDM respectively. These results also show how our Chamfer guidance can effectively close the gap in grounded diversity between the two models. The difference is more than 5% when using only the synthetic data, and we reduce to 1.25% with k 32. An analogous behavior emerges when leveraging real data, reducing the gap from 3.42 to 1.20 with 32k real images.
We also test on ImageNet-V2 [46]. ImagcNet-Sketch [62], ImageNet-R [27], and ImageNet-A [28] to measure out-of-diste of-diisribbttin (OOD) generalization. Models trained with data generated by our Chamfer guidance exhibit strong performance these out-of- disrriibbbttion datasets, surpassing models trained on only real data on all variants, given the same number of real images. For ImageNet- Skeeth, LDMa.5M with our Chamfer guidance surpasses the performance of classifier trained on the full (1.3M samples) ImageNet- kk. Our Chamfer guidance always obtains substantial gains over the default sampling of L.DM,s and LDM3,5M across all ImageNet variants, up to 16% when using synthetic data only for ImageNet-Sketch on LDM Interestingly, on ImageNet-R and ImageNet-A, adding real data to our synthetic ones harms the model's 's performance. In these domains, which differ significantly from ImageNet-Ik, kk the gain resulting from synthetic data is less pronounced w.r.f. ImageNet- V2 and ImageNet-Sketee R(endition) and A(dversarial) represent more challenging domains where more natural-lookin looking images bring reduced benefit. Nevertheless, our Chamfer guidance still surpasses their real-only counterparts. These results confirms the downstream utility of our generated samples for OOD tasks as well. Similarly to the in-domain results, we observe how our Chamfer guidance can effectively close the performance gap between LDM1, and LDM and for ImageNet- -Sketch, the performance of LDM is superior to LDM1.s-
Finally, we would like to observe how the improvement in quality between LDM and L.MM does not correspond to greater utility of the generated data. What we generally observe is a decrease

---

in performance between LDM1s and LDM and that our Chamfer guidance can restore the utility
of the most recent models.
4.5 Ablations
We conduct our ablation studies on LDM,s, using the ImageNet-IK IK dataset. The goal is to understand the robustness of our Chamfer Guidance to the relevant hyperparameters, i.e., w from Equation (2) (in Table 4) and the strength of the Chamfer Guidance 7 from Equation (6) (in the Appendix).
CFG ablation. Table 4 presents the impact of varying the CFG scale on our method. The results clearly demonstrate that a moderate value of 2.0 with k = 32 guiding images achieves optimal performance, yielding the highest Fi score (0.931) and competitive precision (0.950) while achieving high coverage (0.912). This balance is crucial for generating both accurate and diverse images. Notably, the FID score of 8.935 at w = 2.0 indicates better image quality compared to higher guidance settings, which deteriorates to 14.388. This confirms that excessive guidance strength introduces oversaturation artifacts, compromising perceptual quality. The low FDD score (113,301) at w = 2.0 further validates the effectiveness of this configuration in producing faithful, high-quality, and diverse image generations. Interestingly, using the conditional model only (w = 1.0) produces competitive results, confirming that our Chamfer Guidance approach enables state-of the-art results while reducing computational complexity. We present a more detailed study on efficiency in the Appendix.
Table 4: w ablation on ImageNet-IK Ik using LDM,s. Our Chamfer Guidance can achieve near state-of-
ahe-art results employing only the conditional model, reducing the needed inference computation.
k F1 (P, 1 Precision Coverage 1 Density
Recall
FDD 4
FID
2
0.849
0.890
0.811
0.904
0.736
150.748
13:217
1.0
32
0.899
0.923
0.876
1.086
0.735
117.831
9.759
2.0
2
0.881
0.932
0.835
1.051
0.637
124.191
8.840
2.0
32
0,931
0,950
0.912
1,213
0.619
113,301
8.935
7.5 2
0.886
0.947
0.833
1.108
0.480
156.179
13.670
7.5
32
0.925
0.957
0.891
1,238
0.498
163.111
14.388
5
Conclusion
We introduced Chamfer Guidance, novel training-free approach improve the utility of synthetic data from conditional image models. By leveraging a small set of exemplar real images to guide the generation process, our method balances quality and diversity while addressing distribution shifts between synthetic and real data. Experimental results on ImageNet-IK Ik and geo-diversity benchmarks show Chamfer Guidance achieves state-of-the-aa performance and scales its effectiveness as exemplars increase. Additionally, our synthetic data can be used in downstream applications. Furthermore, our approach eliminates the computational overhead of CFG, reducing computational requirements while maintaining superior performance in quality and diversity. These contributions advance synthetic data generation and demonstrate that thoughtfully designed guidance can unlock the full potential of conditional image generative models for downstream applications.
Limitations. Our evaluation of reference- based diversity and quality built on automated metries, which inherit intrinsic biases, e.g., the distribution of the dataset compare against, or the reliance on pre-existing feature extractors that might not capture subtle differences. While we rely on previous works to provide the most accurate metrics possible, these are statistical aggregators and do not ac- count for individual preferences. Currently, our approach is designed for flasscconditonal models and does not directly support text-to-image generation. This limitation opens up several exciting avenues for future work. To extend our method to satttto-mmaee models, we envision a retrieval-base pipeline. First, large text-image dataset would be embedded into semantic vectors to create an offline retrieval database. Then, at inference time, for a user's 's text query, the top-k relevant images would be retrieved to serve exemplars for our guidance. Another interesting extension would be zero-shot, data-free pipeline. We could envision a self- bootstrotttrgiinn technique that first generates initial candidate images for a class, then automatically selects a diverse subset maximizing coverage (or "diameter") in robust feature space, and finally uses this synthetically-genrrate Societal Impact. Our work builds on image generative models, and it inherits some of the societal challenges of image content creation. However, our Chamfer Guidance provides a new way to guide the generations towards set of exemplary images. As such, it equips the user with a broader inference- time tool set to better control the sampling process.
set as the guidance exemplars.
Acknowledgments. This work was sponsored by the project FAIR Future AI Research (PE00000013), funded by NextGeneration EU.
10

---

References
[1]
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, al. Many-shot in-context learning. Advances in Neural Information Processing Systems (NeurlPS), 2024.
[2] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang. Jungwoo Kim, SeonHwa Kim, Hyun Hee Park. Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention attentioo guidance. In Proceedings IEEE/CVF European Conference on Computer Vision (ECCV). 2024.
[3] Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John Collomosse, and Richard Baraniuk. Self-improving diffusion models with synthetic data. arXiv preprint arXiv:2408. 163333 2024.
[4] Reyhane Askari-Hemmat, Mohammad Pezeshki, Elvis Dohmatob, Florian Bordes, Pietro Astolfi, Melissa Hall, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriano. Improving the scaling laws of synthetic data with deliberate practice. arXiv preprint arXiv:2502. 2025.
[5] Pietro Astolfi, Marlene Careil, Mclissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adri- ana Romero Soriano, and Michal Drozdzal. sesiiitteeey-diivsitt---eee pareto fronts of conditional image generative models. arXiv preprint arXiv:2406.1042) 2024.
[6] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenct classification. Transactions on Machine Learning Research (TMLR). 2023.
[7] Harry G Barrow, Jay M Tenenbaum, Robert C Bolles, and Helen C Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In Proceedings: Image Understanding Workshop, 1977.
[8] Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing text-to-image models using semantic guidance. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[9] Manuel Brack, Felix Friedrich, Patrick Schramowski, and Kristian Kersting. Mitigating inappropriatennsss in image generation: Can there be value in reflecting the world's ugliness? arXiv preprint arXiv:2305. 18398 2023.
[10] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fieet. Directly fine-tuning diffusion models on diffcrentiable rewards. In Proceedings of International Conference on Learning Representations (ICLR), 2024.
[11] Gabriele Corso, Yilun Xu, Valentin De Bortali, Regina Barzilay, and Tommi S. Jaakkola. Particle guidance: non-i.idd. diverse sampling with diffusion models. International Conference on Learning Representations, 2024.
[12] Xiaoliang Dai. Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang. Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint arXiv:2309. 15807, 2023.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), 2009.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems (NeurlPS), 2021.
[15] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow. Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research (TMLR), 2023.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[17] Piero Esposito, Parmida Atighehchian, Anastasis Germanidis, and Deepti Ghadiyaram. Mitigating stereo- typical biases in text image generative systems. arXiv preprint arXt::2310.06900 2023.
[18] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[19] Dan Friedman and Adji Bousso Dieng- The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research (TMLR), 2022.
[20] Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-iman io-imaae generation models on faimesss. arXiv preprint arXiv:2302.1082 2023.

---

[21] Rinon Gal, Yuval Alaluf. Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. In Proceedings of the International Conference on Learning Representations (ICLR). 2023.
[22] William Gaviria Rojas, Sudnya Diamos, Keertan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. Advances in Neural Information Processing Systems (NeurlPS), 2022.
[23] Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana Romero Soriano. Dig in: Evaluating disparities in image generations with indicators for geographic diversity. Transactions on Machine Learning Research (TMLR), 2023.
[24] Melissa Hall, Samuel J Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano. Towards geographic inclusion the evaluation of text-to-image models. In Proceedings of the ACM Conference on Fairness, Accountabiliry, and Transparency, 2024.
[25] Reyhane Askari Hemmat, Mclissa Hall, Alicia Sun, Candace Ross, Michal Drozdzal, and Adriana Romero- Soriano. Improving geo-diversity of generated images with contextualized vendi score guidance. In Proceedings of IEEE/CVF European Conference on Computer Vision (ECCV), 2024.
[26] Reyhane Askari Hemmat, Mohammad Pezeshki. Florian Bordes. Michal Drozdzal, and Adriana Romero- Soriano. Feedback- guided data synthesis for imbalanced classification. Transactions Machine Learning Research (TMLR), 2024.
[27] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributin generalization. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[28] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[29] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[30] Jonathan Ho and Tim Salimans. Classifier-frce diffusion guidance. arXiv preprint arXiv:2207.1250
2022.
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[32] Susung Hong. Smoothed energy guidance: Guiding diffiusion models with reduced energy curvature of attention. Advances in Neurul Information Processing Systems (NeurIPS), 2024.
[33] Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
(34] Edward J Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[35] Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a had version of itself. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[36] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[37] Tuomas Kynkäääniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[38] Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv preprint arXiv: 10Xi:1889.090/1
2018.
[39] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[40] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.
[41] Muhammad Ferjad Naeem, Scong Joon Oh, Youngjung Uh. Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
12

---

[42] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2023.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
[44] Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: geographically diverse evaluation dataset for object recognition. Advances in Neural Information Processing Systems (NewrIPS), 2023.
[45] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. 10Xiv:2007.085000 2020.
[46] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? Proceedings of the International Conference on Machine Learning (ICML), 2019.
[47] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image symthesis with latent diffusion models. arxiv 2021. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[48] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Ktir Aberman. Dream- booth: Fine tuming text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[49] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Cads: Unleashing the diversity of diffusion models through condition-anncaled sampling. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[50] Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
[51] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[52] Vikash Sehwag. Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-densit regions using diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[53] Joonghyuk Shin, Minguk Kang, and Jaesik Park. Fill-up: Balancing long-tailed data with generative models. arXiv preprint arXiv:2306.07220 072000, 2023.
[54] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute optimally can be more elfective than scaling model parameters. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
[55] Jascha Sohl- Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Machine Learning (ICML), 2015.
[56] Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. Proceedings of the International Conference on Learning Representations (ICIR),, 2021.
[57] George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony I. Caterini, Erie Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems (NeurlPS), 2023.
[58] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[59] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2xX 2023.
13

---

[61] Patrick von Platen. Suraj Patil. Anton Lozhkov. Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. attps://gisee
com/ cww/ diggggiggaaae/fffeerrse.e 2022.
[62] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[63] Tong Wu, Liang Pan. Junzhe Zhang. Tai Wang, Ziwei Liu, and Dahua Lin. Balanced chamfer distance as comprehensive metric for point cloud completion. Advances in Neural Information Processing Systems (NeurlPS), 2021.
[64] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems (NeurlPS), 2023.
[65] Eric Zhang, Kai Wang, Xinggian Xu, Zhangyang Wang, and Humphrey Shi, Forget-me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2023.
[66] Yifan Zhang, Daquan Zhou, Bryan Hooi, Wang, Jiashi Feng. Expanding small-scale datasets with guided imagination. Advances in Neural Information Processing Systems (NeurIPS), 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
This work studies the use of Chamfer distance as guidance technique for generative models increase grounded diversity in their generation. Description of the approach is presented in Section 3, quantitative and qualitative evaluations w.r.f. previous works are presented Section 4.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The work includes a "Limitations" section, where we describe the limitations of automated metrics when dealing with distributional representation. We rely on previous benchmarks but recognize that a more complete evaluations should include human judges.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all the relevant implementation details in the Experimental section (Section 4) of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
We put an effort make the paper self contained to facilitate reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ public/guide Seuuides/CoeSSubmi ott.oodeesbmmssiooooooooo.ooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cccccc/ppooo cc/public//aee wewlcc/guii////ee Sttps//CooSuubmissce celtamiissooooooocoioo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We report used hyperparameters and data splits in the Experiment section when novel. For the geographical diversity setup we refer to hyperparameters and data splits of previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Although our approach enables the possibility of conducting multiple evalua- tions and statistical significance analyses, we follow previous works [25] and use a single seed to report results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss the type of resources needed run the experiments the Imple- mentation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification: We have carefully inspected the code of ethics, and believe our work conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper includes "Societal Impact" section, where we discuss the potential harm generative model in the wild could bring. Effective filters need to be put in place before these models are released to a broader audience, avoid the generation of harmful content.
Justification:
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We employ pre-trained diffusion models and use the safeguards put in place by the original authors. Additionally, we perform our analyses on non non-controversial datasets, such as ImageNet and geo-diversity datasets.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original creators of all models, datasets and algorithms used in this work are properly credited, with citations in the manuscript. We used their material for only non-commercial purpose of developing this research paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thodde com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new dataset with our submission.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study does not involve human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs have been used only for editing and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: fattps://neurre cw/ccc///ocoo/ceeccee c for what should or should not be described.
21

---

A Efficiency analysis
We analyze the computational overhead and efficiency gains of our proposed Chamfer guidance method compared to traditional Classifier-Fie Free Guidance (CFG) when applied to LDM1s and LDM3.5M. This analysis quantifies the significant computational savings achieved by our approach. We use pytorch's s FlopCounterKode to compute the FLOPs of each module.
A.1 Baseline Diffusion Model Computation
L. The standard diffusion process using L.M1,s requires 800GFI.OPs per forward pass.
With the default implementation using 40 denoising steps, the total computation for generating single sample amounts to 40 x 800 GFLOPs = 32 TFLOPs.
When applying CFG, which requires doubled batch size per step, the computational requirements double, 40 x 1.6 TFLOPs 64 TFLOPs.
LDM The standard diffusion process using LDM requires approximately TFLOPs per forward pass. With the implementation using 40 denoising steps, the total computation for generating a single sample amounts to 40 x 6 TFLOPs = 240 TFLOPs.
When applying CFG, which requires doubled batch size per step, the computational requirements double, 40 x 12 TFLOPs = 480 TFLOPs.
We also have to take into consideration the decoding of the final samples, which account for R 2TFlops for LDM1s,, and = 10TFlops for LDM3,5M-
Therefore we estimate 34 and 66 TFLOPs to generate an image without and with CFG for LDM1ss and 250 and 490 TFLOPs generate an image without and with CFG for LDM
A.2 Our Chamfer Guidance Approach
The Chamfer guidance method introduces two additional computational components:
Reference Image Processing We use DINOV2 (ViT/L) for feature extraction, which requires 160 GFLOPs per image. Given our largest reference set of k 32 real images, the total computation for reference embedding is: 32 x 160 GFLOPs z 5.1 TFLOPs. This is one-time cost incurred at the beginning of the sampling process.
During generation, we apply our guidance at frequency of Gfreg 5, meaning we compute the guidance every 5 steps. Each guidance computation using DINOv2 requires first decoding the samples, and then encoding in DINO space, therefore the cost for LDM,,s is (2TFLOPs + 160 GFLOPs) x 8 Rs 17.3 TFLOPs, while for LDM3.sM (10TFLOPs 160 GFLOPs)
81 TFLOPs.
For the entire sampling process with 40 steps, the reference encoding and the Chamfer guidance adds approximately x 22.4 TFLOPs for LDM and 86 TFLOPs for LDM3,5M-
The total computation required by our Chamfer guidance method therefore about 56.4 TFLOPs for LDM,s, and 336 TFLOPs for LDM
A.3 Efficiency Comparison
Comparing our method with the standard CFG approach:
Computation
Efficiency gain LDMI.
Computation
56.4 TFLOPs
(8)
66 TFLOPs
2 0.15 or 15%
(9)
22

---

Efficiency gain LDMa,5M = ComputatioO
(10)
Computation
336 TFLOPs
(11)
490 TFLOPs
2 0.31 or 31%
(12)
This analysis demonstrates that our Chamfer guidance method achieves superior generation quality while reducing the computational requirements by approximately 31% compared to traditional CFG on LDM This translates to 4s. to generate a sample with LDM on RTX A6000.
On the evaluation metrics
Through our experimentation, we identified several limitations in commonly used metrics for eval- uating generative diversity, leading us to choose the make the following choices to the evaluation framework:
From Recall to Coverage: We examine the of recall grounded diversity metric and find it to be highly sensitive to outliers in the generated distribution [41]. A single generated sample that happens to be far from the other generated points can disproportionately inflate the recall value. As an alternative, we advocate for the use of coverage, which provides more reliable assessment of how well the generated distribution matches the target diverse distribution. Coverage is more robust to outliers, as the manifold is computed w.z.t real points [41].
F, with precision and coverage: To evaluate both fidelity and diversity in single metric, we propose to modify the Fi score in previous works [25] to harmonically combine precision and coverage. We specifically chose precision over density [41] despite their similar purposes, precision is naturally bounded between 0 and 1, making it more intuitive to interpret and combine with other metrics. In contrast, density potentially unbounded, which complicates its use in composite metrics.
From Inception DINO space for evaluation: We assess the choice of feature space for computing Limilarity-based metries. While the Inception feature space has been the de facto standard in generative model evaluation, we advocate for the use of the DINO feature space instead, as supported by previous works [24, 57]. Self-supervised latent spaces better align with human perception of image realism, while Inception focuses on specific objects rather than holistic image features, often ignoring important aspects of images.
C
User Study
While the primary focus of our study the utility of the generated data for representation and as a training source for downstream tasks, which is confirmed by extensive experimental evaluation with downstream classifier training, we also evaluate human perception of our generated images.
To this end, we conduct a small-scale user study to complement our quantitative findings. We collected 965 data points from more than twenty anonymous annotators. In this study, users were presented with samples generated from prompts based the ImageNet dataset. Their task was to choose their preferred generation in a side-by-side comparison between images from the base LDM, model and images generated with our Chamfer Guidance applied to the same model. Users were also presented with real images from the dataset to ground their evaluation in real-world quality and coherence,
show in Figure 3.
The results showed a strong preference for our method: images generated with our Chamfer Guidance were preferred in 92% + 2% of the cases. This suggests that the automatic evaluation of quantitative improvements downstream utility also correlates with enhanced human-perceii quality and fidelity to the target concept distribution.
23

---

Q2: Which of the of generated images matches the style and diversity the real images shown the first row?
Generation
Generation
Generation
Generation
Figure 3: Example of user study question.
D Geographic diversity baselines
We employ the same baselines defined in c-V [25], which uses reference-fiee and eeference-based solutions. We report descriptions for them as follows.
Without any additional information:
LDM This is the baseline setup where the base LDM1,5 is used with the prompt {object} in {region} to condition the generation process.
Synonyms: This strategy maps each object class to its corresponding ImageNet [13] class and WordNet [40] synset. For each class, we generate images that cover all possible meanings (lemmas), including the original object word. This means that each synset, which groups lemmas for specific sense of the class, guides the image generation process through [synonym] in {region}.
With additional information:
LLaMA-2-1 --B------B---- 2-t-hhhtt [60] large language model is used to generate paraphrases of the original prompt template, {object) in {region) The authors include the specifications and descriptions used in the collection of GeoDE and DollarStreet. The metaprompts and paraphrases, as well as the method of tuning prompts and model specifications, are included in the original paper Supplementary material [25].
Paraphrasing:
Semantic Guidance: Generated images often exhibit diversity problems, as highlighted by previous works [23], due to amplification of region-specific object features that go beyond what is present the evaluation task. To mitigate this overemphasis on regional data, the authors utilized Semantic Guidance 20], which involves applying negative guidance corresponding to the regional term for each generated image.
Feedback Guidance (FG): Inspired by [26, 52], the authors employ an external CL.IP-based classifier [43] to provide feedback during image generation by predicting region labels of these

---

images. To promote greater diversity in the generated outputs, they investigate two feedback guidance strategies: the first maximizes the loss of the classifier, while the second maximizes the entropy of its predicted class distributions.
Textual Inversion: Textual Inversion is a technique used to teach text-to-image models new, specific visual concepts from just few example images. achieves this by creating a new "pseudo- word" or token in the model's vocabulary that becomes associated with the visual characteristics of the provided images. We apply textual inversion by learning an embedding for each object in the dataset using four images per object.
E
Additional geographic diversity results
In Table 5 we report the same experiments on GeoDE as in Table 2 of the main body, but with metrics in DINOv2 [42] space, which is known to align better with human evaluation [24, 57]. We observe state-of-the ticc-oftthh-artt results w.E.t relevant baselines, and in particular in terms of F1 by significant margin (+12%) over c-VSG [25], due to an increased precision (+26.7%) and coverage (+7.7%). The results reported here present different scale w.z.f. the object-centric scenario due to the & of the KNN computation of metrics, which we maintain equal to c- c-V benchmark, and set to 3, differently from the default value of 5 when using dgm-eval [57].
Table 5: Comparison on the GeoDE dataset under the geographical representation benchmark of c- c-VSG setting, with model selection Fi (i (P.C). Metrics computed in DINOv2 space. All the results are re-implemented. AF: Africa, AS: Asia, EU: Europe. "label" refers to region label, "desc" text description, and "img" exemplar images.
F, C
Precision
Coveruge
CLIPScore 1
Method
Ref. Info & Worst-Reg. Avg.
Worst-Reg.
Avg.
Worst-Reg.
Avg.
Worst-Reg,
Avg.
Worst-Reg.
LDM,s
AF
0.166
0.126
0.345
0.284
0.109
0.081
0.244
0.233
Guidance
AF
0.234
FG CLIP (Loes)
label
AF
0.186
0.151
0.339
0.308
0.129
0.103
0.216
0.211
PG CLIP (Entropy)
lahel
AF
0.178
0.133
0.328
0.272
0.122
0.088
0.238
0.296
e-VSG (25] (CLP))
AF
0.183
0.139
0.337
0.288
0.126
0.091
0.254
0,252
(25) (CLIP)
img
AF
0.181
0.259
0.127
Chamfer Guidance (Ours, DINOv))
img
2
AP
0.246
0.221
0.551
0.507
0.159
0.141
0.248
0.242
Chamler Guidance (Ours, DINOv2)
AF
0.304
0.292
0.605
0.564
0.204
0.197
0.250
0.244
In Table we present the results computed in Inception space for DollarStreet [22], using the Fi selection precision and coverage. Our Chamfer guidance outperforms c- c-VSG [25] by 3.5% in terms of F1 due to both an increased fidelity (precision) and diversity (coverage) of the generated samples. As previously reported for GeoDE, method exhibits an increased image-text alignment when using CLIP as the feature extractor for the guidance.
Table 6: Comparison the DollarStreet dataset under the geographical rew.esseeatteeeeeeetttttttiiiini representation benchmark of VSV setting, with model selection on FP C). Metrics are computed in Inception space. i indicates re- re-implemented results. AF: Africa, AS: Asia. EU: Europe. "label" refers to region label, "desc" to text description. and "img" to exemplar images. Our Chamfer guidance achieves tfateoof-th--ar in terms of F1, obtaining the best grounded diversity.
Cy
Precision
Coverage
CLIPSScoee
Method
Rd. Info A Werst-Reg.
Avg.
Worst Reg.
Avg.
Worst Reg.
Avg.
Worst Reg.
Avg.
Worst Reg.
L.M,s
AS
0.473
0.445
0.504
0.504
0.447
0.308
0.249
0.255
Symonyms
AS
0.445
0.635
0.448
0.439
0.443
0.432
0.216
0.219
VSG [25]
0.424
0.404
0:421
0.419
0.428
0.300
0.195
0.194
Paraphrasing
desc
AF 0.440
0.438
0.445
0:451
0.436
0.422
0.226
0.215
Semantic Guidance'
label
AS 0.473
0.436
0.504
0.507
0.447
0.398
0.249
0.255
PG CLIP (Loss)
label
AS
0.408
0.429
0.481
0.461
0.458
0.401
0.243
0.244
FG CLIP (Entopyy
hbdd
AS
0.168
0.440
0.457
0.300
0.453
0.393
0.245
0.253
Textual Inversion'
4
AS
0.076
0.037
0.505
0-491
0.042
0.019
0.213
0.216
VSG [25] (CLIP)
ing
4
BU
0.517
0.504
0.510
0:481
0.526
0.529
0.241
0.241
Chamfer Guidance (Ours, CLIP)
ing 4
AS
0.528
0.512
0.515
0.514
0.542
0.510
0.244
0.245
Chamfer Guidance (Ouns, DINOv2)
img
4
AS
0.552
0.532
0.545
0.542
0.560
0.520
0.240
0.211
In Table and Table we report the result with selection of the Fj score between precision and recall, as in the original e- V-V benchmark. Although recall does not represent the best metrie for evaluating
25

---

grounded diversity [41], our Chamfer guidance surpasses c- VSG by 3.9% in terms of recall in GeoDE, and by 2.7% on DollarStreet, leading state-of-the tfrttte-rttttt Fi in both datasets. Interestingly, when using recall, grounding the generation on more samples does not lead to increased diversity with CLIP, and to a marginal one with DINOv2. This was the case when using coverage, and we consider these results to be result of the inflated recall manifold, which promotes outlier generated images.
Table 7: Comparison on the GeoDE dataset under the geographical representation benchmark of c-VSG setting, with model selection on Fi (P, R). Metries computed in Inception space. AF: Africa, AS: Asia, EU: Europe. "label" refers to region label, "dese" to text description, and "img" to exemplar images.
F, Ry
Precision
Recall
CLIPScore
Method
Ref. Info & Worst-Reg.
Avg.
Worst Reg.
Avg.
Worst Reg.
Avg.
Worst Reg.
Avp.
Worst Reg.
LDM,3
AF
0.364
0.322
0.411
0:273
0.337
0.395
0.242
0.218
Synonyms
AP
0.306
0.350
0.201
0.366
0.315
0.215
0,203
VSG (25)
0.399
0.349
0.307
0.424
0.180
0.191
Paraphrasing
desc
WAS
0.384
0.354
0.338
0.309
0.449
0.415
0.231
0.228
Semantic Guidance
label
WAS
0.420
0.401
0.459
0.519
0.391
0.326
0.245
0.253
CLIP
PG CLIP (Entrogy)
label
AP 0.380
0.337
0.340
0.329
0:429
0.345
0.224
0.227
Textual Inversion
img
4
AF
0.369
0.363
0.409
0.464
0.338
01.088
0.234
0.252
eVSG [25) (CLIP)
2
AF
0.455
0.401
0.421
0:417
0.493
0.476
0.254
0,253
Chamfer Guidance (Ours, CLIP)
ing
2
AF
0:454
0:6:0
0.398
0:401
0.532
0.489
0.245
0.245
Chamfer Guidunce (Ours, CLIP)
ing
AFF
0.463
0.449
0.427
0.413
0.509
0.492
0.251
0.246
Chamfer Guidance (Ours, DINOv2)
2
AF
0.451
0.435
0.402
0.470
0.457
0.235
0.225
Chamfer Guidance DINOv2)
img
Table Comparison on the DollarStreet dataset under the geographical representation benchmark of c- C-V setting, with model selection (P, R).. Metries are computed in Inception space. indicates re-implemented results. AF: Africa, AS: Asia, EU: Europe. "label" refers region label, "dese" to text description, and "img" to exemplar images.
F, (.. R)
Precision
Recall
f
CLIPScore ?
Method
Re.. Info A Worst-Reg.
Asg.
Worst Reg.
Avg.
Worst Reg.
Avg.
Worst Reg.
Avg.
Worst Reg.
L.DMs
A8
D.248
0.442
0.428
D:434
0.472
0.451)
0.231
0.235
Symonyms
AS
0.464
0.457
0.451
0.448
0.461
0.467
0.216
0.220
VSG (25)
0.457
0.444
0:413
0:388
0.516
0.518
0.191
0.198
Paraphrasing
dese
AP
0.454
0.445
0.445
0.454
0.465
0.437
0.226
0.215
Semantic Guidanze
label
AS
0.270
0.458
0.487
0:449
0.467
0.467
0.290
0.233
PG CLJP (Loss)
label
AS
0.394
0.401
0.321
0.488
0.510
0.223
0,206
(Eatrogy)
Textual Inversion
img
4
AS
0.425
0.398
0.478
0.491
0.386
0.335
0.217
0.219
VSG (CLIP) (25)
4
0.497
0.483
0.486
0.486
0.511
0.479
0.234
0.238
Chamfer Guidance (Ours, CLIP)
ing
AF
D-102
0:478
0.464
0:458
0.524
0.498
0.231
0.225
Chamfer Guidanee (Ours, DINOv2)
mmg
AF
0.508
0.484
0.482
0.483
0.538
0.486
0.2255
0.215
F Additional ablation
y ablation. This ablation examines the performance using different Chamfer guidance strengths (r). The results in Table show that this hyperparameter significantly impacts model performance, with y = 0.07 and & = 32 achieving the best overall results. This configuration yields the highest Fi score (0.931) and precision (0.950), indicating superior fidelity and diversity. Interestingly, while stronger guidance generally improves these metrics, there is trade-off with image quality, the FID score increases from 8.840 at 7 0.05 to 13.670 7 0.07. This suggests that while stronger Chamfer guidance (7 = 0.07) produces more accurate samples, it somewhat compromises the distribution statisties compared the moderate guidance setting (7 0.05), which maintains a better balance between precision/coverage performance and image quality. This might also be due to the use of the Inception network for the computation of FID instead of DINO.
Feature extractor ablation. To further validate the choice of DINOv2, conducted preliminary empirical study comparing its performance as a feature extractor for performing Chamfer guidance against CLIP the GeoDE dataset in an "object-centrie" setting, e.g., with prompts like "a photo of a car. These experiments were run using LDM1,5.
Our findings as follows:
26

---

Table 9: Chamfer guidance strength y ablation on ImageNet-Ik using LDM Our Chamfer guidance obtains the best coverage results when using high (y = 0.07) strength, but obtains the most balanced
image quality (FID) when using milder amount (7 0.05).
7
k
F1 (P; C)
Precision f Coverage f Density f
Recall
FDD 4
FID L
0.02
2
0.837
0.896
0.786
0.910
0.702
145.390
9.268
0.02
32
0.872
0.912
0.835
0.994
0.699
138.917
9.301
0.05
2
0,881
0.932
0.835
1.051
0.637
124.191
8,840
0.05
32
0.914
0.946
0.884
1.162
0.650
114.847
8,906
0.07
2
0.886
0.947
0.833
1.108
0.480
156.179
13.670
0.07 32
0.931
0,950
0.912
1.213
0.649
113,301
8,935
DINOv2 consistently yielded the best performance, showing substantial gains in both diversity and fidelity. Similarity in DINOv2 space also correlates to human- numan--erceived similarity slightly more than CLIP, as reported in [24].
CLIP improved upon the baselines, and in particular, coverage scales with the number of guiding samples. With higher & we observe reduced marginal improvements, and we deem this the fact that the model tended converge towards generating an average representation of the object. We hypothesize this is because CLIP's pre-training is "concept- centric" (aligning images to general text concepts), whereas DINOv2's is "instance-centric" due to its self-supervised training, making it better at preserving the unique features of specific reference image.
We report these results Table This analysis confirms that DINOv2 the most effective choice for our method, with CLIP being an alternative.
Table 10: Comparison of feature extractors across different & values. DINOv2 shows strong improve- ments in Fi, precision, and coverage compared to LDM and CLIP, while also achieving lower FID.
Feature Extractor
k
FI(P, C)
Precision
Coverage
Density
Recall
FDD
FID
LDM1s 1.0
0.2334
0.4363
0.1593
0.1731
0.6135
684.81
35.27
LDMis 2.0
0.3277
0.5433
0,2346
0.2647
0.4859
524.59
24.04
LDM1s w 7.5
0.2960
0.6222
0.1942
0.3629
0.2025
693.31
42.60
DINOv2
2
0.4242
0.6354
0.3184
0.4027
0.4614
410.84
19.73
DINOv2
4
0.3313
0.8296
0.3907
0,8933
0.1501
368.27
19.08
DINOv2
8
0.6251
0.9010
0.4785
1.3575
0.0708
353.80
18.81
DINOv2
16
0.7527
0,9461
0.6250
2.0309
0.0547
323.11
18.44
CLIP
2
0.3888
0.6367
0,2798
0.3987
0.3604
446.77
19.66
CLIP
4
0.4063
0.5951
0.3084
0.3501
0.4827
421.20
19.01
CLIP
8 0.4163
0.5974
0.3195
0.3567
0.5030
416.75
18.25
CLIP
16
0.4088
0.5757
0.3169
0.3180
0.5735
415.02
18.22
G Additional object-centric results
In Table 11 we show more results of base LDM1s and LDM3s5M sampling, and additional set of parameters for CADS [49] and APG [50]. These results extend those presented in Table 1, and show how these reference-free approaches cannot significantly increase the fidelity and diversity of the samples, and that applying APG on a small w brings moderate improvement over the best base sampling, for both LDM,, and LDM3,sM-
H Additional classification results
We complement our downstream utility analysis by evaluating the downstream utility of images generated using our Chamfer Guidance. We follow the "static" ImageNet-100 100 setup introduced in [4], and generate dataset of 130, 000 synthetic images using our Chamfer Guidance, using 32 real exemplar images from the training set. Each synthetic image is generated with simple prompt (class name). We report the accuracy of a ViT-B [16] classifier trained on this synthetic data and tested on real validation data. In Table 12 compare to the base performance of several LDMs, and
27

---

Table 11: Comparison of different LDM configurations with APG and CADS guidance on ImageNet-
1k.
Method
APG
CADS
C) Precision
Coveraget
Densityt
FDD1 FID
LDM 1.5
LDM 1.5
1.0
0.507
0.723
0,391
0.551
431.2
31.3
LDM 1.5
2.0
0.673
0,802
0,580
0.648
226.2
10.8
LDM 1.5
7.5
0.709
0.862
0.603
0.775
248.7
16.1
LDM 1.5
X
2.0
0.677
0.809
0.582
0.644
226.0
10.8
LDM 1,5
X
4.0
0.723
0.855
0.626
0.752
217.9
13.4
LDM 1,5
X
7.5
0.713
0.853
0.612
0.768
247.9
16.2
LDM 1.5
X
10.0
0.707
0.858
0.601
0.766
261.2
17.0
LDM 1.5
V
0.487
0.713
0.370
0.526
431.4
31.2
LDM 1,5
V
2.0
0.676
0.806
0.582
0.641
226.5
10.9
LDM 1,5
V
4.0
0.718
0.850
0.621
0.743
218.0
13.4
LDM 1.5
7.5
0.715
0.856
0.615
0.771
247.0
16.1
LDM 1.5
10.0
0.707
0.857
0.601
0.770
260.8
16.9
LDM 3.5
LDM 3.5
1.0
0.599
0.752
0,498
0.560
314.7
17.3
LDM 3.5
2.0
0.727
0.872
0.623
0.797
231.9
15.7
3.5
7.5
0.690
0.568
0.829
329.6
24.4
LDM 3.5
X
2.0
0.723
0.856
0.625
0.783
237.6
15.4
LDM 3,5
X
4.0
0.722
0.877
0.613
0.832
287.4
21.4
LDM 3.5
X
7.5
0.695
0.874
0.576
0.820
334.7
23.7
LDM 3.5
10.0
0.681
0.870
0.559
0.798
349.8
23.5
LDM 3,5
V
1.0
0.582
0.751
0.475
0.529
346.6
19.9
LDM 3,5
X
V
2.0
0.717
0.851
0.620
0.749
238.1
15.1
LDM 3.5
V
4.0
0.712
0.870
0.603
0.807
282.3
20.9
LDM 3.5
V
7.5
0.688
0.868
0.570
0.799
328.7
23.2
LDM 3.5
10.0
0.679
0.866
0.559
0.778
345.6
23.1
observe substantial gains when using our Chamfer Guidance. Interestingly, the initial performance gap of 6.5 points between LDM1, and LDM (presumably because reduced sample diversity in LDM sM) reduced to 2.4 points when using Chamfer Guidance. These results confirm the ability of Chamfer Guidance improve downstream utility of synthetic data.
Table 12: Validation accuracy real images of classifiers trained with synthetic data for ImagetNet- 100 classes. All models are trained for 50k iterations.
LDM,,
LDM21
LDMxt.
LDM
IDM,,s + Chamfer
LDMAsM
LDMass + Chamfer
Real Val. Acc.
59.06
55.92
52.8
59.24
67.82
52.72
65.42
I Licenses
We report the licenses for datasets and models used in Table 13.
J Additional qualitative results
Figure shows LDM,,s with w = 1.0 without and with our Chamfer guidance, and w = 2.0 with Chamfer guidance. LDM,, does not have "strong" conditional only model (w 1.0), as quantitatively highlighted in Table I1, which exhibits poor coverage and density, and high FID. Qualitatively, we observe bad-looking images that often lack the correct subject, present distorted one. Our Chamfer guidance, even when applied only the conditional model, can effectively recover the correct texture, shape, and proportions of the subject. For LDM the best results obtained by applying our Chamfer guidance on top of low w value, e.g., 2.0.
Figure shows that LDM3.sM without CFG, iee,, = 1.0 produces poor quality images, often with distorted wrong subjects. can appreciate that Chamfer distance, applied with moderate strength y, can effectively steer the generation towards the right subject, increasing the quality and diversity
28

---

Table 13: Links and licenses for the datasets and pre-trained models use our study.
Name
Link
License
ImageNet
httpa://vwa. wewgu.ingeeeeee .arg org
Apache License 2.0
GooDE
stsp:://geoaleera---aa------l-l- coleetatoollllioon..oooo c8. princeton, edu/
CC BY 4.0
DollarStreet
street/
CC-BY-SA 4.0
DINOv2
https:
Apache License 2.0
CL.IP
http:///gttabb.ccomoppee///////
MIT License
Llama2-706
hitpe htgg:://uugglggfeeeeooo/ddd--lia
Llama Community License Agreement
LDM
httpe: htape://bugginggfae.com http:///.......eeoo/////////aaaa
v1-5
CreativeML Open RAIL-M
LDM
http:///wgggggggseecccco//raabls http:///a/arrabbbooo..uoeeeeoo - 3.6-mediun
Slubility Al Community License
dgm-eval
httpa://gtthub.com http:///gtttbbb.cc///yrrrraaaaaa labs///g http://////oooraaaaaaaaaaooo/a//
of the generation. When using high CFG strength w i.e., 7.3, steering the generation with our Chamfer guidance requires higher strength before becoming effective. Our Chamfer guidance helps to recover naturalness of the image by reducing oversaturation, and to increase the variety in the backgrounds.
Figure 6 qualitatively shows the effect of increasing the number of exemplar images & when using our Chamfer guidance. We can see greater diversity in both subjects and background when using higher number for k; in agreement with the quantitative results presented in Figure 2.
In Figure 7 show the comparison of the geographic diversity benchmark, in particular the GeoDE dataset. We show how base generations using LDM1s introduce stereotypical elements when using the geographical indicator, e.g., dilapidated cars, rusty pans. c- C-G [25] partially mitigates the issue, but comes at the expense of quality, generating oversaturated images that do not resemble the dataset. Our Chamfer guidance effectively mitigates both issues, by generating natural-looking images without stereotypical elements, due to the guidance of real exemplars.
29

---

1.0 (No Chanker
1.0
Datesic
CMA CEN
Figure 4: LDM,s generations ImageNet- Ik with different values. h 32, 7 0.07 for our Chamfer guidance. The classes are from top to bottom: container ship, pelican, brambling. and dutch oven.
30

---

0
Y= 0.005
Y 0.01
yy 0.02
1.
7.5
111 10
Figure 5: LDM3.sM on ImageNet- Ik with different and gamma values, À 32. The classes
from top to bottom: Irish wolfhound and hamster
31

---

16
32
Figure 6: IDM ImageNet- Ik with different k values. and y picked based on the best F score. The classes from left to right are: sports car, golden retriever, and goose.
32

---

(a) {object) in (region} baseline.
C-SG [25] generations.
(c) Our chamfer guidance.
(d) Samples from the GeoDE dataset.
Figure 7: Generated images and examples from the GeoDE dataset cars (left) and cooking pots (right) using LDM GeoDE. Colors indicate images in Africa and Europe. Our Chamfer Guidance exhibits better-looking images with less saturated colors, increased subject quality and diversity in the backgrounds.
33


Reviews:
[{"review_id": "T6SNpPEZ2T", "reviewer": "Reviewer_mRsF", "summary": "The paper introduces Chamfer Guidance, a novel training-free method to improve the utility of synthetic images generated by conditional diffusion models. By leveraging a small set of real exemplar images, it uses the Chamfer distance to guide the generation process toward better fidelity and diversity with respect to real data distributions. The method achieves state-of-the-art performance across object-centric and geo-diversity benchmarks and significantly boosts downstream classifier performance using only synthetic training data. Additionally, Chamfer Guidance avoids the computational overhead of classifier-free guidance, offering clear FLOPs reduction while maintaining or improving image quality and diversity.", "questions": "Include a comparative analysis of multiple embedding spaces, although DINOv2 is well-motivated due to its self-supervised learning and semantic alignment.\n\nIncluding a small-scale user study or human preference ranking would help validate whether the improvements measured by Chamfer Guidance align with human judgment.\n\nThe paper includes ablations on CFG scale, but does not fully study the effect of the Chamfer guidance strength. A sensitivity analysis across a range of γ values could demonstrate whether the method needs careful tuning per task or dataset, or if it is relatively stable.", "limitations": "Limitations are briefly discussed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The paper presents a novel training-free guidance method that effectively enhances the utility of synthetic images by balancing fidelity and diversity through Chamfer distance. It achieves state-of-the-art performance across both object-centric and geo-diversity benchmarks while significantly reducing computational cost compared to classifier-free guidance. The method scales well with the number of exemplar images and improves downstream classification accuracy using only synthetic data. Its theoretical motivation, practical utility, and clean experimental design make it a strong contribution to synthetic data generation.\n\nWeaknesses: \n1. DINOv2 serves the main embedding space, while the paper does not compare different feature extractors (e.g., CLIP, Inception) to quantify how sensitive Chamfer Guidance is to the underlying embedding space.\n2. No Human Evaluation: All evaluations rely on automated metrics, which may not fully capture perceptual quality or semantic coherence.\n3. There is limited discussion on the sensitivity of Chamfer Guidance strength and how tuning affects trade-offs between diversity and fidelity.", "was_revised": true, "final_justification": "The rebuttal resolves most of my concerns. I upgrade my rating accordingly.", "rebuttal": "We thank the reviewer for finding our work **a novel training-free guidance method that effectively enhances the utility of synthetic images** that **achieves state-of-the-art performance across both object-centric and geo-diversity benchmarks while significantly reducing computational cost**, and that **theoretical motivation, practical utility, and clean experimental design make it a strong contribution to synthetic data generation**.\n\n## [Q1] Embedding space\n\nFor the **“Geographic Diversity”** setting (Table 2, and Table 6 and 7 in the Appendix), we explored DINOv2 and CLIP as feature spaces for our guidance, as DINOv2 and CLIP feature extractors correlate better with human judgment of similarity than Inceptionv3 [1]. Both led to state-of-the-art results over prior works, with DINOv2 leading to better results than CLIP. \nTo further validate our choice of DINOv2, we conducted a preliminary empirical study comparing its performance as a feature extractor for performing Chamfer guidance against **CLIP** on the GeoDE dataset in an \"object-centric\" setting, e.g., with prompts like “a photo of a car.”. These experiments were run using LDM1.5.\n\nOur findings were as follows:\n- **DINOv2** consistently yielded the best performance, showing substantial gains in both diversity and fidelity. Similarity in DINOv2 space also correlates to human-perceived similarity slightly more than CLIP, as reported in [1].\n- **CLIP** improved upon the baselines, and in particular coverage scales with the number of guiding samples. With higher k we observe reduced marginal improvements, and we deem this to the fact that the model tended to converge towards generating an average representation of the object. We hypothesize this is because CLIP's pre-training is **\"concept-centric\"** (aligning images to general text concepts), whereas DINOv2's is **\"instance-centric\"** due to its self-supervised training, making it better at preserving the unique features of a specific reference image. T\nWe have included a summary of these results in the table below. This analysis confirmed that DINOv2 was the most effective choice for our method, with CLIP being an alternative. We will include these results in the final version of the manuscript. \n\n| Feature Extractor    | k   | F1     | Precision | Coverage | Density | Recall | FDD    | FID   |\n| :------------------- | :-- | :----- | :-------- | :------- | :------ | :----- | :----- | :---- |\n| LDM 1.5 $\\omega=1.0$ | --  | 0.2334 | 0.4363    | 0.1593   | 0.1731  | 0.6135 | 684.81 | 35.27 |\n| LDM 1.5 $\\omega=2.0$ | --  | 0.3277 | 0.5433    | 0.2346   | 0.2647  | 0.4859 | 524.59 | 24.04 |\n| LDM 1.5 $\\omega=7.5$ | --  | 0.2960 | 0.6222    | 0.1942   | 0.3629  | 0.2025 | 693.31 | 42.60 |\n| DINOv2                 | 2   | 0.4242 | 0.6354    | 0.3184   | 0.4027  | 0.4614 | 410.84 | 19.73 |\n| DINOv2                 | 4   | 0.5313 | 0.8296    | 0.3907   | 0.8933  | 0.1501 | 368.27 | 19.08 |\n| DINOv2                 | 8   | 0.6251 | 0.9010    | 0.4785   | 1.3575  | 0.0708 | 353.80 | 18.81 |\n| DINOv2                 | 16  | 0.7527 | 0.9461    | 0.6250   | 2.0309  | 0.0547 | 323.11 | 18.44 |\n| CLIP                 | 2   | 0.3888 | 0.6367    | 0.2798   | 0.3987  | 0.3604 | 446.77 | 19.66 |\n| CLIP                 | 4   | 0.4063 | 0.5951    | 0.3084   | 0.3501  | 0.4827 | 421.20 | 19.01 |\n| CLIP                 | 8   | 0.4163 | 0.5974    | 0.3195   | 0.3567  | 0.5003 | 416.75 | 18.25 |\n| CLIP                 | 16  | 0.4088 | 0.5757    | 0.3169   | 0.3180  | 0.5735 | 415.02 | 18.22 |\n\n## [Q2] User Study\nWhile the primary focus of our study is the **utility** of the generated data for representation and as a training source for downstream tasks, which is confirmed by our extensive experimental evaluation with downstream classifier training, we agree that human evaluation provides valuable insights.\n\nTo that end, we conducted a small-scale user study to complement our quantitative findings. We collected 965 data points from more than twenty annotators. In this study, users were presented with samples generated from prompts based on the ImageNet dataset. Their task was to choose their preferred generation in a side-by-side comparison between images from the base LDM3.5M model and images generated with our Chamfer Guidance applied to the same model. Users were also presented with real images coming from the dataset to ground their evaluation in real-world quality and coherence.\n\nThe results showed a strong preference for our method: images generated with our **Chamfer Guidance were preferred in 92% ± 2% of the cases**. This suggests that the automatic evaluation of quantitative improvements in downstream utility also correlates with enhanced human-perceived quality and fidelity to the target concept distribution. We will include these results and details of the study in the revised manuscript.\n\n## [Q3] $\\gamma$ ablation\nWe would like to direct the reviewer to our detailed ablation study in **Appendix E**, where we analyze this point on the ImageNet1k dataset with LDM1.5. Our results show that our method is robust across different values of $\\gamma$. In fact, for all tested strengths, our method **surpasses the LDM1.5 baseline** in terms of F1 score, demonstrating its consistent effectiveness.\n\nOur analysis reveals a trade-off between guidance strength and perceptual quality as measured by FID:\n- **Optimal Performance**: The configuration with $\\gamma=0.07$ and k=32 achieves the best overall results, yielding the highest F1 score (0.931) and precision (0.950). This indicates that stronger guidance produces samples with increased fidelity and diversity.\n- **Performance Trade-off**: While stronger guidance boosts precision and coverage, it can slightly degrade the FID. For example, increasing $\\gamma$ from 0.05 (FID: 8.840) to 0.07 (FID: 13.670) shows this effect. The moderate guidance maintains a better balance between precision/coverage and overall image quality.\n    \nWe hypothesize that this increase in FID with stronger guidance might be partially due to a **metric mismatch**. The FID metric is computed using features from the **Inception** network, which is optimized for classification. Our guidance, however, operates in the **DINOv2** feature space, which is richer in instance-level semantics. This discrepancy may lead the Inception-based FID to penalize valid, high-fidelity generations that are perfectly aligned in the DINO space.\n\nWe report the table below.\n\n| $\\gamma$ | k   | F1​   | Precision | Coverage | Density | Recall | FDD     | FID    |\n| -------- | --- | ----- | --------- | -------- | ------- | ------ | ------- | ------ |\n| 0.02     | 2   | 0.837 | 0.896     | 0.786    | 0.910   | 0.702  | 145.390 | 9.268  |\n| 0.02     | 32  | 0.872 | 0.912     | 0.835    | 0.994   | 0.699  | 138.917 | 9.301  |\n| 0.05     | 2   | 0.881 | 0.932     | 0.835    | 1.051   | 0.637  | 124.191 | 8.840  |\n| 0.05     | 32  | 0.914 | 0.946     | 0.884    | 1.162   | 0.650  | 114.847 | 8.906  |\n| 0.07     | 2   | 0.886 | 0.947     | 0.833    | 1.108   | 0.480  | 156.179 | 13.670 |\n| 0.07     | 32  | 0.931 | 0.950     | 0.912    | 1.213   | 0.649  | 113.301 | 8.935  |\n\n—-\n\n[1] Hall M., et al. Towards Geographic Inclusion in the Evaluation of Text-to-Image Models. In ACM FAccT 2024."}, {"review_id": "O5qHZI3rlg", "reviewer": "Reviewer_tTXY", "summary": "This paper proposes a novel inference-time method to guide image generation using a small set of real exemplar images. They compute the Chamfer distance between features of generated images and a few real reference images, and use the gradient of this distance as a guidance signal during the diffusion sampling process. Empirical results on ImageNet-1k and GeoDE/DollarStreet demonstrate strong performance.", "questions": "Why did the authors choose Chamfer distance specifically as the utility metric? What were the other choices they considered but decided not to pursue?\n\nDid the authors study how results vary with different random selections of exemplars?\n\nIf the exemplars themselves are from a distribution somewhat shifted from the model’s training distribution, how robust is the guidance?\n\nThe results in Table 1 use a guidance scale that differs for LDM1.5 and LDM3.5M. How was this guidance weight selected?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\nThe proposed method delivers a clear advance in generation utility, yielding simultaneously higher diversity and fidelity than strong baselines.\n\nThe idea of using Chamfer distance as a guidance reward is a conceptually elegant alternative to previous approaches like c-VSG that required maintaining a memory bank and carefully tuning two competing diversity metrics.\n\nThe paper validates the method on both object-class generation and geographic diversity, and uses extensive metrics; Precision, Recall, Density, Coverage, F1, FID, FDD, to assess both quality and diversity.\n\n[Weaknesses]\nThe paper does not deeply analyze sensitivity to which exemplars are used, so we don’t know how robust the method is to different choices or outliers in the exemplar set.\n\nGiven the stochastic nature of diffusion sampling, it’s unclear how robust the reported gains are across multiple runs or different random seeds.\n\nThe method is tailored to scenarios where a target distribution is available as reference, which is a slightly specialized use case in practice.\n\nThe proposed method adds overhead by requiring feature extraction and backpropagation through a vision transformer multiple times during sampling.\n\nThe approach fundamentally relies on pretrained feature extractors like DINO to measure quality and diversity, so it inherits the biases or limitations of those features.\n\nIn zero-shot generation scenarios or when one truly has no real data, Chamfer Guidance cannot be applied.", "was_revised": true, "final_justification": "The authors have sufficiently addressed my initial concerns in their rebuttal. I am updating my score accordingly.", "rebuttal": "We thank the reviewer for highlighting that ours is a **novel inference-time method to guide image generation using a small set of real exemplar images**, that **delivers a clear advance in generation utility, yielding simultaneously higher diversity and fidelity than strong baselines**, and that **the idea of using Chamfer distance as a guidance reward is a conceptually elegant alternative to previous approaches**.\n\n## [W1,2 Q2] Sample selection and stochastic process\nRegarding the sample selection, the reference samples were chosen **at random** from the dataset without a specific sampling strategy. This approach is standard and consistent with prior works like c-VSG [2]. While exploring advanced sampling techniques is interesting for future research, our focus is on the novel generation method itself. Our experiments confirm that even with random sampling, our method produces high-quality, useful images. \nTo provide empirical evidence, we ran an additional experiment on the GeoDE dataset using two different random seeds to select exemplar images. As the table below demonstrates, the **standard deviation across runs is negligible**.\n\nConcerning the stochastic nature of the diffusion process, we mitigate this through the scale of our experiments. By generating large numbers of images (e.g., 50,000 for ImageNet-1k), we ensure robust metrics that represent stable averages.\n\n| k   | F1               | Precision        | Coverage         | Density          | Recall           | FDD             | FID           |\n| :-- | :--------------- | :--------------- | :--------------- | :--------------- | :--------------- | :-------------- | :------------ |\n| 2   | 0.4399 ± 0.0263 | 0.6782 ± 0.0285 | 0.3256 ± 0.0222 | 0.5017 ± 0.0676 | 0.3562 ± 0.0221 | 405.36 ± 9.49  | 19.65 ± 0.13 |\n| 4   | 0.5421 ± 0.0189 | 0.8342 ± 0.0086 | 0.4016 ± 0.0191 | 0.9229 ± 0.0514 | 0.1490 ± 0.0057 | 362.08 ± 10.74 | 18.90 ± 0.31 |\n| 8   | 0.6361 ± 0.0191 | 0.9065 ± 0.0095 | 0.4900 ± 0.0199 | 1.3736 ± 0.0279 | 0.0709 ± 0.0054 | 349.47 ± 7.61  | 18.76 ± 0.09 |\n| 16  | 0.7598 ± 0.0099 | 0.9481 ± 0.0028 | 0.6339 ± 0.0126 | 2.0409 ± 0.0142 | 0.0592 ± 0.0064 | 326.16 ± 5.32  | 18.46 ± 0.08 |\n\n## [W3] Specialized use case\nWe respectfully disagree that our work is a \"specialized use case.\" Our work addresses a well-established and critical problem in generative modeling.\n\nThe problem of adapting a generative model to a target distribution given few reference examples is a significant problem. This setting is foundational to:\n- **Few-shot Adaptation:** Common for tasks like data synthesis for imbalanced classification [1, 2]\n- **Personalization:** Personalizing large text-to-image models for subject-driven generation, as in Textual Inversion [3], DreamBooth [4], and SVDiff [5]\n\nMore broadly, our work is an instance of **In-Context Learning (ICL) for diffusion models**. ICL is a cornerstone of modern LLMs [6], and its application to generative vision models is a critical emerging direction [7].\n\nOur primary contribution is an approach that is both effective and highly efficient. We are among the first to successfully adapt large diffusion models via ICL **without requiring any model finetuning**, achieving state-of-the-art results with few reference samples.\n\n## [W4] Computational overhead\nOur Chamfer Guidance can be **significantly more computationally efficient** than standard sampling with Classifier-Free Guidance (CFG), as analyzed in **Appendix A**.\n\nStandard CFG sampling requires a **double forward pass** through the model at each diffusion step, doubling the floating-point operations (FLOPS). When applied to conditional-only models (CFG scale = 1), our guidance mechanism is more efficient. With the LDM3.5M model, our method achieves **up to a 30% reduction in FLOPS** while reaching state-of-the-art results.\n\n## [W5] Feature extractor\nFor the **“Geographic Diversity”** setting (Table 2, and Table 6 and 7 in the Appendix), we explored DINOv2 and CLIP as feature spaces for our guidance. Both lead to state-of-the-art results over prior works, with DINOv2 leading to better results than CLIP.\n\nWe then chose **DINOv2** also for the other settings because of its **self-supervised learning** pre-training, which learns rich, instance-level visual features more suitable for our guidance task than features from classification models (Inception) or image-text alignment models (CLIP).\n\nWe validated this choice empirically by testing DINOv2 and CLIP on the GeoDE dataset. Our findings confirm that **DINOv2 provides superior performance**, with CLIP bringing improvements w.r.t. baselines as well.\n\n| Feature Extractor    | k   | F1     | Precision | Coverage | Density | Recall | FDD    | FID   |\n| :------------------- | :-- | :----- | :-------- | :------- | :------ | :----- | :----- | :---- |\n| LDM 1.5 $\\omega=1.0$ | --  | 0.2334 | 0.4363    | 0.1593   | 0.1731  | 0.6135 | 684.81 | 35.27 |\n| LDM 1.5 $\\omega=2.0$ | --  | 0.3277 | 0.5433    | 0.2346   | 0.2647  | 0.4859 | 524.59 | 24.04 |\n| LDM 1.5 $\\omega=7.5$ | --  | 0.2960 | 0.6222    | 0.1942   | 0.3629  | 0.2025 | 693.31 | 42.60 |\n| DINOv2                 | 2   | 0.4242 | 0.6354    | 0.3184   | 0.4027  | 0.4614 | 410.84 | 19.73 |\n| DINOv2                 | 4   | 0.5313 | 0.8296    | 0.3907   | 0.8933  | 0.1501 | 368.27 | 19.08 |\n| DINOv2                 | 8   | 0.6251 | 0.9010    | 0.4785   | 1.3575  | 0.0708 | 353.80 | 18.81 |\n| DINOv2                 | 16  | 0.7527 | 0.9461    | 0.6250   | 2.0309  | 0.0547 | 323.11 | 18.44 |\n| CLIP                 | 2   | 0.3888 | 0.6367    | 0.2798   | 0.3987  | 0.3604 | 446.77 | 19.66 |\n| CLIP                 | 4   | 0.4063 | 0.5951    | 0.3084   | 0.3501  | 0.4827 | 421.20 | 19.01 |\n| CLIP                 | 8   | 0.4163 | 0.5974    | 0.3195   | 0.3567  | 0.5003 | 416.75 | 18.25 |\n| CLIP                 | 16  | 0.4088 | 0.5757    | 0.3169   | 0.3180  | 0.5735 | 415.02 | 18.22 |\n\n## [W6] Zero-shot scenario\nOur method is designed for a **few-shot setting** with reference data to guide generation, related to **In-Context Learning (ICL)**. However, we find extending to a data-free pipeline interesting for future work. We could envision a **self-bootstrapping** technique:\n1. Generate initial candidate images for a given class\n2. Automatically select a diverse subset maximizing \"diameter\" (coverage) in a robust feature space\n3. Use this synthetically-generated set as guidance exemplars\n\n## [Q1] Chamfer distance choice\nOur decision was based on theoretical and practical advantages. The Chamfer distance is well-established for comparing point clouds, which is how we decided to represent images, with components naturally mapping to **fidelity** and **diversity** in generative modeling. It is **computationally efficient**, relying on simple 1-NN search, critical for guidance in the diffusion sampling loop.\n\nWe considered other targets but found them unsuitable:\n- **FID**: Not feasible as it requires the full distribution to be stable, and it is computationally prohibitive to backpropagate through.\n- **Recall/Coverage**: They require full data distributions and are **non-differentiable**.\n\n## [Q3] Shifted distribution\nOur geographic diversity study used **GeoDE** and **DollarStreet** datasets, known to be challenging with distributions shifted from generative model training data. Base models struggle with these specialized datasets [2, 8].\n\nOur results demonstrate that Chamfer Guidance is **robust and effective** in this setting, successfully steering generation to align with few-shot exemplars from shifted domains, proving practical utility in real-world scenarios.\n\n## [Q4] Guidance scale\nThe CFG scale for each model was chosen by performing a sweep and selecting the value maximizing **F1 score on a validation set**. For the large LDM3.5M model, we chose $\\omega = 1$ also to speed up generation, given the size of the model.\n\n---\n\n[1] Hemmat, R. A., Pezeshki, M., Bordes, F., Drozdzal, M., & Romero-Soriano, A. Feedback-guided Data Synthesis for Imbalanced Classification. TMLR 2024\n\n[2] Hemmat, R. A., Hall, M., Sun, A., Ross, C., Drozdzal, M., & Romero-Soriano, A. Improving geo-diversity of generated images with contextualized vendi score guidance. ECCV 2024\n\n[3] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR 2023\n\n[4] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023\n\n[5] Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. CVPR 2023\n\n[6] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A Survey on In-context Learning. EMNLP 2024\n\n[7] Wang, Z., Jiang, Y., Lu, Y., He, P., Chen, W., Wang, Z., & Zhou, M. In-context learning unlocked for diffusion models. NeurIPS 2023\n\n[8] Hall, M., Ross, C., Williams, A., Carion, N., Drozdzal, M., & Soriano, A. R. Dig in: Evaluating disparities in image generations with indicators for geographic diversity. TMLR 2024"}, {"review_id": "azBMtir9yW", "reviewer": "Reviewer_wmJJ", "summary": "This paper uses the average closest-point distance between two sets (Chamfer distance) to guide the diffusion process. It leverages a set of previously generated images and a set of real images to enhance both fidelity and diversity. The authors present experiments on geo-diversity benchmarks using LDM diffusion models (1.5M and 3.5M parameters).", "questions": "- Why is Chamfer distance preferred over more distribution-aware metrics such as FID or KID? A brief justification would clarify the rationale behind this design choice.\n- How do diversity metrics (e.g., Density/Coverage, Vendi Score, CLIPScore) behave across different classifier-free guidance (CFG) scales? Specifically, does the proposed method demonstrate improved diversity at higher CFG values compared to existing baselines?\n- Can the method improve diversity in the absence of a reference dataset, as in open-ended text-to-image generation tasks using models like FLUX or SDXL?\n- Is the method applicable across different prompts? In other words, can it generalize to multi-prompt scenarios while enhancing diversity?\n- Does the method scale to large diffusion models (e.g., SDXL), and does it yield measurable improvements in diversity both quantitatively and qualitatively?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an important question in the literature.\n- A key strength of this work compared to c-VSG is the reduced computational complexity of using the Chamfer distance.\n\nWeaknesses:\n- The primary concern is the limited novelty of the proposed approach.\n- The method assumes access to real datasets, which may not be feasible in many real-world settings or for natural prompts.\n- It also requires that the generated image set corresponds exactly to the same prompt, which limits flexibility and scalability.\n- The paper misses important baselines such as Particle Guidance [1] and Interval Guidance [2], which are directly relevant for promoting diversity and distributional alignment. This exclusion narrows the scope of the evaluation.\n\n\n[1] Corso et al. \"Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models\", In ICLR 2024\n\n[2] Kynkäänniemi et al, \"Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models\", In NeurIPS 2024", "was_revised": true, "final_justification": "Bellow are my remaining concerns:\n\n> Accessing to real data, Few-shot Adaptation and Personalization:\n\nI believe that having access to real data, especially for text-conditioned models, is difficult to obtain and represents a special case, as mentioned by Reviewer tTXY \"scenarios where one truly has no real data, Chamfer Guidance cannot be applied\". The authors have justified this with Few-shot Adaptation and Personalization. To strengthen their argument, I believe the manuscript should include experiments and evidence demonstrating that the methods perform well in these tasks, which is currently lacking.\n\n> Extending our method to text-to-image is a promising future direction\n\nI respectfully believe that this is essential for the submission and should be included in the manuscript rather than relegated to a future direction.\n\n\n> Early experiment using FAISS for fast retrieval\n\nThe results looks promising and extending the results could definitely help the manuscript\n\n> Multi-prompt applications would be an interesting future direction\n\nThe authors frame multi-prompt generation as \"an interesting future direction.\" However, I believe this is necessary for the generalizability and applicability of their method. The authors mentioned that their method \"would suit this task because our guidance operates in the feature space of the exemplars,\" which is not trivial for me and requires experimental backing it up.\n\n> Overhead of feature extraction and reliance on them\n\nAs noted by Reviewer tTXY, I also agree that reliance on these feature extractors incurs considerable overhead in terms of GPU usage and may introduce biases.\n\nIn summary, most of my concerns remain unresolved, and I believe there is a room for improvement", "rebuttal": "We would like to clarify the models used in our experiments:\n- **LDM1.5** refers to **Stable Diffusion 1.5**, a widely-used, **U-Net-based** implementation with 860 million parameters.\n- **LDM3.5M** refers to **Stable Diffusion 3.5 Medium**, a flow-based **Multimodal Diffusion Transformer** with 2.5 billion parameters representing **state-of-the-art** among openly available generative models.\n\nWe then thank the reviewer for finding that our work **tackles an important question in the literature**, and that **a key strength compared to c-VSG is the reduced computational complexity of using the Chamfer distance**.\n\n## [W1] Novelty\nWe believe our work presents a significant contribution, as highlighted by other reviewers (**a novel training-free guidance method that effectively enhances synthetic image utility**, and that **the idea of using Chamfer distance as guidance reward is conceptually elegant**).\n\nOur primary contribution is a **simple, effective, and computationally efficient In-Context Learning (ICL) framework for generative models**. While ICL has transformed language modeling, its application to steer generative vision models is **significantly under-explored**. We are among the first to propose a practical method for this purpose.\n\nThe novelty lies in creatively adapting the **Chamfer distance**, traditionally used for point clouds, as a guidance signal within a diffusion model feature space. This introduces a new way to control generation based on few-shot examples.\n\nWe demonstrate **state-of-the-art results** in the downstream usefulness of generated synthetic data with minimal data. These constitute a substantial contribution.\n\n## [W2] Access to real data\nWe argue that our work addresses a well-established and increasingly critical problem in generative modeling.\n\nThe problem of adapting a generative model to a target distribution given a few reference examples is a significant area of research, which is foundational to several applications:\n- **Few-shot Adaptation:** This is a common setting for tasks like data synthesis for imbalanced classification, as explored in prior work [1, 2], which we also include as baselines.\n- **Personalization:** The goal of personalizing large text-to-image models for subject-driven generation, as seen in impactful methods such as Textual Inversion [3], DreamBooth [4], and SVDiff [5].\n\nMore broadly, our work is an instance of **In-Context Learning (ICL) for diffusion models**. ICL is a cornerstone of modern LLMs [6], and its application to generative vision models is a critical emerging direction [7].\n\nWe are among the first to successfully adapt large diffusion models via ICL **without requiring finetuning**.\n\n## [W3,Q3] Scalability and extension to text-to-image\n\nOur work focuses on **object-centric** and **class-conditional** generation, aiming to improve the downstream utility of synthetic data (e.g., for classification and geographic representation). This is a well-established research area, distinct from open-ended text-to-image synthesis, which is a different scope. Our contribution is a novel guidance mechanism that excels within this paradigm.\n\nHowever, extending our method to text-to-image is a promising future direction. One approach could be:\nStep 1. **Offline retrieval database**: Embed a large text-image dataset into semantic vectors based on their captions.\nStep 2. **Inference-time Retrieval**: For a user's text query, retrieve the top-k relevant image-text pairs and use their images as exemplars for our guidance.\n    \nWe perform an early experiment by implementing this using COCO. For Step 1, we use the image-text pairs from the COCO training dataset. We embed the captions with CLIP, and we use FAISS for fast retrieval. For Step 2, we subsample 5000 captions from the validation set and retrieve the top-k closest images to measure the Chamfer distance over. In this setting, we compared the performance of LDM1.5 with default sampling and our Chamfer Guidance. Results are reported in the table below, and we can observe that our method brings substantial improvement in this setting as well.\n\n| Method     | F1   | Precision | Coverage | Density | Recall | FDD    | FID   |\n| ---------- | ----- | --------- | -------- | ------- | ------ | ------ | ----- |\n| LDM1.5     | 0.785 | 0.851     | 0.729    | 0.716   | **0.742**  | 305.67 | 28.69 |\n| Ours (k=2) | **0.937** | **0.937**     | **0.938**    | **1.294**   | 0.726  | **202.65** | **21.96** |\n\n## [W4] Additional baselines\nWe initially omitted them as they address **orthogonal** research goals. Those methods increase _ungrounded diversity_ without specific targets, similarly to CADS, which we employed as a baseline, while our work focuses on **grounded generation**.\n\nWe have implemented these methods. As shown below, while they may increase ungrounded diversity (recall), they don't perform well on our grounded diversity task requiring fidelity to exemplars.\n\n| Method                       | F1​       | Precision | Coverage  | Density   | Recall    | FDD        | FID      |\n| ---------------------------- | --------- | --------- | --------- | --------- | --------- | ---------- | -------- |\n| CADS                         | 0.718     | 0.850     | 0.621     | 0.743     | 0.546     | 217.96     | 13.434   |\n| Limited Interval             | 0.708     | 0.837     | 0.613     | 0.686     | 0.631     | 219.17     | 11.405   |\n| Particle Guidance            | 0.719     | 0.846     | 0.625     | 0.744     | 0.544     | 222.26     | 14.52    |\n| Chamfer Guidance (ours) k=2  | 0.886     | 0.947     | 0.833     | 1.108     | 0.480     | 156.18     | 13.67    |\n| Chamfer Guidance (ours) k=32 | **0.931** | **0.950** | **0.912** | **1.213** | **0.649** | **113.30** | **8.94** |\n\n\n## [Q1] Why Chamfer distance\nOur decision was based on its combination of theoretical and practical advantages of the Chamfer distance for our specific task.\n\nChamfer distance is a well-established and principled metric for comparing point clouds, which is how we represent features in the embedding space. Its two components naturally map to the desirable properties of **fidelity** and **diversity** in generative modeling. It is also **computationally efficient**, as it relies on a simple 1-Nearest Neighbor (1-NN) search. This efficiency is critical for its usage as a guidance signal in the diffusion sampling loop.\n\nWe also considered other candidate optimization targets but found them unsuitable:\n- **Fréchet Inception Distance (FID)**: It was not feasible. It requires large distributions of both real and generated data to be meaningful. Moreover, its computational cost would make it prohibitive to optimize at each sampling step.\n- **Optimizing Recall/Coverage**: These metrics also require access to the full data distributions for accurate estimation. More importantly, because they involve counting, they are **non-differentiable**, posing a significant optimization challenge.\n\n## [Q2] Different CFG scales\nThis analysis was included in our original submission (**Table 4** and **Table 10** in Appendix). Results demonstrate our method's improvements are consistent across CFG scales.\nFor the reviewer's convenience, we are reporting these key results below. \n\n| ω   | k   | F1    | Precision | Coverage | Density | Recall | FDD     | FID    |\n| --- | --- | ----- | --------- | -------- | ------- | ------ | ------- | ------ |\n| 1.0 | --  | 0.507 | 0.723     | 0.391    | 0.551   | 0.656  | 431.2   | 31.3   |\n| 1.0 | 2   | 0.849 | 0.890     | 0.811    | 0.904   | 0.736  | 150.748 | 13.217 |\n| 1.0 | 32  | 0.899 | 0.923     | 0.876    | 1.086   | 0.735  | 117.834 | 9.759  |\n| 2.0 | --  | 0.673 | 0.802     | 0.580    | 0.648   | 0.684  | 226.2   | 10.8   |\n| 2.0 | 2   | 0.881 | 0.932     | 0.835    | 1.051   | 0.637  | 124.191 | 8.840  |\n| 2.0 | 32  | 0.931 | 0.950     | 0.912    | 1.213   | 0.649  | 113.301 | 8.935  |\n| 7.5 | --  | 0.709 | 0.862     | 0.603    | 0.775   | 0.415  | 248.7   | 16.1   |\n| 7.5 | 2   | 0.886 | 0.947     | 0.833    | 1.108   | 0.480  | 156.179 | 13.670 |\n| 7.5 | 32  | 0.925 | 0.957     | 0.894    | 1.238   | 0.498  | 153.111 | 14.388 |\n\n## [Q4] Multi-prompt scenario\n\nWhile our current focus is on object-centric generation, tackling text-to-image, for which we showed preliminary results in [Q3], and in particular multi-prompt applications would be an interesting future direction. We interpret \"multi-prompt\" as using a single set of visual exemplars to guide generation for several different text prompts.\n\nOur approach would suit this task because our guidance operates in the feature space of the exemplars. This effectively **decouples the visual concept** (the exemplars) from the **semantic context** (the prompt).\n\nOur guidance would ensure the generated concept is consistent with the exemplars, while the context and style are dictated by the specific prompt. Evaluating such a use case would require suitable data and parameter tuning.\n\n## [Q5] Large diffusion models\nOur experiments include LDM3.5M, the 2.5 billion parameter Stable Diffusion 3.5 Medium model. This is a state-of-the-art Multimodal Diffusion Transformer, competitive with other large models like SDXL and FLUX. Thus, our method has been validated on current, large-scale diffusion models.\n\n---\n[1] Hemmat, R. A., et al. Feedback-guided Data Synthesis for Imbalanced Classification TMLR 2024\\\n[2] Hemmat, R. A., et al. Improving geo-diversity of generated images with contextualized vendi score guidance ECCV 2024\\\n[3] Gal, R., et al. An image is worth one word: Personalizing text-to-image generation using textual inversion ICLR 2023\\\n[4] Ruiz, N., et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation CVPR 2023\\\n[5] Han, L., et al. Svdiff: Compact parameter space for diffusion fine-tuning CVPR 2023\\\n[6] Donq Q., et al. A Survey on In-context Learning EMNLP 2024\\\n[7] Wang, Z., et al. In-context learning unlocked for diffusion models NeurIPS 2023"}]

Author rebuttal:

Meta-review: The submission received divergent ratings: 2 positive and 1 negative. A mutual concern raised by Reviewer wmJJ and Reviewer tTXY is the applicability of the proposed method in the zero-shot setting. The authors provided a rough idea of extending the current method. While Reviewer tTXY is satisfied with this response, Reviewer wmJJ is not convinced, as there is no experiment verification. The ratings of Reviewer tTXY and Reviewer wmJJ are opposite, accept vs. weak reject. Reviewer mRsF kept the positive rating after the rebuttal. 

The AC has checked the review, the rebuttal, the manuscript, and the discussion. The AC found that the focus of this paper is to advance the utility of synthetic data, which is underpinned by adequate evaluations. Extending this method to other domains/settings is worth exploration, but can be follow-up work afterward. The AC decided to support this work on the condition that: i) the manuscript will be revised based on the reviews; 2) elaborate on the limitation or extension of the current method in sec. 5. 

The final recommendation is acceptance. The decision has been discussed with and confirmed by the SAC.
Decision: Accept (poster)