Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Authors: Michael Kirchhof
James Thornton
Louis Béthune
Pierre Ablin
Eugene Ndiaye
marco cuturi
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
Guidance
Repellency
Diversity
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Guiding text-to-image diffusion trajectories away from protected images.

Abstract: The adoption of text-to-image diffusion models raises concerns over reliability, drawing scrutiny under the lens of various metrics like calibration, fairness, or compute efficiency. We focus in this work on two issues that arise when deploying these models: a lack of diversity when prompting images, and a tendency to recreate images from the training set. To solve both problems, we propose a method that coaxes the sampled trajectories of pretrained diffusion models to land on images that fall outside of a reference set. We achieve this by adding repellency terms to the diffusion SDE throughout the generation trajectory, which are triggered whenever the path is expected to land too closely to an image in the shielded reference set. Our method is sparse in the sense that these repellency terms are zero and inactive most of the time, and even more so towards the end of the generation trajectory. Our method, named SPELL for sparse repellency, can be used either with a static reference set that contains protected images, or dynamically, by updating the set at each timestep with the expected images concurrently generated within a batch, and with the images of previously generated batches. We show that adding SPELL to popular diffusion models improves their diversity while impacting their FID only marginally, and performs comparatively better than other recent training-free diversity methods. We also demonstrate how SPELL can ensure a shielded generation away from a very large set of protected images by considering all 1.2M images from ImageNet as the protected set.

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Michael Kirchhof 12
James Thornton
Louis Béthune
Pierre Ablin
Eugene Ndiaye
Marco Cuturi
Abstract
1. Introduction
Diffusion models (Song et al., 2021; Ho al., 2020) are by now widely used for engineering and scientific tasks, to generate realistic signals (Esser et al., 2024) or structured data (Jo et al., 2022; Chamberlain et al., 2021). Diffusion models build upon a strong theoretical foundation used to guide parameter tuning (Kingma & Gao, 2023) and net- work architectures (Rombach et al., 2022), and are widely adopted thanks to cutting-edge open-source implementa- tions. As these models gain applicability of problems, their deployment reveals important challenges. In the specific area of text-to-image diffusion (Nichol et al., 2022; Saharia et al., 2022), these challenges can range from an expensive compute budget (Salimans & Ho, 2022) to a lack of diversity (Ho Salimans, 2022; Shipard et al., 2023) and/or fairness (Cho et al., 2023; Shen et al., 2024).
The adoption of rext-to-image diffusion models raises concerns over reliability, drawing scrutiny under the lens of various metrics like calibration. fairness, or compute efficiency. We focus in this work on two issues that arise when deploying these models: a lack of diversity when prompt- ing images, and tendency recreate images from the training set. To solve both problems, we propose method that coaxes the sampled trajec- tories of pretrained diffusion models to land on images that fall outside of a reference set. We achieve this by adding repellency terms to the dif- fusion SDE throughout the generation trajectory, which are triggered whenever the path is expected to land too closely to image in the shielded ref- erence set. Our method is sparse in the sense that these repellency terms are zero and inactive most of the time, and even more so towards the end of the generation trajectory. Our method, named SPELL for sparse repellency, can be used either with a static reference set that contains protected images, or dynamically, by updating the set at each timestep with the expected images concur- rently generated within batch, and with the im- ages of previously generated batches. We show that adding SPELL to popular diffusion models improves their diversity while impacting their FID only marginally, and performs comparatively bet- ter than other recent training-free -fee diversity meth- ods. We also demonstrate how SPELL can ensure shielded generation away from a very large set of protected images by considering all 1. 1.2M im- ages from ImageNet as the protected set.
wide range
Controllable Generation. We focus the problem of en- suring that images obtained from model are sufficiently different from a reference set. This covers two important use-cases: (i) the purveyor of the model wants images gen- crated with its model to fall outside a reference set of pro- tected images; (ii) the end-user wants high diversity when generating multiple images with the same prompt, in which case the reference set could consist of all previously gener- ated images, batch. While the problem of avoiding generating images in a protected training set (Carlini et al., 2023) originates natu- rally when deploying products, that of achieving diversity within a batch of generated images with the same prompt should not arise, in theory, if diffusion models were per- fectly trained. However, shown for instance by Sadat et al. (2024), ofate-of-the-art models that incorporate classifier- free guidance (Ho & Salimans, 2022, CFG) do a very good job at outputting a first picture when provided with prompt, but will typically resort to slight variations that same im- age when re-prompted multiple times. This phenomenon is illustrated Figure for three popular diffusion mod- els, Stable Diffusion 3 (Esser al., 2024), Simple Diffusion (Hoogeboom et al., 2023) and MDTv2 (Gao et al., 2023).
other images generated concurrently in
'Apple UUiiersiity Tübingen. Correspondence to: Michael Kirchhof <Contact info see website>..
Contributions. We propose guidance mechanism coined sparse repellency (SPELL), which repels the backward diffu- sion at generation time away from a reference set of images.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Stable Diffusion 3
Simple Diffusion
MDTv2
Prompt: "A close-up of apple"
Prompt: "The Eiffel Tower"
ImageNet class 145 (king penguin)
SPELL (Our))
+ SPELL (Ours)
+ SPELL (Ours)
Figure 1. SPELL interventions can change the diffusion trajectory of any pre-trained diffusion model by self-avoiding other images generated, in the same or previous batches (and also any other non-generated image). This makes SPELL achieve higher diversity above, with prompts, and noise seeds as the base modcls. We provide more qualitative examples in Appendix N.
SPELL interventions are sparse by design; they only con- sider very few active shielded images (typically one) at each time f, and happen mostly early in the generation.
port X' C R4, such as image dataset, by simulating the reverse stochastic differential equation (SDE) (Haussmann & Pardoux, 1986; Anderson, 1965), initialised from some easy to sample prior p1 E P(R)), X1
SPELLI can increase the diversity of outputs by dynami- cally updating the shielded reference set to be images that were generated in previous batches and those that are ex- pected to be generated in the current batch.
dx, [S(,,,,) 9 (t) log pe (X. )ld g(t)dB, (1)
where (B:), denotes Brownian motion and pt is defined as the density of X, from forward process:
We apply SPELL to numerous saaeeof-thh-aa art diffusion models and find that the generated images better reflect the diversity of the true data (see Figure 1) with a marginal or even increase in the Fréchet inception distance (FID).
dx, = f(t, X)) + g())dB, Xo Po == Pdata (2)
for drift f: (0. 1) x X -> X' and diffusion scale g: [0. 1] + R, where the time t is increasing in equation 2 and time de- creasing in equation 1. The score term V log (og):(XX) is typi- cally approximated by a neural network through denoising score matching (Vincent, 2011)
SPELL is simply parameterized by the shields' radius. We show that increasing r increases accordingly the out- put's diversity, with a better eiversity-precision trade-off than other recently proposed methods (Sadat et al., 2024; Corso et al., 2024; Kynkäänniemi et al., 2024).
Training, The solution forward diffusion in equation for an affine drift is of the form X, axXo + o,5, where
We scale SPELL to a reference set of millions of images. This enables a second use-case: We shield the whole ImageNet- 1k train dataset and generate images that are novel, without requiring to filter regenerate images.
(O0, I) for some coefficients 0₂ E R. a, E R (Song et al., 2021; Särkkä & Solin, 2019). The intractable score term may be expressed via denoiser using Tweedie's formula (Efron, 2011: Robbins, 1956): V Vl = a,E|X, = Hence rather than estimating the score directly, one may approximate E E[X, X, = xx]] via regres- sion, by minimizing:
2. Background
Diffusion Models, also known as score-based generative models (Song et al., 2021; Song & Ermon, 2019; Ho al., 2020), enable sampling from data distribution Pdata on sup-
8* := arg min Ex, Do (t,) xxol2
(3)
2

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
known mean-prediction, for optional condition de- noted y, then estimate the score via V log (1 y) Rs sg. (t, 2x,3) := (a, (f, Zt, v) /ef/of. Notice that we do not train model parameters in this work, and will always assume that 0* is given by the purveyor of a model.
Setup. Our goal is sample from the data distribution Po whilst satisfying the important requirement that gener- ated samples Xo ~ po are far enough away from each ele- ment of the reference set of repellency images 2; E X, k = 1,..., K. That set may be populated by real-world pro- tected images, samples generated in earlier batches, im- ages expected to be generated by other trajectories in the current batch, or mix of all these types. More formally, we wish to sample a conditioned trajectory X, (Xo S), where S is the collection of shields, i.e. balls of ra- dius r > 0 around repellency images, S == (U, Bx) with B₂ = {x E X || - 2x112 S r). A brute-force mecha- nism to guarantee generation outside of S is to generate and discard: resample multiple times both initial noise and Brownian samples, follow the diffusion trajectory and re- peat until a generated image falls outside of S. In the con- text of computationally expensive diffusion models, this would be wasteful and inefficient. Instead, we seek mech- anism which satisfies the protection in each generation, for any conditional or unconditional diffusion model.
Conditioning and Guidance. Conditional diffusion mod- els requires access to the conditional score V log pr (z/ | y) for some condition y such as text or label. It is typically ap- proximated either with explicit conditioning during training of the score/ / denoising network using equation 3 or as post- hoc additional guidance term added to the score. Given dif- fusion models have lengthy training procedures, likely due to their high variance loss (Jeha et al., 2024), it is desirable to guide diffusion models with inexpensive post-training guidance (Dhariwal & Nichol, 2021; Zhang et al., 2023; Denker et al., 2024), using e.g. classifier guidance (Dhari- wal & Nichol, 2021)
V logge (xx vv) = V (or) + 7 V po xt) (4)
whereby the gradient V log mi) of classifier p(v | (x)) for label y is added to the score, heuristically multiplied by a scalar 7 2 1 for increased guidance strength. Another approach which circumvents training a time-indexed clas- sifier is using the approximation ply x²) Rs p(y] Xo- =
A Geometric Interpretation of SPELL. To ensure that generation falls outside of the shielded set S, we modify the diffusion trajectory each time step, as presented in Figure 2, without having to discard any samples. To do so, we correct the trajectory whenever the expected output, E[x, X, = 2x] falls within a shield. Here the expected given current state It, is approximated by the denoising network Dg. (t, æt). Using the notation dg = Dø (t, xz), we test whether for any index & one has || - zæl12 < r. that is the case, the minimal modification ò that can ensure | + d - 2x lll2 is
Dg- (t, rt)), for pretrained denoiser Dg- alias Diffusion Posterior Sampling (DPS) (Chung al., 2023).
Classifier-Free Guidance and Lack of Diversity in text- to-image Diffusion Models. Classifier-free free guidance (CFG) (Ho & Salimans, 2022) is the dominant conditioning mech- anism in text-to-image diffusion models, sharing proper- ties with both explicit training and guidance. Similar to classifier guidance, CFG may be used to increase guid- ance strength but without resorting approximating den- sity P(v at)). Instead, the difference V log xT)) V p(r; 1 y) V (xrz) is used as a guidance term, where each term is approximated with the same conditional network: V Vlogp(:; y) x st(t, 22,y) 3) and V (logp(az)) R so xe. 0). for null condition 0, trained as in equation 3. Adding CFG to the unconditional score yields V Vlg pe (are | y) (ts 22, y)- (7- 1) )))o(t, .z, 0). De- spite its widespread popularity and good performance, CFG weighting is heuristic. It is not clear what final distribution is being generated; and practitioners observe a lack of di- versity in generated samples (Somepalli et al., 2023; Chang et al., 2023; Wang et al., 2024).
6G = (āo 2k)rr
(00 2k).
100
2x 112
Across all k I,..., K, we modify the trajectory only for those i that do is too close to, giving
K
A
1 1B (fo) (20
(5)
k=1
K
1:00 - zu|12 -1) (8o zx) E IRd
r
Trelu
where the set of indices k that the ReLU is non-zero for at each individual timestep is usually very small. Under the assumption that all of their shields Bş are disjoint, for example when the radius is small enough, this update strictly ensures that qy + A & S. When shields overlap, we do not have such guarantee. While more complicated projection operators might still yield exact updates in that case, they would involve the resolution of quadratic program. We take the view in this paper that A strikes a good balance between accuracy and simplicity.
3. Sparse Repellency
In this section, we introduce SPELL. We first give a geomet- rical intuition of how its repellency terms steer the diffiusion trajectory out of shielded areas. Then, we make a deeper dive into theoretical connections to DPS (Chung et al., 2023) and particle guidance (Corso al., 2024).
3

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
(3)
Images (data)
density
24-2.z
0
Initial noise
X2
Shield radius covering
(1)
Image space
SPELL intervention during generation
Diffusion time
(a) shielded generation, static set
(b) intra-batch repellency
(c) both
22).. we detect that the trajectory is headed (in expectation) into the shield radius r centered around 22. Our sparse repellency (SPELL) term depicted as black arrow adds correction when generating 2;- Ar ensure that the trajectory pushed out of shield. This again in the case in the next step, when starting from ft- (b) In batched generation, the shields are dynamically recreated at every iteration around each trajectory's 's expected output. This prevents two elements in the batch, =!") and (2) from generating the same output. (c) Both approaches can be combined to yield a diverse set of images that won't fall into protected images and previously or concurrently generated images.
Figure (a) At time t, by computing X|X,| X.
Figure 2(a) visualizes the repellence mechanism away from static protected images while Figure 2(b) shows how it dynamically repels from trajectories within the same batch.
cable approach to conditioning and guiding diffusions. Un- fortunately, Doob's h transform is generally intractable. We may however appeal to diffusion posterior sampling (Chung et al., 2023) and approximate Poj: as a Gaussian with mean ig Rs E[X,, zz]. which is available from diffusion model pre-training, see Section 2. This approximation results in the following correction:
The batch generation produces B samples (0) in parallel and its repellency mechanism uses a time-evolving set of repellency points æ4, = E[X, X. x corresponding to the currently predicted end-state of each sample in the batch. Overall, this leads to a blue-noise like coverage of the distribution, as we visualize in 2D for the two-moons dataset in Figure As we made no further assumptions on 24, these mechanisms can be mixed as Figure 2(c) to enable diverse generation across arbitrary numbers of batches. This makes it possible to generate large numbers of diverse images even when the VRAM for each batch is limited. Note that SPELL is a post-hoc method that does not require retraining and can be applied to any diffusion score, RGB space or latent space, unguided or classifier-free guided. Appendix D provides pseudo code and further implementation details.
K V log Poje PXo (r)) > a(lio =xll2,r) (fo- zx):
where r) is weighting factor detailed in Appendix A that decreases in its first variable. This DPS approxima- tion is similar to SPELL in that both push away trajectories in the directions (&o - zx), weighted by a factor that de- pends on r and the distance f 2 ll2- The difference is that DPS based on Gaussians provides a soft guidance that slowly vanishes d. moves away from 2k, and not hard guarantee that we respect the protection radii around each 2k. We have struggled in preliminary experiments to set hy- perparameters of such "softer" DPS schemes, because the weight factor to scale the Gaussian by ultimately depends on the magnitude the likelihood of the shields, which is un- known, and because the Gaussian's 's weight never becomes exactly zero, hindering sparsity. This is why focus attention on the simpler and much cheaper SPELL.
SPELL DPS guidance term. We propose to derive SPELL as a guidance mechanism, by Bayes' rule
Vx, (a Io & S) -Va, log (x::)
+ Vx, log Poje & S xx)
Hence, we may sample X, xo & S by adjusting a pre- trained score function and simulating:
& (X,, 5) log (X.) Vlog Po (Xo S xx))
(Intra-batch) SPELL and Particle Guidance. When using SPELL to promote diversity within the generation of a sin- gle batch (but without the more general protection against arbitary or previously generated images), SPELL can be re-
dx, = [f(t, - (()) (X,, 5)] dt Bt) (6)
The term log Poje (zo & S 23) in the score adjustment is known as Doob's 'sbbss h transform, and provides broadly appli-

---

V .n()) log 0 (1r) 2. (1)) is then used to correct each in- dividual trajectory to guarantee diversity. In contrast to this approach, SPELL draws insight on the expected future loca- tions of points, the end of the trajectory, i.e the expected Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
1:25
1:00
0.75
0.75
0.50
0.50
0.25
0.00
0.00
-0.25
-0.25
-0.50 1550
00.50
1.0
0.5
1.0
2.0
(a) DDPM without SPELL
(b) DDPM + Intra-batch SPELL (r = 0.075)
Figure 200 samples generated with an unconditional DDPM diffusion model on the two moons toy dataset, with and without intra-batch SPELL with minimum shield radius of r 0.075. SPELL's shields lead to blue noise- coverage of the distribution.
lated the self-interacting particle guidance (PG) approach proposed by (Corso et al., 2024). That approach defines an interacting energy potential Φ; time using the lo- cations in space of all B particles within a batch at time t. The gradient of that potential w.r.t. each particle, (+)
of getting more hectic dynamics. We illustrate this addition in Figure 5 with value À = 1.6, which find work favorably across multiple models.
4. Related Works
Most closely related to our method is that of Corso et al. (2024), who promote diversity through an intra-batch, repul- sion term, used as guidance for pre-trained diffusion mod- els. Similarly to our work the repulsion term can be applied at E E|Xo| [2tx] or features. The primary difference is the sparsity of the repulsion term in SPELL, and using this on fixed reference set addition to intra-batch.
denoised images (0 and (8) where co = Da- (t,, m4). In- deed, the correction for each particle explicitly given as:
A
K Trelu
|| 20 1/ 1) (4.0)
200)).
(7)
Chen et al. (2024) also closely related. Here, the authors apply anti-memorization guidance pretrained models through three terms: desspecification; caption de- duplication and dissimilarity guidance. Similar to Sadat et al. (2024), desspecification adjusts the CFG scale depen- dent on nearest neighbor in the training data to E|Xo xx). Caption de-duplication adds a negative guidance term (Liu et al., 2022) based on the network applied to the caption of the nearest neighbor. Finally, dissimilarity guidance applies additional guidance term based on similarity between E [[Xo|22 and nearest neighbor search. Our approach is most similar to dissimilarity guidance using multiple nearest neighbors and carefully chosen guidance scale to encourage generated samples to be outside a radius of reference points.
The B correction terms A ( considered by SPELL can- not be seen to our knowledge as the gradients of an in- teracting potential. While we prove in Appendix C that h(x) = Trelu ( | dient of a potential), we find no guarantee for the more com- plex SPELL updates above which involve compositions of with the denoiser Dø-. Even if SPELL was conserva- tive field, the biggest difference between PG and SPELL that PG defines dense interventions between all particles using soft-vanishing kernels that are never zero and thus al- ways perturb diffusion trajectories. SPELL, conversely, in- tervenes sparsely and rarely, both in time and w.r.t. points in the reference set. As a result, the original diffusion process less perturbed, notably towards the end of a trajectory, and SPELL scales large reference sets of millions of shields.
P - 1)x is conservative field (i.e. the gra-
Since initial release our work, Koulischer et al. (2024) has provided more theoretical grounding for weighting negative guidance (Liu et al., 2022) based an estimated class- based likelihood. Kim et al. (2025) demonstrates negative guidance using the empirical score of reference set of images. Unlike aforementioned guidance based approaches, Thornton et al. (2025) introduces a gradient-free SMC-based sampler that encourages diversity by negatively weighting the energy a Feynman Kac potential, demonstrated only toy examples.
Overcompensation. While method gives the exact weight required to land outside the shielded areas in Equa- tions (5) and (7), we have experimented with scaling these terms by an overcompensation multiplier A. Intuitively, the larger that multiplier, the earlier the trajectory will be lead out of the shielded areas, with the possible downside
5

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Table 1. SPELL improves the diversity of text-to-image and class-to-image diffusion models considerably, at only small trade-off in
terms of precision. The results are reported as mean + std, computed over 5 independent runs with different seeds over the full dataset.
Model
Setup
Recall 1
Vendi
Coverage 1
Precision 1
Demsity 1
FID 4
DDasoo 4
Latent Diffasion
text-to-image
0.236 = 0.003
2.527 4 0.005
0.447 + 0.001
0.589 0.000
0.768 + 0.002
9,501 = 0.024
106,244 = 0.384
+ SPELL (Ours)
text-to-image
0,289 0.003
2.695 1:09:000 0.002
0.457 = 0.001
0.551 0.001
0.745 + 0.002
9.554 0.043
98.761 0.441
SD3-Mcdiom SPEEL (Ours)
texi-to-image
0.379 = 0.004
3.749 +: 0.005
0,294 + 0.000
0.313 + 0.001
0.345 # 0.001
20.103 = 0.090
230.248 = 0.812
lex--o-inuge
0.483 0.002
4.711 : 0.013
0.229 0.001
0.211 0.002
0.213 1 0.002
35.174 0.153
482.246 0.948
Simple Diffusion
lext-to-imin
0.230
0.003
2.799
0.006
0.355 + 0.002
0.441
0.001
0.556 4: 0.002
19.879
0.003
248.138 + 0.586
+ SPELL (Ours)
text-lo to-inage
0,248 1 0.002
2.886 0.005
0.355 1 0.001
0.433 1 0.002
0.541 1 0.002
19.959 1: 0.033
245.748 1 0.562
FDMv2
class-to-image
0.589 + 0.002
11.645 + 0.022
0.551 0.002
0.518
0.002
1.404 + 0.005
3.377
0.022
68.452 0.298
SPELL (Ours)
class-to-image
0.600 0.002
11.800 + 0.013
0.547 + 0.001
0.508 0.001
1,364 + 0.005
3:456 + 0.021
68.909 0.161
SLD3 Medium-Class
cluss-to-image
0.143 =: 0.002
8.861 +: 0.028
0.202 +: 0.002
0.323 1 0.002
0.801 1 0.005
22.246 = 0.020
328.032 1: 0.571
SPELL (Ours)
class-to-image
0.206 0.002
12.190 + 0.032
0.146 0.001
0.181 0.002
0.420 + 0.006
38.709 0.054
778.286 0.553
MDTT2
class to image
0.623 1. 0.002
12.546 1 0.021
0.505
0.001
0.401
0.002
1,020 1. 0.002
4.884
1. 0.052
133.175 1 0.721
+ SPELLI (Ours)
class-tn in-inage
0.634 + 0.002
12.772 + 0.027
0.505 + 0.001
0.407 0.001
1,029 + 0.005
4.381 + 0.047
122.125 + 0.291
5. Experiments
images each (in total 41,596 images). We explain the con- struction of this dataset in Appendix E.
We now show that SPELL increases the diversity of modern sotttto-mmae and clndicooniiiinal diffusion models (Sec- tion 5.2), with better trade-off than other recent diversity methods (Section 5.3). We quantify the sparsity of SPELL interventions in Section 5.4. In Section 5.6, we demonstrate SPELL's scalability and a new use-case, shielded genera- tion, by generating novel ImageNet images while shielding all 1.2 million ImageNet-I Ik train images.
To evaluate diversity, we track the recall (Kynkäänniemi et al., 2019), coverage (Naeem et al., 2020), and Vendi score (Friedman & Dieng. 2023). To evaluate image qual- ity, we use precision (Kynkäänniemi et al., 2019) and den- sity (Naeem et al., 2020). We track these metrics per class/prompt and average across classes/prompis. To mea- sure whether the generated images match the true image distributions, we use the marginal FID (Heusel et al., 2017) and the marginal Fréchet Distance with DINOv2 features (FDDINov2, Stein et al. (2024); Oquab et al. (2024)).
5.1. Experimental Setup
In the class-to-image setup, we use Masked Diffusion Trans- formers (MDTv2) (Gao et al., 2023), EDMv2 (Karras et al., 2024), and Stable Diffusion 3 Medium (SD3) (Esser et al., 2024), three recent state-of-the-art diffusion models. We use the pretrained model checkpoints to generate 50,000 256x256 images of ImageNet-Ik classes(Deng et al., 2009) without and with SPELL and compare them to the original ImageNet-16 1k images. We use the validation dataset as a comparison, since we will conduct experiments that repel from the training dataset in Section 5.6, which would render comparisons to the training dataset meaningless. Given the text-to-image setting, we follow (Esser et al., 2024) using "a photo of class name" as caption.
5.2. Benchmark
We first examine whether adding SPELL post-hoc increases the diversity of trained diffusion models. To this end, we compare each diffusson model to the same model run with the same random generation seeds but with SPELL. In par- ticular, we use intra-batch repellency together with repel- lency from previously generated batches, to enable repel- lency across the up 128 images per prompt/class.
Table shows that SPELL consistently increases the diver- sity, both in terms of recall and Vendi score, across all text- to-image and class-to-image diffusion models. This demon- strates that SPELL works independent of the model archi- tecture and the space the models diffuse in (RGB space for Simple Diffusion, VAE space for all others). Coverage re- mains within -1% +2% of its original value in all models except SD3. The difference between coverage and recall is that coverage uses a more tight neigborhood radius to de- termine whether an image of the original dataset is covered by the generated ones. In other words, coverage measures a form of dataset match, which can counter-intuitivey be decreased by more diverse outputs if the diversity takes dif- ferent forms or is higher than in the reference dataset. This stands out most for SD3, which was not trained on the ref- erence datasets ImageNet- 1ICCC We find that SPELL
In our text-to-image -oooi setup, use SD3, Latent Diffusion (Rombach et al., 2022), and RGB-space Simple Diffusion (Hoogeboom et al., 2023) in resolution 256x256. For the latter two, we use the checkpoints of Gu et al. (2023). De- tails on hyperparameters are provided in Appendix D. We evaluate these models on CC12M (Changpinyo et al., 2021). dataset of (caption, image) pairs, with captions ranging between 15 and 491 characters. As we aim to investigate
whether diffusion models with and without SPELL capture the full diversity of images related to each prompt, we cre- ate a sub-dataset of captions with multiple corresponding images, which gives a ground-truth target diversity. This gives a one-to-many setup with 5000 captions and 4 to 128
6

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Base Model
CADS
PG
IG
SPELL (Ours)
0.34
0.4550
27.85
2.9
0.32
0.4525
27.80
CLIP Score 1
Recall
Coveraae
0.4500
Vendi Score
2.8
27.75
0.28
0.4475
2.7
27.70
0.4450
0.26
0.4425
2.6
27.65
0.24
0.4400
2.5
27.00
0.53
0.54
0.55
0.56
0.70
0.75
27.5
27.6
27.7
27.8
100
110
120
Precision 1
Density f
CLIP Score
FDDINOO2 4
Figure 4. Latent Diffusion on CC12M. The three plots on the left highlight how the hyperparameters of diversity methods trade off image quality (x-axes) and diversity metrics (y-axes). SPELL provides better trade-off than other concurrent approaches. In the rightmost plost, highlighting 2 quality metries, SPELL also shines. IG not visible on all plots as it strongly decreases image quality.
correctly helps SD3 generate images that are generally more diverse, as evideneed by the 26% and 37% increases in the Vendi score, but in other attributes than in the reference datasets, explaining the decrease in coverage. Out of the six experiments, precision and density decrease very slightly in three of them, increase for one, and decrease more clearly in 2 (when using SD3). This trade- off between diversity and precision is common in the litera- ture (Kynkäänniemi et al., 2024; Sadat et al., 2024; Corso al., 2024), and we show in the next section that SPELL provides more favorable Pareto fronts than alternative re- cent methods. This tradeoff improves the overall FDDINO score considerably in Latent Diffusion and MDTv2, while staying within 3% of the original value on Simple Diffusion and EDMv2, and increasing on SD3. Overall, we find that SPELL increases the diversity considerably across all mod- els, with only minor tradeoffs in precision.
sion. -aneele diffusion sampling (Sadat et al., 2024, CADS) noises the text or class condition that guides the CFG, lowering the noise in later timesteps. Closer in spirit our proposal, particle guidance (Corso al., 2024) adds gradient potential to the backward diffusion at every timestep, such that the intra-batch diversity is increased. We reimplement and tune these baselines (Appendix F) to com- pare them against SPELL.
eeference-dataset
Figure 4 shows that SPELL achieves more favorable trade- off curves in three different diversity vs quality Pareto fronts (recall precision, coverage vs density, and Vendi score vs CLIP Score) well in prompt-adherrence quality Pareto front (CLIP vs FDDINOv2). At low repellency radii r, the diversity is achieved without reducing the CLIP score. In Appendix K, we find that SPELL generates diverse images both for short and for longer, more specific prompts.
One reason for SPELL's improved performance sparsity, While other methods change the diffusion trajectories at ev- ery timestep and every image, due to it's ReLU weighting, guidance is not applied if the diffusion trajectories are al- ready heading to a diverse outputs. This leads to increased diversity while leaving high-precision images unchanged. We study this sparsity further the following section.
5.3. Comparison to Other Diversity-inducing Methods
We now consider SPELL's hyperparameter, the repellency radius Intuitively, a small radius will only prevent gen- erating a nearly-same image twice, which increases the di- versity without compromising on the closeness the train dataset (precision/densitty) Choosing to further increase the repellency radius should add more diversity while trading- off precision, as SPELL pushes the repellency trajectories to explore novel modes further outside the train distribution. We find in Figure of Appendix that this intended effect plays out in practice. We also find that there is a sweet-spot for where the diversified samples better reflect the true im- age distribution than the base model, enhancing FDDINov2 and diversity without decreasing precision.
5.4. Sparsity Analysis
This section investigates the dynamics of when and how strongly SPELL'S corrective interventions arise. Fig- ure 5a tracks the magnitude of the SPELL correction vec- tor || Al normalized by that of the diffusion score vec- tor V Po (22) We track this relative magnitude throughout 452 backward trajectories for 50 prompts of CC12M with both intra- and inter-batch repellency (Equa- tions (5) and (7)). Appendix adds further setups with sim- ilar results. We find that the repellency correction typ- ically small. Its magnitude is most often less than 5% of
There are some recent methods enable controlling simi- trade-offs. Namely, Interval Guid- lar (Kynkäänniemi al., 2024, IG) applies CFG in a ance limited time interval in the middle of the backward diffu-

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Rel. Magnitode ReRee lency
0.35
SD1
0.30
0.25
SP23
0.20
5005
0.15
Repelled from
(2]
(3)
(2.5)
[3)
0.10
10
0.05
0.00
0.0
0.2
0.6
0.8
Diffusion Time
(a) Repellency Strength
(3.7)
[.5.6
( [.3]
Figure 6. High-resolution images for the prompt "a dog plays with ball" generated one-by-on by-eby-oee (B 1) with SD3 (top) and SD3 SPELL (bottom), using the same random seeds. While SPELL does not intervene on the first two images, as they are different enough, it intervenes on the 10 marked with orange borders, as they are too similar to previously generated images. This changes the output from what would have been without SPELL (at the top) to slightly or completely novel images the hottom.
Stronger repellency 40)
Roos with Repellency' >
Repellency (r 20)
20 and uverconuensatiin () 1.6)
0.2
0.0
0.0
0.2
0.4
0.8
5.5. Qualitative Example of SPELL Interventions
Diffusion Time 1
This section shows what SPELL's correction terms change in practice. Figure 6 shows 14 images in 1024x resolution, generated iteratively using SD3 with and without SPELL interventions. Images are generated one by one (B = 1), and when generating the i + 1-th image, SPELL repels from the reference set of all images 1 to it has generated thus- far. Images are highlighted in orange SPELL enforced changes during their generation trajectory. When SPELL does not intervene, the SD3 SPELL image (bottom row) coincides with the SD3 output (top row), since same seeds. For images and 2, SPELL did not intervene as it detected that the 2nd trajectory was heading to an image that was novel enough from the 1st. The 3rd image was expected to come out too close to the 2nd at some time during generation, triggering SPELL alter the background, ball color, and details on the dog. As more images added to the reference set, SPELL intervenes more often, For example, image 13 is guided away from an image of a dog on a grassy ground with trees in the background, which images 2, 3, 5, and 11 already show, and explores an entirely novel mode, with both a new dog race and a previously unseen surface.
(b) Diffusion steps with repellency
Figure 5. (a) The gradient that SPELL adds is only a fraction of the magnitude the diffusion's score, thus adjusting without drowning it out. (b) Repellency happens primarily in carly back- wards steps (f E [0.6, 1-0]) and then remains zero, thus making it sparse. Overcompensation allows finishing the repellency earlier, whereas runs with high repellency radius repel longer. Latent Dif- fusion traeectories with intra- batr-batch and previous mevious-batch repellency.
the
that of the diffusion score and never exceeds 35% This ex- plains why our repellency does not reduce image quality or introduce artifacts. A second reason for this is that SPELL corrections happen mostly in the early stages of generation, which literature claims be when the rough image is out- lined, rather than in late steps, where the image is refined (Biroli et al., 2024: Kynkäänniemi et al., 2024). Recall that the backwards diffusion starts at f = and outputs the final image at t = 0. Figure 5b shows that at t = 0.8, only 40% of the trajectories have non-zero repellency term, further declining to 21% at t 0.6. If we impose a higher repel- lency radius r, the repellency acts for longer. Especially in this case, adding overcompensation helps. As intended, the repellency strength is increased and in return stops earlier. These stops are often final: The repellency stays zero for the remainder of the generation, verifying that the trajecto- ries do not bounce back into the repellency radii, as shown in Appendix H. and reaffirming SPELL's sparsity.
similarly strong intervention happens in image 14. SPELL's sparsity means that it not always applied, even when there are already many shielded images. This is the case for image 8, which is novel enough to remain unchanged. Note that none of the images has visual artifacts due to SPELL's early and sparse interventions. Appendices! J and N present hundreds of images generated with SPELL on further models, affirming this finding.
8

---

Shielded Diffusion:
Generating Novel and Diverse Images using Sparse Repellency
Table 2. Without SPELJ,, 7.60% of FDMv2's generated images too close to the protected ImageNet-Ik 1k train set. Adding SPELL re- duces this rate, and the more anferencc--untine is spent on search- ing Voronoi cells (1, 2, 3, 5, or 10), the better the protection be- comes. The runtime is reported on a single A A100-40GB GPU.
EDMv2
samples
ImageNNtt
neighhor
Gen. images within shields
Time per
Model
Prec. f
Recall f
image (8)
EDMv2
EDMv2
7.60%
0.792
0.242
2.43
SPELL
SPELL-1
1.08%
0.792
0,181
4.63
SPELL-2
0.55%
0.788
0.175
6.06
ImageNeet
3PELL-3
0.33%
0.777
0.162
7.79
neighhor
SPELL.5
0.22%
0.771
0.163
9.94
SPELL-10
0.16%
0.768
0.160
13.54
Figure 7. The images generated by EDMv2 in the first row are too close existing images in the ImageNet-11 1k train set in the second row, which EDMv2 was trained on. SPELL ensures that they maintain a protection radius. The images in third row, generated from the same seeds but with SPELL, are sufficiently different from the ImagetNet neighbors in the second row, and also from their own nearest neighbors in the forth row.
5.6. Image Protection at Scale
Last, we present a second use-case of shielded generation, where the goal is to create images that are sufficiently novel from given large set of images. In particular, we scale SPELL to shield all 1.2 million ImageNet- 1k train images by employing approximate nearest neighbor search (Douze et al., 2024, see Appendix l). We then generate 50k images and track how often they fall into a protected shield.
mantic spaces, c.g., by comparing the DINOv2 embeddings of expected image outputs, which we leave for future works.
6. Discussion
Table 2 shows that 7.6% of the 50k images that MDTv2 gen- erates without SPELI are within an L2 distance of r = 60 of their nearest neighbor on ImageNet. Figure 7 shows examples and verifies that such images are indeed nearly copies of existing images. Adding SPELL with reduces this rate down to 0.16%. Figure 7 shows that the images are indeed not too close to their ImageNet neigh- bors anymore. However, the runtime increases. This is not due to SPELL-in all previous experiments with K = 128, SPELL does not increase the runtime, see Appendix G- but due the approximate nearest neighbor search algorithm over the K 1.2M images that we implement on CPU. Re- ducing the number of Voronoi cells that the nearest neighbor algorithm searches for shields allows to speed up the gen- eration time, at the cost of a catching less shields. Further improvements in Le based neighbor search techniques will further increase the protection rate and compute overhead.
We introduce sparse repellency (SPELL), a training- free post-hoc method to guide diffusion models away from set of images. SPELL increases diversity by preventing repeat generation, and allows protecting set of given reference images. SPELL can be applied any diffusion model, whether it is class-to-image or text-to-image, and whether it is unconditional or glassifier(-free guided, even at high resolution and arbitrarily sized reference sets.
60
We find three limitations of SPELL: SPELL currently only guarantees generating images outside the shields if all shields are disjoint. If there are overlap- ping shields and the diffusion trajectory heads exactly into the middle of them, their repellency terms cancel. While we do not see this problem in practice even the scale of 1.2M images, in theory, the algorithm could be improved: we can merge overlapping shields, or search for direction that points out of the convex hull of all shields rather than their naive sum of parts, at the cost of scalability. Second, we currently apply SPELL with respect to the L2 distance inside the diffusion models' (VAE encoder) latent spaces, which lead to visually different outputs. Checking the dis- tances inside a downstream semantically structured embed- ding space and propagating their directions back could lead to generating more semantically different images. Third, SPELL is applied to expected E [zz] under the forward joint distribution and not samples from Poj: required for Doob h transform. see Appendix A. This expectation will only correspond to the generative distribution if using the time-reversal diffusion, and not for example when using the probability flow ODE sampler.
First, theoretically,
We take one final look at precision and recall in this special use-case, where the goal is to generate images that are simi- lar but not equal to the train images. Expectedly, the recall over the validation images decreases when repelling from all training images, because validation images may fall into the shield radii around train images. However, the precision remains largely unaffected. This demonstrates again that SPELL stays on the image manifold, even when repelling from many images. Finally, the last two images in Figure give more insight into the workings of the I.2 similarity in the VAE latent space that MDTv2 diffiuses in. Apparently, image distances inside the VAE space encode a visual simi- larity where images with similar colors and compositions are close to one another. SPELL could also create shields in se-

---

Shielded Diffusion:
Generating Novel and Diverse Images using Sparse Repellency
Impact Statement
Chung, H., Kim, J., Mecann, M. T., Klasky, M. L., and Ye, J. C. Diffusion posterior sampling for general noisy in- verse problems. In The Eleventh International Confer- ence on Learning Representations, 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Corso, G., Xu, Y., Bortoli, V. D., Barzilay, R., and Jaakkola, T. S. Particle guidance: non-i..d. diverse sampling with diffusion models. In International Conference on Learn- ing Representations (ICLR), 2024.
References
Anderson, D. G. Iterative procedures for nonlinear integral equations. Journal of the ACM (JACM), 12(4):547-560, 1965.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Biroli, G., Bonnaire, T., De Bortoli, V., and Mézard, M. Dynamical regimes of diffusion models. arXiv preprint arXiv:2402.18491, 2024.
Denker, A., Vargas, F., Padhy, S., Didi, K., Mathis, S., Dutordoir, V., Barbano, R., Mathieu, E.,, Komorowska, U. J., and Lio, P. Deft: Efficient finetuning of conditional diffusion models by learning the generalised h-transform. arXiv preprint arXiv:2406.0177
Bradley, A. and Nakkiran, P. Classifier-fre free guidance is predictor-corrector. arXiv preprint arXiv:2408.0900 09000, 2024.
2024.
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. 32nd USENIX Security Symposium (USENIX Security 23), pp- 5253-5270, 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H. The faiss library. 2024.
Chamberlain, B., Rowbottom, J., Gorinova, M. I., Bron- stein, M., Webb, S., and Rossi, E. Grand: Graph neu- ral diffusion. In Meila, M. and Zhang, T. (eds.), Pro- Ma-
Efron, B. Tweedie's formula and selection bias. Jour- nal of the American Statistical Association, 106(496): 1602-1614, 2011. doi: 10.1198/jasa.211 201 1.ml 1181. URL https: /dw.. org/10.../... 1198/ jasa 2011.
ceedings of the 38th International Conference chine Learning, volume 139 of Proceedings of Machine Learning Research, 1407-1418. PMLR, 18-24 Jul 2021. URL v1 39 cwwmberelbbaeceec n22 html.
tm11181. PMID: 22505788.
www.mll...e.... mlr.press/
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K. P., Freeman, W. T., Rubinstein, M.. et al. Muse: Text-to-image generation via masked generative transformers. In International Confer- on Machine Learning, pp. 4055 4075. PMLR, 2023.
Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
Changpinyo, S., Sharma, Ding, N., and Soricut, Con- ceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Computer Vi- sion and Pattern Recognition (CVPR), 2021.
Gao, S., Zhou, P., Cheng, M.-M., and Yan, S. Mdtv2: Masked diffusion transformer is a strong image synthe- sizer. arXiv preprint 1rX1v 2023.
Chen, C., Liu, D., and Xu, C. Towards memorization- free diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1422588434, 2024.
Gu, J., Zhai, S., Zhang, Y., Susskind, J. M.. and Jaitly, N. Matryoshka diffusson models. In International Confer- ence on Learning Representations (ICLR), 2023.
Haussmann, U. G. and Pardoux, E. Time reversal of diffu- sions. The Annals of Probability 14(4):1188-1205, 1986.
Cho, J., Zala, A., and Bansal, M. Dall-eval: Probing the rea- soning skills and social biases of text-to-image generation models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 3043-3054, 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update
10

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable diffu- sion models. In European Conference on Computer Vi- sion, pp. 423 439. Springer, 2022.
Ho, J. and Salimans, Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., and Yoo, J. Re- liable fidelity and diversity metrics for generative mod- els. In International Conference Machine Learning (ICML), 2020.
Ho. J. Jain, A., and Abbeel, Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 6840-6851, Curran Associates, Inc., 2020. URL
Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and edit- ing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
cc/
rebbcfecssaaf0006711111011118660 pdf.
Hoogeboom, E.,, Heek, J., and Salimans, T. Simple diffusion: End-to-end diffusion for high resolution images. In Inter- national Conference Machine Learning (ICML), 2023.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I,, Rab- bat, M. Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. DI- NOv2: Learning robust visual features without supervi- sion. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Jeha, P.. Grathwohl, W., Andersen, M. R., Ek, C. H., and Frellsen, J. Variance reduction of diffusion model's gra- dients with taylor ipproximationnaaseee control variate. arXiv preprint arXiv:2408. 0104400../22244 12270, 2024.
Jo, J., Lee, S., and Hwang, S. J. Score-based generative modeling of graphs via the system of stochastic differen- tial equations. In International conference on machine learning, pp. 10383, PMLR, 2022.
Robbins, H. E. An empirical bayes approach to statistics. 1956. URL https: //s:://aa /tppp/app../a/// semant icscholar. org / /ttpC/oooSI::21111....
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, Analyzing and improving the training dynamics diffusion models. In Computer Vision and Pattern Recognition (CVPR), 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Om- mer, B. High-resolution image synthesis with latent dif- fusion models. In Computer Vision and Pattern Recogni- tion (CVPR), 2022.
Kim, M., Kim, D., Yusuf, A., Ermon, S., and Park, M. Training-fre free safe denoisers for safe use of diffusion mod- els. arXiv preprint arXiv:2502.080// 2025.
Sadat, S., Buhmann, J., Bradley, D.,, Hilliges, O., and Weber, R. M. CADS: Unleashing the diversity of diffusion mod- els through condition-anneales sampling. In International Conference on Learning Representations (ICLR), 2024.
Kingma, D. and Gao, Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2023.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif-- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Koulischer, F., Deleu, J., Raya, G., Demeester, T., and Am- brogioni, L. Dynamic negative guidance of diffusion models. arXiv preprint 2:xxvv:240.143330 2024.
Salimans, T. and Ho, J. Progressive distillation for fast sam- pling of diffusion models. In International Conference Learning Representations, 2022.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. Advances in neural information processing systems, 32, 2019.
Särkkä, S. and Solin, A. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Kynkäänniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint rrXiv:2404.07724 2024.
Shen, X., Du, C., Pang, T., Lin, M., Wong, Y., and Kankan- halli, M. Finetuning text-to-image diffusion models for fairness. In The Twelfth International Conference

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Learning Representations, 2024, URL https://
openreview. net.
Shipard. J., Wiliem, A., Thanh, K. I., Xiang, W., and Fookes, Diversity definitely needed: Improving model-agnostic zero-shot classification via stable diffu- sion. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 769-778, 2023.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusion digital forgery? investigat- ing data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6048-6058, 2023.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A.., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Represenlations (ICLR), 2021.
Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Ross, B., Villecroze, V., Liu, Z.. Caterini, A. L., Taylor, and Noaiza-Ganemi G. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Thornton, J., Béthune, I.,, ZHANG, R., Bradley, A., Nakki- ran, P., and Zhai, S. Composition and control with dis- tilled energy diffusion models and sequential monte carlo. In International Conference on Artificial Intelligence and Statistics, pp. 3259-3267. PMLR, 2025.
Vincent, P. A connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661-
1674, 2011.
Wang, X., Dufour, N., Andreou, N., Cani, M.-P., Abre- vaya, V. F., Picard, D., and Kalogeiton, V. Analysis of classifier-free guidance weight schedulers. arXiv preprint 1:Xiv:2404.130400 2024.
Zhang, L., Rao, A., and Agrawala, M. Adding condi- tional control to text-to-image diffusion models. In 2023 IEFF/CVF International Conference Computer Vision (ICCV), pp. 381338133333 33-3824. IEEE Computer Society, 2023.
12

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
A. Guidance via Doob's h-transform
Doob's h-transform provides a definitive approach to conditioning and guiding diffusions. the context of avoiding points let U, where B = {a E X' || 2 z I r) are balls of radius around centers (zx)&. We may approximate Doob's transform with some simplifying assumptions based on diffusion posterior sampling (DPS (Chung et al., 2023)). DPS entails approximating Poju( m²) with Po (x))), where D(xa) E[X, ax] for some choice of density p.
Let us observe that
V log Poje (Xo (Xo V log Poje (Xo & Ug [1)=
$ BB xt).
For simplicity, we approximate the conditional density Pojel- ax) with Gaussian density with mean E[X, xxr] and variance Σ, = ld. Since, for X ~ (W, Id), the random variable ||XX - =x follow non-centered chi-square distribution (A) where À == A(u)= = | 2 11? As such
PF Xac (2) (n2)
V Pogg (o & BB 14)
(11- 2x) x ((((H)),))
1 F2z (1)
with weights function
OF (n) (12).
w(, Fx2 X2 (2) (2)( - X
ax
We recall that the CDF of a is combination of the CDF of standard x3:
Fx (2)(()) Σ
(A) (x)
1=0
OF (1))
(x) = Σ cj(A)
F F2 (x) - (F) (x)
01
2 1=0
0F
OF (x) 2(1 2x)
V μ F 2 (2) (z)
(z)
(x) x 0 01)(1)
0x
0x
where we denoted (A()) (2/2)
3!
13

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
B. Repellence Guarantee
Consider again our adjusted SDE
dx = ((x,1)-92(1)) (v
K
1) di + g(t)dw,
Va logg(()) +
at
@0 zn ReLU
r
|.00 2イ
where Zu = #etof (err is the expected image if we did not intervene.
as This section shows that the SDE leads to a output distribution with Po( L..., K3) = 0. This ensures that it does not create samples within radius r around the repellence images 2g, k = 1,.... . K. To this end, assume we have a set of repellence images and that their repellence balls do not overlap (otherwise, one can merge them and select an according higher radius). Let's consider an arbitrary timestep I. Then Tweedie's formula (Efron, 2011; Bradley Nakkiran, 2024) gives that
3t + of V logpt(t) + St Ek=1 (go - 2k) ReLU
(1
i00-2al12
E [[oo| =
3t + V log (x))
of at
Σ (20 2hd ReLU
K
r T
oi
7
zx|12
k=1
K
Cu +
(go =x)
ReLU
|100
2xl12
Case 1: |1ā0 2x|12 rkkk 1.... K.
Then the ReLU term becomes 0 and ft remains unadjusted and | [,/x,]
Fy 1x 2 r.
Case 2: 3k* E {l,..., K} 180 zille < <. Since the balls are non-overlapping.
K
Σ
(Bu (bu
2k) ReLU |100
1) (fo =x) ReLU
1100 1180-2211 22112 -11-11 -1)
2x 2
Then
[[[[o]
=xll2 = |1ã0 + (āo =x) ReLU
1
=fll2
|
zil12
( =f) + (ão =f}
1)112
| - 2%
(1 ={) + (ão =f)
(5o
=2)112
|.0
||r (āo 2f)
1|2
1 = 2
So, in all cases, | X [r]] - 2x12 2 for any 4. Especially, for / = 0. the SDE does not add any noise anymore and the sampled un is equal to the expectation.
Hence || 2ul12
1,...,K K.
14

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
C. Conservative Field Interpretation
The function k((x) = Trelu ( | 1)ar is the conservative field associated to (family of) potential R2 - I::
||| | when |>|||
(x)
2 otherwise,
where Gauge (0) is chosen arbitrarily, as illustrated in Figure 8.
Furthermore, observe that the mapping 24 + 2o defined by ft = 2 (o logp:(()) + xx) is a conservative field given by the potential Cgg (of (e =111)1
Therefore, SPELL guidance in Equation 5 is the composition of two conservative fields. But note that, in general, conservative fields are not stable by composition, unless the Hessians of their potentials commute everywhere.
Theorem C.I. consider f TRd R a twice difserentiable function. The Jacobian of the map & x
V/(a) [V
given
by
1 Jac(o)(x) = Illl |
H 99" H. with V f(x) and H f2)(
Hence, in the case where H and gg" commute, this Jacobian is locally symmetric. If they commute everywhere, then this Jacobian globally symmetric, and & is a conservative field.
0.5
0.4
0.3
0.2
0.1
0.0
0.1
1.00
1.00
-0,5
0.75
-050
0.50
0.25
0.25
0.000
0.00
1/g
0.25
-0.25
0.50
-0.50
0.75
-0.75
1.968.00
Figure 8, Potential function whose gradient field is drelu | 1)x, displayed for E R² Repellence force is dynamic: closer to the center (i.e., when diffusion trajectory expected close to protected image) it applies stronger gradients, shown by the arrows, while outside the repellecy radius, applies no gradient at all, letting the diffusion trajectory continue without any intervention.
15

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
D. Implementation Details and Hyperparameters
Since SPELL a training-fres post-hoc method, we use the trained checkpoints of diffusion models provided by their original authors. For EDMv2 and MDTv2, we use the hyperparameters suggested by their authors. Latent Diffusion, Simple Diffusion, and Stable Diffusion come without recommended hyperparameters, so we tune the classifier-free guidance (CFG) weight by the F-score between precision and coverage the 554 validation captions
CC12M split.
For the repellence radius the latent spaces that the different models diffuse on have different dimensionalities, hence the scales of the repellence radii differ. To get a sense of the scales, we first generate one batch of images without repellence and tracked the pairwise L2 distance between generated latents at the final timestep. We then test 16 values from 0 to times the median distance. This yields the following hyperparameters for the results in Table
EDMv2: CFG weight 1.2, 50 backwards steps, miin 0.002, max 80, p 7, Smin 0, Smax 30, repellence radius 20, batchsize 8.
MDTv2: CFG weight 3.8, 50 backwards steps, repellence radius = 45, batchsize 2.
Stable Diffusion 3: CFG weight 5.5, 28 backwards steps, repellence radius r = 200, on CC12M overcompensation 1.6 (no overcompensation on ImageNet), batchsize 8.
Simple Diffusion: CFG weight 5.5, backwards steps, repellence radius 50, overcompensation 1.6, batchsize 16.
Latent Diffusion: CFG weight 5, 50 backwards steps, repellence radius 20, overcompensation 1.6, batchsize 8.
Algorithm I gives high-level pseudo-code for SPELL and Algorithm 2 details how we implemented SPELL in parallelized way in Python.
Algorithm SPELL added the backwards diffusion step. This a simplified example, see Appendix D for Python code that is parallelized and supports sparse neighbor retrieval.
Input: Batch of latents {x/ 16m1..... set of shielded images {zu (zm,k., radius X for = I,..., B do
4o + Dg- (t, (t))
(b)
Expected diffusion output without repellency
end
for b 1..... B do
Ag 0 for k: 1,..., K do
b Repel from the shielded set
if 10 - =+ll2 < r then
A A. + +1/(0) 2x) ReLU
1
2
1 (b)
=2
end
end
for 6'
l,... .,..,,,,'' # b do
D Repel within the batch
if
1 (b)
(6) 1
|| < r then
A A (=( =00²)
ReLU
|60 -2!" 1 '112
end
end
Calculate 2012) by taking a step towards (0(0) XI (using the diffusion scheduler)
10
end
Output:
E. Construction of the Soft-label CC12M Dataset
CC12M is a recent text-to-image dataset that contains pairs of image links and the title scraped from their metadata. To turn this into our soft-label subset of CC12M, where each caption has set of multiple possible images related to it, we first group all images in CC12M by their caption and keep only captions with at least four and at most 128 images.
Some of these images are falsely grouped together. For example, there are photo albums whose images were assigned the
16

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
def beckneniLetepp..t. t, protection_set, Lambda, sepel_.tthir_bsrch))
generation step from t-1 difusiio with repellancy
XLt: Natrix size [batch, dimensions] containing the current latenta
Di float, current time
protect Either mazcix size [num prutppooooiion dimensions] with labents we want Lo xepel From dstabase that will output closest neighbors in this format radius: Flost, repellancy radius
lanbda: Flost, overoompensation factor
withn_ Boolean, whecher to apply intra batch repellency
en*
Predict x_0 using the diffusion model (using diffusion without zepellency) x_O_hat diffusion__rore peeererpreeeeiie stffusioo_.ooro.peeiiiit... ix_t,
Repel protection
tepellency_Les -
protection_set
None:
if Leinstance (protection_sel, daLabase):
protection_set protection_set.. htt.://s...........iiieeeeeeiiia diff_ve= x_0 =____attttusuuuueeoeeee wns.uuoooe # d' ff haa size (batch, nun umm_protecttor for__mages, dimensiona]
1x___hat, redius)
unsqueezzzze
weight (diff_vesvA2) (dimm=)=)) (s trune_weight ReLU diff -1]
repellency_term (diff_ved
Repel within batch
if sypel_within_aae difr_veo =
unsqueeze (0)
diff_vec
[barch, batch, dimensioas]
weight (diff_vezr*2) (dmm))))))) .sgemt)))))
trunc_ weight Relu(radisa diff
slag(ttrun_weeighh)) = Don't repel from the image itself repellency_term += (diff_ved trune_weight)..mmmtan
Add our repellency term to the current x_D_hat prediction
x x_t_hat x_0_hat lambda zepellency_term
Step from to t-1 vsing diffusion update tsame as typtoss di diffusion) x_t_minus_l (aloclate__pdatteee_2MM.
t)
if t 0:
x_t_minua_1 generate_rofsett
retura x_t_minus__
Our repellency added the backwards algorithm of existing diffusion models, without retraining. Since the expected x_C hat is often already computed as part of the backward process, the only runtime overhead are the pairwise differences and the possible neighbor search.
Algorithm 2:
generic title in their metadata. A useful heuristic filter out such cases is analyze the top-level domains of the images. We filter out where the most frequent top-level domain belongs to 75% more of the image urls. Second, we filter out automatically generated captions by removing captions that include the strings 'Display larger image', 'This image may contain', 'This is the product title', or 'Image result for'. Last, due to privacy guidelines, we filter out any caption whose image may include individuals. This done by filtering out caption that include <<ERSON>> which is a placeholder that the CC12M dataset overwrote any possible person name with. After these filtering steps, we arrive at 5554 captions. We randomly split them into validation set of 554 captions and a test set of 5000 captions. Table shows how many images belong to each caption.
We did not filter any images although there some duplicupl This is done purpose in order to not skew the distributions. Filtering out captions amounts to deciding which subset of the dataset we test our models on. But filtering out images would change the conditional distributions P(X le) to something different from the training distributions. In other words, a model that learned the train distribution ideally is expected to have a stronger mode at near-duplicate images but testing it on changed P(X c) distribution would punish it for learning the correct distribution. If future work intends to test models on unseen images, we note that removing near-duplicates may be a possibility, depending on the experiment design.
17

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Table Number of captions that have certain number of images attached to them in our soft-label CC12M dataset.
Images per caption
Validation split
Test split
4
270
2600
6- 10
174
1485
11 - 20
75
555
21 30
20
219
31 40
10
86
41 - 50
32
51 - 128
23
F. Further Diuarsyty-quality Tradeoffs
In addition to the tradeoff experiments in Section 5.3, Table 4 provides the full combinations of metrics attainable with each method, depending on how one chooses the hyperparameters. This is the raw data underlying Figure 4 and allows the curious reader to compare arbitary tradeoffs.
G. Runtime Analysis and Comparison
The scale of the overhead that SPELL adds is negligible when contrasted with the diffusion generation cost. It amounts to computing (up to) [B, K distance matrices per time I, where both the batchsize B and the size of the protection set K do not exceed hundreds, and adding one single correction vector to the score. Table 5 confirms that the runtime that SPELL adds (as well as the other benchmarked diversity methods) is negligible, here using B = 8 and intra- batch repellency, hence K = B 1 = 7. This also further confirms that the runtime observed in Section 5.6 is due to the next-neighbor search algorithm, not SPELL'S correction terms.
H. Ablation: Repellency Strength Throughout the Generation
In this section, we scrutinize how and when repellency acts during the generation. We also use these insights to run ablations that foster the intuition on the role of the repellence radius.
To begin with, Figure shows repellency in the standard setting with a repellency radius of 25 in Latent Diffusion. We first generate 8 images per prompt, and then generate another 8 images that repel from the first ones, without intra-batch repellency. Figure 9a shows how high the L2 norm of the total gradient is that our repellency adds to the score, divided by the La norm of the score. It can be seen that the repellency term is in most cases most 20% strong as the original diffusion gradient field. Intuitively, this means that our repellence does not drown out the diffusion model, but is more a corrective term. Repellency mostly takes place early in the backwards diffusion (t [0.6, 1.0)), with Figure 9b demonstrating that more than 50% of the generations have already finished their repellency in the first quarter of timesteps (note that Latent Diffusion uses lincarly scheduled timesteps). This leaves sufficient time for the diffusion model to generate high quality images in the remainder of steps.
Figure 10 uses intra-batch repellency instead of repelling from 8 previously generated images. The dynamics are very similar Figure (see also the comparison in Figure 15). This shows that our repellency smoothly can be used both intra- batch iteratively, or in a mixture of both, to generate arbitrary amounts of diverse data even when GPU memory is limited. This mixed setup is presented in Figure 11, where we generate two images at time that repel both intra- batra-batch and from the previous images. It behaves similarly in both magnitude and duration of repellency. Figure 12 further investigates scalability. Despite repelling from 64 previously generated images, the repellency magnitudes and times are only slightly increased compared to Figure 9. Note that this is despite generating 64+8 images conditionally on the same prompt, repellency from a dataset of more various images like in Section 5.6 even less effected.
If repellency needs to protect large radius, the repellency takes place longer in the backwards diffusion process, as shown in Figure 13, where we use an increased radius of 37.5. Here, 43% of the backwards diffusions apply repellency until the end of the generation. The repellency magnitude is increased but still stays below 50% of the magnitude of the diffusion score. One option to speed up the repellency if it runs until the end like here is overcompensation. Figure 14 shows that compared to Figure 9, the repellency is stronger at start and manages to push the trajectories into the diffusion cones of different modes, return allowing to stop the repellency earlier. This implies that overcompensation can also be used as a
18

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Table Metries of all approaches in the tradeoff experiments Figure 4.
Vendi
Coverage
Precision
Density
F FDnno
Base Model
0.237
2.527
0.446
0.558
0.768
9.566
105.96
27.789
Partiele Guidance, strength = 1024
0.099
1.987
0.249
0,300
0.326
84.115
705 661
24.470
Particle Guidance, strength
26.740
Particle Guidance, sirength 256
0.252
2.656
0.429
0.523
0.682
11,934
154,897
27.440
Particle Guidance, strength 128
0.248
2.591
0.447
0.553
0.754
9,442
109.257
27.704
Particle Guidunce, strength
0.245
2.561
0,449
0.559
0.771
9.072
101. (01.7966
27.781
Partiele Guidance, sirength
0.235
2.528
0.445
0.557
0.763
9.724
108. 382
27.812
Partiele Guidance, strength 16
0.236
2.529
0,446
0.557
0.764
9.596
107 04.04
27.813
Guidance, [0. 1.0.9
0.372
Interval Guidance, [0,2,0
0.419
2.994
0,442
0,514
0.689
8.359
85 09.
26.813
Interval Guidance. [0.1.088]
0,470
3.174
0.448
0,500
0.663
7.507
76.104
27.215
Interval Guidance. [0.3,0.9]
0.471
3.208
0.421
0,483
0.635
8.406
87.971
25.885
Interval Guidance. (0.2,0.8)
0,518
3,340
0.434
0.478
0.624
7,478
75.250
26,544
Guidance, [0.1,0.7|
0.567
26.784
Interval Guidance, [0.4.0.9)
0.525
3.495
0.395
0.442
0.569
8.623
96.611
24.630
Guidance, (0.3.0.8]
3.575
Interval Guidance, [0.2,0.7]
0.614
3,770
0.417
0.426
0.536
6,771
72.972
25.979
Interval Guidance. [0.1.0.6]
0.673
4.138
0.396
0,385
0.466
6.885
81.643
26.020
CADS, mixture factor = 0, 0.6
0.262
2.598
0.447
0.553
0.753
9.248
105 000
27.746
CADS, mixture
0.253
27.757
CADS, mixture factor = 0, =0=08 =088
0.245
2.561
0,449
0,557
0.762
9,356
105, 856
27.771
CADS, mixture factor 71
0.239
2.545
0.450
0.559
0.767
9,452
106.455
27.790
CADS, mixture factor 0.001.
0.6
0.325
2816
0,442
0,531
0.696
8.897
105,081
27.534
CADS, mixture factor 0.001,
73 0.7
0.297
2.734
0.446
0.540
0.719
8.963
104.006
27.617
CADS, mixture factor 000001,
0.277
2.600
0,447
0.548
0.739
0.098
103,766
27.697
CADS, mixture factor 0.000.
0.9
0.256
2,588
0.448
0.554
0.755
9.273
105.268
27.754
CADS, mixture factor
CADS, mixture factor 0.002.
0.7
0.380
3.028
0,429
0,501
0.637
9,143
114.333
27.242
CADS, mixture facter 0.002. T1
0,330
2,837
0.442
0.529
0.692
8,893
105.511
27.506
CADS, mixture factor 0.002,
0.9
0.277
2.660
0.446
0.548
0.739
9.098
103,762
27.696
140.841
SPELL, shield radius
0.359
2.935
0.445
0,518
0.665
11.452
120.346
27.556
SPELT
SPELL, shicld radius 25
0.312
2.774
0.454
0,542
0.723
9,794
100.123
27.739
SPELL, shield radius
0,287
2.691
0.455
0.552
0.746
9.535
98.666
27.781
SPELL, shicld radius
0.263
2.616
0.454
0.762
100.709
27.811
means to realize higher repellency radii, without needing to repel until = We leave this, and possibly expansions with overcompensation or repellency radius schedulers, for future works.
0.7
Rel. Magnitude of Repelleney
0.6
% of Runs where Repellency
0.8
0.5
0.4
0.8
0.3
0.2
0.2
0.1
0.0
0.0
D.4
Diffusion Time
Diffusion Time
(a) Repellency strength per trajectory
(b) Diffusion steps with repellency
Figure 9, Generating images that repel from protected images (generated with the same prompt). Latent Diffusion, 256 generations
total.
19

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Table Generation times per image. Neither SPELL other diversity inducing methods add considerable runtime. The runtime
dominated by the diffusion backbone. Mean + standard deviation across 500 images, run on an NVIDIA V100 GPU.
Model
Generation time per image (seconds)
Baseline (Simple Diffusion)
2.93 + 0.12
Simple Diffusion PG
2.96 0.13
Simple Diffusion IG
2.93
Simple Diffusion + CADS
2.96 + 0.12
Simple Diffusion SPELL
2.94 + 0.13
1.0
Rel. Magnitude of Repellency
% of Runs where Repellency
0.6
0.8
0.5
0.6
0.4
0.3
0.2
0.1
0.0
0.0
0.2
0.4
0.2
0.4
Diffusion Time
Diffusion Time
(a) Repellency strength per trajectory
(b) Diffusion steps with repellency
Figure 10. Generating images with the same prompt in batches of with intra-batch repellency. Latent Diffusion, 256 generations total.
Image Protection on Large Datasets
Image protection involves computing the repellence between the current batch being generated with large dataset D of size N, with N > 10 This dataset will be typically too large to fit entirely in GPU memory. Furthermore, computing the repellence term of each element of the batch with every element of the dataset would be prohibitive. However, since the repellence term is zero for vectors that are far-away, this opens the possibility of an optimization: first, the closest images from the batch are retrieved using a vector similarity index (stored in RAM), and only then these images are moved into GPU memory for the actual computation of the repellence term. An efficient implementation of this technique is provided by the Faiss library (Douze al., 2024). We the IndexIV object, that rely Voronoi cells to cluster vectors and speed-up search. We chose a number of Voronoi cells equal to the square root of dataset size, i. 1 1131 cells containing typically 1132 examples each. During generation, we probe only the two voronoi cells closest to the current expected outputs. The behavior of the repellence term ensures that false positive are rarely a problem. False negatives (if any) are typically "far-away" which means that their contribution to the sum of all ReLU repellency terms would have been small. In Table 2, we show that one Voronoi cell is often enough. Searching the ten closest cells gives an even higher protection rate, though the cost of higher searching costs. This shows that advances in efficient search algorithms will directly benefit SPELL when is applied large repellency sets.
20

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
0.35
1.0
Rel Magnitude of Repellency
0.30
% of Runs where Repellency
0.25
0.20
0.6
0.15
0.4
0.10
0.05
0.00
0.0
0.2
0.6
0.3
0.2
0.4
0.6
Diffusion Time
Diffusion Time
(a) Repellency strength per trajectory
(b) Diffusion steps with repellency
Figure 11. Generating images iteratively, generating 2 images time. They repel both intra-batch and from the previously generated images. We use 50 different prompts, generating 4-32 images each, giving a realistic setup. Latent Diffusion, 452 generations in total.
1.0
Rel. Magnitude of Repellency
1.6
1.4
% of Runs where Repellency
1.2
0.6
0.8
0.6
0.4
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
D.4
0.6
0.8
1.0
Diffusion Time
Diffusion Time
Repellency strength per trajectory
(h) Diffusion steps with repellency
Figure 12. Generating images that repel from 64 protected images (generated with the same prompt). Latent Diffusion, 256 generations in
total.
1.0
Rel. Magnitude of Repellency
% of Runs where Repellency =
0.8
1.5
0.6
1.0
0.4
0.0
0.0
0.4
0.4
Diffusion Time
Diffusion Time
(a) Repellency strength per trajectory
(b) Diffusion steps with repellency
Figure Generating images that repel from protected images (generated with the same prompt), using a 1.5 times larger repellency radius. Latent Diffusion, 256 generations in total.
21

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
1.0
Rel. Magnitude of Repellency
10
% of Runs where Repellency
0.6
0.4
0.2
0.0
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Diffusion Time
Diffusion Time
(a) Repellency strength per trajectory
(b) Diffusion steps with repellency
Figure 14, Generating images that repel from 8 protected images (generated with the prompt), with overcompensation factor of 2. Latent Diffusion, 256 generations in total.
% of Runs that Finished Repellency
1.0
0.175
Previous 8 images repeliency
0.150
Mixed repeliency
Intra-batch repellency
% of Runs where Repellency
0.8
Previous images repellency
Stronger repellency (1=4))
0.100
Overoonipensatio
Previous imaget repellency
0.075
0.4
Intra butch repellency
0.050
Mixed repellency
0.2
Previous 64 images repellency
0.025
Smrnger repellency (m-40)
Overcompennationn
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Diffusion Time
Diffusion Time
(a) Mean repellency strength per timestep
(b) Diffusion steps with repellency
Figure 15. Comparison of the previous ablations.
Runs Finished Repellency
1.0
0.8
images repellency
0.4
Intrabbtch repellency
Mixed repellency
Previous 64 images repellency
Stronger repellency (r=40)
Overcompensation
0.0
0.2
0.4
0.6
088
1.0
Diffusion Time
Figure 16. Timesteps at which the repellency has finished, in that the term is zero and stays zero for the remainder of the generation.
22

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
J. Examples of Images Generated with Repellency
lce CAO
STLE
PEEST
S
FRE
88:33
Figure 17. Randomly chosen images where repellency actively pushed EDMv2 away from the protected ImageNet- Ik train in Section 5.6. All images have repellency applied to them but do not show visual artifacts. Low-quality images are by design because the underlying EDMv2 model learned to generate this style of images from the ImageNet-ID 1k train dataset.
23

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
K. Ablation: SPELL on Different Prompt Lengths
To verify that SPELL does not only increase the diversity of short prompis, where is easier to find different images fitting to prompt, we stratify our analysis by prompt length. Fig. 18 shows that the SPELL increases the diversity, here of the Latent Diffusion model, throughout all prompt length quantiles. Notably, this is on a relatively small repellency radius r. which does not decrease the prompt-adherent
measured by the CLIP score.
Base Model
29.0
Base Model
SPELL (Ouns)
SPELL (Ours)
28.5
0.30
CLIP-Score
28.0
Recal
27.5
0.25
27.0
26.5
26.0
25.5
736
27
31
35
58
80
123
491
27
31
47
58
80
491
Prompt length (charactens)
Prompt length (characters)
(a) Recall
(b) CLIP-Score
Figure 18. Diversity and prompt-adherence for short and long prompts. We split the CC12M prompts into 10 categories based on the number of characters. SPELL achieves consistently higher diversity than the baseline Latent Diffusion model both for short and long prompts, while maintaining the same CL.IP-ccoee as the baseline model. An example for a short prompt is "Bird on tree hranch" whereas long prompts include "Head Medusa, creature Greek mythology. pieces made by hand with goldsmiths and metals such as gold and copper. wears a helmet of green and gold snakes". Errors bars denote the standard error over 5 seeds.
L. Ablation: Changing the Guidance Weight
In this section, test if the diversity improvements can be achieved by changing the classifier-fres guidance weight. We find that it does improve diversity, however adding our SPELL top consistently increases the performance further. We use the same SPELL hyperparameters as in the main paper for Latent Diffusion, namely 20 and overcompensation 1.6.
Base Model
SPELL (Ours)
3.4
Vendi Score
3.2
3.0
2.8
Guidance Weight
Figure Our repellency added to Latent Diffusion model different classifier-free guidance weights.
24

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Base Model
SPELL (Ours)
0.5
Recall 1
0.4
0.3
3
Guidance Weight
Figure 20. Our repellency added to Latent Diffusion model with different classifier-free guidance weights.
0.460
Base Model
SPELL (Ours)
0.155
Coverage f
0.450
0.445
0.440
2
3
7
Guidance Weight
Figure 21. Our repellency added to the Latent Diffusion model with different classifier-free guidance weights.
0.56
0.54
Precision f
0.52
0.50
0.48
Base Model
SPELL (Ours)
0.46
Guidance Weight
Figure 22, Our repellency added to the Latent Diffusion model with different classifier-fres guidance weights.
25

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
0.750
Density f
0.700
0.673
0.650
0.625
0.600
Base Model
0.575
SPELL (Ours)
Guidance Weight
Figure 23, Our repellency added to Latent Diffusion model with different classifier-free guidance weights.
10.0
Base Model
SPELL (Ours)
9.3
9.0
FID
8.5
8.0
2
3
Guidance Weight
Figure 24. Our repellency added to the Latent Diffusion model with different classifier-free guidance weights.
120
Base Model
SPELL (Ours)
FDDINOv2
100
80
Guidance Weight
Figure 25. Our repellency added to the Latent Diffusion model with different classifier-fres guidance weights.
26

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
M. Ablation: Changing the Repellence Radii
3.0
0.56
FDDINO<1 +
Vendi Score 1
0.35
1:00
Precision t
Recall t
0.54
2.8
0.30
0.52
2.6
0.25
0.50
20
40
20
40
20
40
20
40
Repellence radius
Repellence radius
Repellence radius
Repellence radius
Figure 26. Effect of SPELL's hyperparameter on Latent Diffusion metries on CC12M. A small radius (r = 15) improves the Vendi score, recall, FDDINov2 without compromising precision. The radius can be further increased trade-off precision for additional diversity.
N. Further Diversity Examples
In order to extend Figure 1, we provide further examples of Simple Diffusion without and with SPELL in Figure 27 to Figure 36. The prompts are chosen from MS COCO, which Simple Diffusion was not trained on. As opposed to Figure this features both of SPELL's capabilities: Intra-batch repellency (every row is batch of size four), and inter-batch repellency from previous batches, which we treat the shielded set. The examples affirm qualitatively that SPELL increases the diversity of generated images. Notably, this is without lowering the prompt adherence, which other baselines like IG are prone to, see Table 4 and Figure 4.
We note that some images have copyright overlays, likely learned from the underlying dataset. To investigate this further, we generate 1600 examples for Simple Diffusion without SPELL and 1600 with SPELL. Without SPELL, 62/1600 images have a shutterstock (or similar) overlay, with SPELL it's 105/1600. To confirm that this a stable trend, we also generate images with Latent Diffusion (which trained on the same dataset as Simple Diffusion), where it's 79/800 without SPELL and 98/800 with SPELL. While the second result could still be random chance (Chi-Square independence test with Yates' continuity correction gives p-value 0.15), the first result is beyond random (p=0.001), and also the effect size is quite measurable (7% vs 4% overlay rate). To improve the understanding of the inner workings, we make two more observations. First, we note that the copyright overlays tend to happen clustered at specific prompts. E.g., one motorcycle prompt has 21/32 images with overlay while most other prompts have So, the distribution quite skewed and seems to depend on the prompt. Second, we find that the watermarks can serve to push away images from similar ones without watermark and to pull together images with the same watermark. This informs our best understanding. namely that the copyright overlays serve "highway" between modes that allows SPELL easily explore modes. SPELL only uses this if the true images/ / mode distribution of given prompt actually includes these copyright modes.
27

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
(a) Simple Diffusion without SPELL.
(b) Simple Diffusion + SPELL.
Figure Images generated with Simple Diffusion without and with SPELL for the COCO prompt "A bed and a mirror in a small rom.... " Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds the runs without SPELL.
(a) Simple Diffusion without SPELL
(b) Simple Diffuusio + SPELL
Figure 28. Images generated with Simple Diffusion without with SPELL for the MS COCO prompt "Baked pizza herbs displayed on serving tray at table.". ", Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds as the runs without SPELL..
28

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
REEE 1
O
THE THN
(a) Simple Diffusion without SPELL
(b) Simple Diffusion + SPELL
Figure 29, Images generated with Simple Diffusion without and with SPELL for the MS COCO prompt "The clock tower in the center of the building." ". Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds as the runs without SPELL.
(a) Simple Diffusion without SPELL
(b) Simple Diffussin + SPELL
Figure 30. Images generated with Simple Diffusion without and with SPELL for the MS COCO prompt "A bed and desk in a small room.".
Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the seeds as the without SPELL.
29

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
(a) Simple Diffusion without SPELL
(b) Simple Diffusion + SPELL
Figure Images generated with Simple Diffusion without and with SPELL for the MS COCO prompt "A furry, black bear standing rocky, weedy, area in the wild.". Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds the runs without SPELL.
(a) Simple Diffusion without SPELL
(b) Simple Diffusion SPELL.
Figure 32. Images generated with Simple Diffusion without and with SPELL for the MS COCO prompt "A group of seagulls flying over a wooden dock that sitting a lake during the early part of the evening." ". Five batches (rows) with each four images, with both intra- and inter-ber batch repellency, with the same seeds the without SPELL.
30

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
(a) Simple Diffusion without SPELL.
(b) Simple Diffusion + SPELL.
Figure Images generated with Simple Diffusson without and with SPELL the COCO prompt "A plane flies over water with two islands nearby.". Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds as the runs without SPELL..
(a) Simple Diffusion without SPELL
(b) Simple Difffusion + SPELL
Figure Images generated with Simple Diffusion without and with SPELL for MS COCO prompt "A traffic light over street surrounded by tall buildings." Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds runs without SPELL.
31

---

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
(a) Simple Diffusion without SPELL
(b) Simple Diffusion SPELL
Figure 35. Images generated with Simple Diffusion without and with SPELL for the MS COCO prompt "a white boat is out on the water",
Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds as the runs without SPELL.
(a) Simple Diffusion without SPELL
(b) Simple Diffusion + SPELL
Figure 36. Images generated with Simple Diffusion without and with SPELL for the MS COCO prompt "A table layed with food such as, salad, steamed peas and carrots, steamed corn, and bread rolls." ". Five batches (rows) with each four images, with both intra- and inter-batch repellency, with the same seeds as the runs without SPELL.
32


Reviews:
[{"review_id": "R33Qn8AsaT", "reviewer": "Reviewer_E31r", "summary": "This paper proposes sparse repellency (SPELL) to prevent diffusion models from generating images in a set of L2 balls. SPELL can be used to (1) protect diffusion models from generating training images; (2) encourage diversity between multiple generations. Extensive experiments shows the superiority of SPELL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for recognizing that SPELL is a simple way to tackle two current problems in diffusion models at once. We are happy to share explanations and your requested FLUX experiments below.\n### Can image protection with SPELL be applied to text-to-image models?\nYes, there is no difference in applying SPELL for image protection on class-to-image, noise-to-image, or text-to-image diffusion models. Unlike approaches like Interval Guidance and CADS, *SPELL is generalized away from the conditioning signal*, i.e., it does not rely on classifier- or classifier-free guidance, but acts on the diffusion trajectory and the output space of any diffusion model itself, even if the conditioning signal tries to push the diffusion trajectory close to a protected image. **SPELL can thus protect any-to-image diffusion models**. The reason why we demonstrate this on the class-to-image model MDTv2 in Section 4.6 as opposed to a text-to-image model is simply that MDTv2’s exact training dataset and preprocessing are public (ImageNet-1k) so that we can protect the exact training data, whereas text-to-image models like SD3 are trained on proprietary dataset splits.\n### Why does SD3 + SPELL have significant degradation in FID and FD_DINO? Does this imply a drop in image quality?\nWe attribute the decrease in FID to the fact that it is a reference-based metric that compares distributional similarity with a reference dataset. As SPELL increases the diversity and explores more of the image manifold, its output distribution is intentionally broader than the reference dataset that SD3 without SPELL is trained to match as closely as possible. This issue occurs with all reference-based metrics (precision, coverage, recall, density), which is why we report reference-free metrics like the Vendi and CLIP score. **SPELL’s trade-off is, however, better than for other diversity-inducing metrics** (CADS, IG, PG in Fig. 3), even in reference-based metrics. **We’ve also generated additional high-resolution SD3 + SPELL images**, generated one-by-one with the same seeds as in Figure 5, to show the image quality https://imgur.com/a/Dsh7gxb https://imgur.com/a/Uz0LhOz https://imgur.com/a/VGcvqNE https://imgur.com/a/zrZRuV2 https://imgur.com/a/cuSAADZ\n### Can the SPELL method be applied to rectified flow-based models? For example, FLUX.\nYes, SPELL can be applied to any diffusion model. **We have added examples for FLUX1.schnell** in 1024x1024: https://imgur.com/a/oHouASK https://imgur.com/a/3mI2ynU https://imgur.com/a/knqPFhc https://imgur.com/a/FwJvkhG https://imgur.com/a/qigDza4  Since FLUX1 already generates more diverse images by itself compared to SD3, SPELL’s sparsity means that it is activated on fewer pictures, and in more details, like background objects or colors. If you are interested in one-step diffusion models, we denote that SPELL should be used with a number of diffusion steps greater than 1, because it is applied after each diffusion step. If there is only one step, it will be applied effectively after the generation has already ended, which could introduce artifacts. We will add this discussion to the camera-ready version of the paper to outline SPELL’s applicability.\n### Which model does Figure 3 compare?\nFigure 3 shows results for Latent Diffusion, as denoted in the caption.\n### The assumption of $B_k$ being disjoint in L165-183, page 4, limits the range of $r$ ( if $r \\rightarrow \\infty$, $B_k$ will inevitably overlap). Can the authors provide a principal way for choosing $r$?\nIn the protection case, the question of how big of a shield a user wants to create around a protected image is ultimately a user choice. Should they use a radius that large that shields start overlapping, and should a trajectory point exactly to the middle of two shields (and inside the radius of both shields), their terms cancel each other out, as we openly address in L179-180. In this case, the overlapping shields $z_1$ and $z_2$ can be merged with a combined radius of $r_\\text{merged} = r + d(z_1, z_2)$ and $\\frac{z_1+ z_2}{2}$ as shield center to retain the protection guarantee, as mentioned in L673 in the appendix. However, in practice we did not observe this to be necessary – even in the protection experiment on all 1.2M ImageNet-1k images, we did not have issues with overlapping shields. Due to the high-dimensional space, it is unlikely that a trajectory points exactly in the middle of two shields, where their terms would cancel each other out exactly. Rather, their terms point in slightly different directions, so that SPELL guides the trajectory points away from both shields throughout the diffusion. So, **a user is able to choose a shield radius as large as it suits their needs**. As a side note, for the diversity use case, we provide a principled way of choosing $r$ in L779 in the appendix.\n\n**Thank you again.** We would be happy to learn if the additional experiments and explanations update your evaluation of our paper.", "claims_and_evidence": "Claim #1: SPELL is a new effective method for preventing diffusion models from generating samples inside shields (which are essentially L2 ball neighborhoods of points in a reference set).\n\nEvidence: The methodology and discussion in Section 3 explains the SPELL method and its relationship to related works very well. \n\nClaim #2: SPELL is empirically effective in terms of shielded generation of diffusion models.\n\nEvidence: The experimental results in Section 4 shows SPELL can improve diversity with a little drop in image quality. In Section 4.3, the authors compare SPELL with other methods and presents a better Pareto front. In Section 4.6, SPELL decreases the proportion of samples   inside shields for EDMv2 in ImageNet-1k, demonstrating its potential in large-scale setting and data privacy protection.", "methods_and_evaluation_criteria": "The method and evaluation criteria looks reasonable to me.", "theoretical_claims": "I didn't redo the proof by myself, but the connection between SPELL and DPG/PG makes sense.", "experimental_designs_or_analyses": "1. Why does SD3 + SPELL (row2 and row 5 in Table 1) has significant degradation in FID and FD_DINO? Does this imply drop in image quality?\n\n2. Can image protection with SPELL be applied to text-to-image models? Compared with the method in Section 4.6, T2I may need retrieval augmented generation to apply SPELL. Could the authors comment on this?", "relation_to_broader_scientific_literature": "The paper proposes a new method for training image protection and diversity enhancement for diffusion model inference. I regard these two problems as important problems for diffusion models. The proposed method, SPELL, although being simple, is a sensible method for these two methods that may enlighten future research on this direction.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, I think the submission proposes a simple, reasonable method that is suitable for two important problems in sampling diffusion models, training image protection and diversity enhancement. The authors also did extensive experiments to support their method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. which model does Figure 3 compare? There are 3 Text-to-Image models listed in the setting part, SD3, Latent Diffusion and Simple Diffusion.\n\n2. Can the SPELL method be applied to rectified flow-based models? For example, FLUX.\n\n3. The assumption of $B_k$ being disjoint in L#165-183, page 4, limits the range of $r$ ( if $r \\rightarrow \\infty$, $B_k$ will inevitably overlap). Can the authors provide a principal way for choosing $r$ for a specific reference set?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wm0YYHYlSI", "reviewer": "Reviewer_eLTh", "summary": "This paper introduces Shielded Diffusion, which aims to generate images outside of protected sets. These protected sets may include protected images, other data in the current batch, or data used during training. The authors determine whether the diffusion trajectory is expected to fall into protected sets during the sampling process, dynamically triggering a repellency term to ensure the sampling endpoint stays away from protected sets. The proposed method is training-free, capable of enhancing generation diversity while reducing the risk of model infringement (preventing the generation of training data).\n\n## update after rebuttal\nAlthough the author's reply to some extent resolved my concerns, I still believe that there is some room for improvement in the writing of this paper, as reviewer jFPF also mentioned \"The paper lacks organization, making it difficult to follow and identify the key points.\"\nIn addition, in the author's reply, I found that flux+spell (https://imgur.com/a/FwJvkhG) seems not to work very well, with a relatively small difference before and after, and it affects the image quality (It seems to have affected the image harmony, such as color temperature, saturation.).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for acknowledging that SPELL tackles meaningful practical applications while remaining training-free. We are happy to provide the experimental results for high-resolution images, complex prompts, and compute overhead that you have requested below.\n### Examples and analysis under the SD3 model with higher resolution images and more complex prompts\n**We have added 1024x1024 images with prompts of varying length, generated both via SD3 and FLUX**, under the following links: https://imgur.com/a/Dsh7gxb https://imgur.com/a/Uz0LhOz https://imgur.com/a/VGcvqNE https://imgur.com/a/zrZRuV2 https://imgur.com/a/cuSAADZ https://imgur.com/a/oHouASK https://imgur.com/a/3mI2ynU https://imgur.com/a/knqPFhc https://imgur.com/a/FwJvkhG https://imgur.com/a/qigDza4 . We’ve generated the images one-by-one from the same seeds like in Figure 5. The results show that while the first samples of the unconditional model are already diverse enough (top rows) and SPELL does not act due to its sparsity, it starts diversifying the images in the later generations (bottom rows). This holds both for simple and complex prompts.\n\nThe experiments in our paper also already use complex prompts, with the CC12M prompts ranging from 15 to 491 characters. **We have added an ablation on SPELL’s diversity for different prompt lengths** here https://imgur.com/a/T6KdRAR . It shows that SPELL increases the diversity of the generated images even in the longest 90-100% percentiles of the prompts compared to the baseline model, without compromising on the prompt adherence measured by the CLIP score.\n### I believe the authors should express their content using more accessible, standardized, and logically structured narratives. \nThank you for the feedback! We will highlight the narrative in the camera-ready version (this year's reviewing form does not allow to upload revised pdfs).\n### Figure 1 only shows results for Simple Diffusion and MDTv2, with no results for Stable Diffusion 3.\nWe apologize for this editing issue. **We provide a Stable Diffusion 3 example here**, https://imgur.com/a/qNYX5tX , on top of the other SD3 examples linked above.\n### Experiments on larger resolution images (greater than 256*256)\nWe have added results for 1024x1024 images, see the links above.\n### More complex scenarios beyond \"a dog plays with a ball,\". Can more diverse generations could be achieved in such cases?\nThe “a dog plays with a ball” prompt was only used in Figure 5 to give a qualitative example. **Our quantitative experiments already use CC12M prompts with 15-491 characters, and we have added qualitative results for longer prompts in the links above**, following your recommendation. We will outline this in the camera-ready version.\n### Is there additional computational overhead introduced, and could they provide an analysis of this overhead?\nWe provide an analysis of computational overhead in Table 5 in the Appendix for the diversity use-case, and also reposted here. As can be seen, **SPELL does not increase the generation time during diverse generation**, since it adds only one matrix operation to calculate distances and one to calculate the repellency direction (and in many timesteps does not even do this thanks to its sparsity (see Appendix H)) which is a small computation compared to the diffusion backbone.\n\n| Model                          | Generation time per image (seconds) |\n|--------------------------------|-------------------------------------|\n| Baseline (Simple Diffusion)    | 2.93 ± 0.12                        |\n| Simple Diffusion + SPELL       | 2.94 ± 0.13                        |\n### Could the authors distill the paper's greatest contribution\nWe provide SPELL, a sparse and training-free mechanism to push diffusion trajectories away from already-generated or protected images. We will make this clearer in the abstract and introduction, thank you for your feedback!\n\n**Thank you again for your reviewing time.** We would be happy to learn if the additional experiments and explanations resolve your concerns and update your evaluation of our paper.", "claims_and_evidence": "This paper claims that the proposed technique enhances the generation diversity of diffusion models while only slightly disturbing the FID. It compares text-to-image and class-conditional diffusion models in Table 1, compares with other diversity-inducing methods in Figure 3, and conducts further Sparsity Analysis and qualitative result comparisons, which to some extent validate the paper's claim.", "methods_and_evaluation_criteria": "The method proposed in this paper helps with copyright protection and improves generation diversity, which is meaningful in practical applications.", "theoretical_claims": "This paper's geometric explanation for SPELL is reasonable, as it aims to add disturbance to push $\\hat{x}_0$ away when it potentially falls within radius r (equations 4 and 5).", "experimental_designs_or_analyses": "This paper conducts experiments on class-to-image and text-to-image diffusion models, measuring multiple metrics including Recall, Cendi, FID, and Coverage. Additionally, it performs Sparsity Analysis, with experimental design and analysis that are reasonable to some extent.\n\nHowever, I believe it could provide examples and analysis under the SD3 model with higher resolution images and more complex prompts, as practical usage tends to favor the generation of complex scenes and high-resolution images.", "relation_to_broader_scientific_literature": "Sparsity is the biggest difference between this paper and other methods [1][2][3] controlling similar diversity-precision trade-offs. Other methods apply disturbances to change diffusion trajectories throughout the entire sampling time steps, while the method proposed in this paper uses ReLU weighting for dynamic activation, adding no disturbance when trajectories are moving toward more diverse directions. This helps ensure quality while improving diversity.\n\n[1] Kynkäänniemi T, Aittala M, Karras T, et al. Applying guidance in a limited interval improves sample and distribution quality in diffusion models[J]. arXiv preprint arXiv:2404.07724, 2024.\n[2] Sadat S, Buhmann J, Bradley D, et al. CADS: Unleashing the diversity of diffusion models through condition-annealed sampling[J]. arXiv preprint arXiv:2310.17347, 2023.\n[3] Corso G, Xu Y, De Bortoli V, et al. Particle guidance: non-iid diverse sampling with diffusion models[J]. arXiv preprint arXiv:2310.13102, 2023.", "essential_references_not_discussed": "Based on my understanding of this field, the authors appear to have included most of the relevant literature discussions.", "other_strengths_and_weaknesses": "Strengths: The method proposed in this paper helps with copyright protection, which is meaningful for mitigating risks in AI-generated content; the proposed method is training-free and does not introduce additional computational overhead.\n\nWeaknesses: This paper is obscure and difficult to understand, especially for non-domain experts. The reading experience is particularly poor in terms of logical flow. I believe the authors should express their content using more accessible, standardized, and logically structured narratives. For example, Sparse Repellency is a core concept of this paper, and the authors should explain this term first.", "other_comments_or_suggestions": "In the introduction, the authors mention \"This phenomenon is illustrated in Figure 1 for three popular diffusion models, Stable Diffusion 3 (Esser et al., 2024), Simple Diffusion (Hoogeboom et al., 2023) and MDTv2 (Gao et al., 2023).\" However, Figure 1 only shows results for Simple Diffusion and MDTv2, with no results for Stable Diffusion 3.", "questions_for_authors": "1. Could the authors explain why they did not conduct some experiments on larger resolution images (greater than 256*256)?\n2. Could the authors explain why they did not test more complex scenarios beyond \"a dog plays with a ball,\" and whether more diverse generation could be achieved in such cases?\n3. Is there additional computational overhead introduced, and could they provide an analysis of this overhead?\n4. Could the authors distill the paper's greatest contribution, rather than listing 5 lengthy points as in the Introduction?\n\nI will maintain a rejection stance; further explanation from the authors would help me improve my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P4XpRajNfA", "reviewer": "Reviewer_U4mF", "summary": "This paper introduces an application of negative guidance on reference datasets (e.g., {training, validation} datasets) to enhance diversity and generate novel samples that differ from reference images. The paper leverages geometric steering to guide samples away from the reference dataset. To minimize the performance degradation caused by the proposed method, the smallest perturbation is applied to pull samples outside a ball centered at the datapoint in the reference dataset closest to the current sample. This approach enables the use of diverse images across various forms of latent diffusion, such as text-to-image and image generation. The experimental results demonstrate that the proposed method maintains generation performance while producing more diverse images.\n\n## **Update after rebuttal**\n\nI appreciate the authors graphical explanation of how SPELL performs unconditional generation. While I want to zoom in understanding the limitations and complete failure cases of this paper by varying their hyperparameters and reference datasets, the authors focused on presenting favorable scenarios where SPELL operates smoothly.\n\nOverall, I am satisfied with the current state of this paper because they addressed effective negative guidance, which leads to diverse samples while also generating specific samples away from the reference images. As taking into to all aspects of manuscript and response, I maintain my original score", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your interest in the theory behind SPELL that enables its outperformance. We delve into the theoretical connections between SPELL and other methods below, as well as the experimental results for pixel-space diffusion. We also provide further results in the responses to the other reviewers.\n### I believe the paper needs to validate the proposed method with pixel-space diffusion to enhance its visibility.\nWe already validate on a pixel-space diffusion model, Simple Diffusion, as noted in lines 239 and 292. Please refer to Table 1 for its results. In general, **SPELL works independently of the space the diffusion model diffuses in**, whether it is a latent VAE space or a direct RGB diffusion, and whether it is an unguided, classifier-guided, or classifier-free guided model. This sets it apart from alternative approaches like CADS and IG.\n### This paper doesn’t rigorously offer a theoretical framework for negative guidance using diffusion models. However, their conditions and the magnitude of negative guidance are based on a geometric interpretation.\nSPELL’s repellency directions and magnitudes are based on a theoretical framework, which is explained in appendices A to C. The general problem we tackle is that we want to bring the probability / density that a diffusion model generates (an r-Ball around) a point $z_k$ to exactly zero, i.e. $P_0(B_r(z_k)) = 0$. The interesting part about this proof is that we need to choose the direction and magnitude of the intervention despite not knowing the density at $P_0(z_k)$ of the original diffusion model, since diffusion models do not provide density estimates. In Appendix B **we use Tweedie’s formula for a theoretical derivation of the exact direction and magnitude required to achieve this, leading to Equation (5)**. While in the main paper, we start with a geometric interpretation of Equation (5), we also give a conservative-field interpretation in Appendix C to connect SPELL to other negative-guidance based frameworks, and show SPELL’s theoretical relation to Diffusion Posterior Sampling in L165-219. We hope that these rigorous theoretical analyses help future researchers connect and develop SPELL as well as negative guidance.\n### The proposed method consistently improves recall metrics, but the FID score is not maintained. However, the paper does not provide an explanation for this issue. \nWe attribute the decrease in FID to the fact that it is a reference-based metric that compares distributional similarity with a reference dataset. As SPELL increases the diversity and explores more of the image manifold, its output distribution is intentionally broader than the reference dataset that the diffusion models without SPELL are trained to match as closely as possible. This issue occurs with all reference-based metrics (precision, coverage, recall, density), which is why we report reference-free metrics like the Vendi and CLIP score. **SPELL’s trade-off is, however, better than for other diversity-inducing metrics** (CADS, IG, PG in Fig. 3), even for reference-based metrics. __To demonstrate the preservation of image quality, we’ve generated additional high-resolution SD3 + SPELL images here__ https://imgur.com/a/Dsh7gxb https://imgur.com/a/Uz0LhOz https://imgur.com/a/VGcvqNE https://imgur.com/a/zrZRuV2 https://imgur.com/a/cuSAADZ . \n### It would be beneficial to review the paper [1] that discusses negative guidance aiming for mitigation of memorization. Furthermore, [2] also addresses dynamic negative guidance while preserving generation performance and effectively negating a portion of the data distribution.\nThank you for those interesting pointers! We will discuss them in the camera-ready version of the paper (this year's reviewing form does not allow to upload revised pdfs).\n\n**Thank you again for your time during the busy rebuttal period.** We would be happy to learn if the additional experiments and explanations resolve your concerns and update your evaluation of our paper.", "claims_and_evidence": "The paper’s claim is clear and easy to understand. This paper introduces a novel negative guidance approach with a geometric interpretation. This geometric approach seems more clear and straightforward compared to particle guidance and CADS. Empirical evidence shows its superiority over previous methods \n\nHowever, its limitations lie in its applications to latent diffusion models rather than pixel-space diffusion. This method heavily relies on the hyperparameter $r$, which acts as a reference data shield. However, the suggested value and its analysis are only based on latent diffusion. I believe it needs to validate the proposed method with pixel-space diffusion to enhance its visibility.", "methods_and_evaluation_criteria": "This paper compares the proposed model to baselines using appropriate benchmarks and metrics. To assess generating performance, they not only measure {FID}, but also demonstrate {Recall, Coverage, Density} to evaluate the diversity of the generated images by the proposed method.", "theoretical_claims": "This paper doesn’t rigorously offer a theoretical framework for negative guidance using diffusion models. However, their conditions and the magnitude of negative guidance are based on a geometric interpretation. Their messages are understandable in terms of measuring distances and determining how close the current sample is to reference data points.", "experimental_designs_or_analyses": "Its experimental design evaluates the performance of the model on diversity of generated samples. In particular, the hyperparameter $r$ plays a crucial role in determining both diversity and quality of the generated samples. However, as shown in Table 1, this method does not effectively enhance diversity while preserving image quality. Notably, the FID score is consistently sacrificed in favor of improving recall.", "relation_to_broader_scientific_literature": "This paper suggests that negative guidance can be interpreted geometrically.", "essential_references_not_discussed": "It would be beneficial to review the paper [1] that discusses negative guidance aiming for mitigation of memorization. Furthermore, [2] also addresses dynamic negative guidance while preserving generation performance and effectively negating a portion of the data distribution. \n\n[1] Chen, Chen, Daochang Liu, and Chang Xu. \"Towards memorization-free diffusion models.\" CVPR2025.   \n[2] Koulischer, Felix, et al. \"Dynamic Negative Guidance of Diffusion Models.\" ICLR2025", "other_strengths_and_weaknesses": "### **Weakness**\n\n- This paper demonstrates that the proposed method consistently improves recall metrics, but the FID score is not maintained. However, the paper does not provide an explanation for this issue. While the FID score also considers the diversity of generated samples, the numerical evidence presented in Table 1 suggests that the generated images lack plausibility. I believe that this phenomenon is asked for further investigation.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dlf4EEqu3u", "reviewer": "Reviewer_jFPF", "summary": "The paper addresses the issues of limited diversity and replication of training images in text-to-image diffusion models by introducing a method to ensure that generated images are novel and diverse using sparse repellency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your assessment that our experimental evaluation is extensive. To add to this, we add the additional experiments you have requested below, namely a runtime analysis and SPELL’s performance under text prompts of varying length. We also provide further results, including qualitative examples, in the responses to the other reviewers.\n### The SPELL method introduces additional computational overhead due to the repellency terms, potentially making it less efficient than simpler diffusion models.\nA runtime analysis can be found in Table 5 in the appendix, we also post it here for your convenience. It shows that **the generation time with and without SPELL is equal in the diversity setup**. This is because SPELL only adds one matrix operation to calculate distances and one to add the repellency terms, which is small compared to the computational cost of the diffusion backend. \n\n| Model                          | Generation time per image (seconds) |\n|--------------------------------|-------------------------------------|\n| Baseline (Simple Diffusion)    | 2.93 ± 0.12                        |\n| Simple Diffusion + SPELL       | 2.94 ± 0.13                        |\n\n### The authors should compare the performance of the proposed SPELL method with varying text prompts\nThank you for the suggestion, **we have added your suggested experiment here** https://imgur.com/a/T6KdRAR . We sliced the CC12M dataset into groups of prompts with increasing length. As can be seen in the plot, SPELL consistently increases the diversity of both short and long prompts compared to the baseline model, without losing on the CLIP-Score, which measures prompt adherence. \n### The paper lacks organization, making it difficult to follow and identify the key points.\nThank you for the feedback, we will add intuitions with the additional space of the camera-ready version of the paper.\n\n**Thank you again for your feedback.** We would be happy to learn if the additional experiments and explanations resolve your concerns and update your evaluation of our paper.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes, the proposed method is validated through comprehensive experiments.", "relation_to_broader_scientific_literature": "The concept of repellency mechanisms has been explored in other areas of machine learning, such as anomaly detection and outlier detection.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces the SPELL method, a novel approach to improve the diversity of generated images in text-to-image diffusion models by using sparse repellency terms.\n\n2.  The proposed SPELL significantly enhances the diversity of generated images without substantially impacting image quality.\n\n3. The paper provides extensive experiment evaluations.\n\nWeaknesses:\n\n1. The SPELL method introduces additional computational overhead due to the repellency terms, potentially making it less efficient than simpler diffusion models.\n2. The authors should compare the performance of the proposed SPELL method with varying text prompts, as the diversity of generated images can be significantly influenced by changes in the prompts.\n3. The paper lacks organization, making it difficult to follow and identify the key points.", "other_comments_or_suggestions": "Some visualization results should be removed to the main context.", "questions_for_authors": "I noticed artifacts in Figures 25 and 26, suggesting that the introduced SPELL method may negatively impact the accuracy of the generated images.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received 3 Weak Accept and 1 Weak Reject scores pre-rebuttal. The rebuttal did not affect the reviewers' decisions. Overall, the reviewers applauded the paper for its novel and reasonable method that effectively addressed training image protection and diversity enhancement, as well as the extensive experiment evaluations. Reviewer eLTh still concerned about paper writing and unsatisfactory image quality produced by FLUX+SPELL, just keeping his Weak Reject score unchanged.

The ACs checked and found the concern from Reviewer eLTh minor. The contributions of the paper outweigh its shortcomings. Hence, we agree to accept the paper to ICML.
Decision: Accept (poster)