Color Conditional Generation with Sliced Wasserstein Guidance
Authors: Alexander Lobashev
Maria Larchenko
Dmitry Guskov
Venue: NeurIPS 2025 spotlight ()
Keywords: color conditional generation
diffusion models
sliced Wasserstein distance
optimal transport
Primary area: deep_learning
TL;DR: Training-free approach for image generation with diffusion models conditioned on the color distribution of a reference image.

Abstract: We propose SW-Guidance, a training-free approach for image generation conditioned on the color distribution of a reference image. While it is possible to generate an image with fixed colors by first creating an image from a text prompt and then applying a color style transfer method, this approach often results in semantically meaningless colors in the generated image. Our method solves this problem by modifying the sampling process of a diffusion model to incorporate the differentiable Sliced 1-Wasserstein distance between the color distribution of the generated image and the reference palette. Our method outperforms state-of-the-art techniques for color-conditional generation in terms of color similarity to the reference, producing images that not only match the reference colors but also maintain semantic coherence with the original text prompt. 
Our source code is available at https://github.com/alobashev/sw-guidance.

Color Conditional Generation with Sliced Wasserstein Guidance
Alexander Lobashey'
Maria Larchenko Dmitry Guskov 1,3
Glam AI, San Francisco, USA
"Magicly AI, Dubai, UAE, MccGill University, Montreal, Canada eobashevalezander, mariia. larchenko, guskovolddmitttyy@@@aaa.....
Abstract
We propose SW-Guidance, a training-free approach for image generation con- ditioned on the color distribution of reference image. While possible generate image with fixed colors by first creating image from text prompt and then applying a color style transfer method, this approach often results semantically meaningless colors in the generated image. Our method solves this problem by modifying the sampling process of a diffusion model to incorporate the differentiable Sliced I-Wasserstein distance between the color distribution of the generated image and the reference palette. Our method outperforms state-of. the-art techniques for color-conditional generation in terms of color similarity the reference, producing images that not only match the reference colors but also maintain semantic coherence with the original text prompt. Our source code is available at hhtpss: //gi giithb....o com/ wwwo/allbeee www.asheesss-wweeeoeeaeeeeeesa.a sttaashee//xguuddnccee guidance
Figure 1: color-conditional generation by Sliced Wasserstein guidance achieves unprecedented match with reference color palette without transferring other stylistic features.
Introduction
To get desired picture from text-to-image models we usually need precise prompt and bit of luck. However, natural language is not expressive enough to accurately describe colors, and even specific
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

terms such as "turquoise blue" yield varying tones. Moreover, prompt length constraints make full palette descriptions impractical. Using reference images for color styles addresses these limitations and establishes the color transfer problem, that is, applying a reference color style to a content image.
Color transfer is closely related the artistic style transfer. Notably, artistic style is not linked the depicted objects but is instead shared between patches of an image. This insight was utilized in the seminal work by Gatys al. [. where artistic style defined as the distribution of activations in the VGG-19 network (2]. To match the artistic style between generated and reference images, Gatys et al. minimized the difference between the Gram matrices of activations from internal layers of VGG-19 (which is equivalent to matching the first two moments of the distributions of activations).
Strictly speaking, the style loss by Gatys et al. is not proper distance in the space of probability distributions, as, for instance, Jensen-Shannon 3]. Total variation and various Wasserstein distances 4. Unfortunately, these metrics are hard to approximate in a differentiable fashion. To address the complications of Wasserstein metrics, a new family of metrics called Sliced Wasserstein (SW) distances was developed in 2012 (). First, Sliced Wasserstein distances are differentiable. Second, they can be efficiently estimated from samples. Importantly, for bounded distributions, convergence of the Sliced Wasserstein distance implies the convergence of all moments. However, in high- dimensional spaces the sliced approach requires a large number of projections to accurately estimate the distance. To generalize the SW distance and enhance its performance higher dimensions other 12
its variants were proposed 7 8 9 10
13 14].
Following Gatys al., various CNN-based color transfer methods were proposed, such as DPST 15 WCT [16), PhotoWCT [7]. WCT2 [8]. PhotoNAS [9]. PhotoWCT2 (20), and DAST 21]. These algorithms can address the problem of color-conditional image generation, transferring reference colors to the image created by text-to-image model.
Another way to achieve color conditioning is control the generation process of diffusion model [22 2 24]. This problem setting is broadly called the stylized image generation. The approaches for stylized generation could be categorized into three groups:
Modification of weights The first group includes additive corrections of model's weights. which
require fine-tuning for every new style of images: Textual Inversion (25. DreamBooth [26), and LoRA (27]. The introduction of ControlNet (28] and T21-Adapter (29 in 2023 enabled adjustments of weights in a single pass of hyper-network, ControlNets and adapters are trained on fairly large paired datasets and cover tasks such as pose, depth, and edge conditioning.
Modification of attention The examples of attention-related algorithms are IP-Adapter (30), StyleAdapter (3.) StyleDrop (32), StyleAligned [3 InstantStyle (4 (35). Training-free, they change attention output on each step and are effective for controlling structural and high-level features, such as painting style and composition, but do not target a color distribution separately.
Modification of sampling The third way to impose condition is to add a new term to the denoising process. The first work of this kind was classifier guidance (22), which requires specific classifier trained on noisy data " Diffusion Posterior Sampling (DPS) (6) addresses the main weakness of classifier guidance by replacing a noisy classifier with composition of a predicted noiseless image and a classifier trained on clean data (i.e., any pre-trained one). Universal Diffusion Guidance (37) and FreeDoM (38) generalize the DPS approach by replacing the MSE loss used by DPS with general distance function. These ideas were further developed in RB-Modulation [39].
In current approaches to stylized image generation style and color conditioning are often entangled, making it challenging to control these aspects independently. Our goal is to propose a way to condition solely color and independently control the palette and general style of image.
Our Contributions This work makes the following key contributions:
For the first time, we incorporate the differentiable Sliced Wasserstein distance and its generalizations into the conditioning of a diffusion model
We achieve state-of-te of-the-art results in problem of color-specific conditional generation, without transferring unwanted textures or other stylistic features (see Fig. I
'This guided denoising procedure resembles the optimization process of Gatys et al., which also generates an image from Gaussian noise by iterative denoising. In this case, the unconditional score function is equal to the gradient of content loss, and the classifier guidance term corresponds the gradient of style loss.

---

2 Background
2.1 Conditioning Process in Diffusion Models
Diffusson models [40. 41 are a class of generative models that learn to iteratively denoise a data distribution. To describe the conditioning process in diffusion models, the posterior distribution in terms of the gradient of the log-likelihood and the unconditional score:
Bayes' rule to express
Vx log p(<e(()))) V., (ygp()) V.. log p(zz),
(1)
where y represents the conditioning, and 2; is the noisy sample at noise level t.
Diffusion Posterior Sampling (DPS)
36 introduced an approximation for the conditional likelihood based on predicted noiseless sample, To = |E (2)), as
ply x4) R p(ylúo(a))
In the DPS approach, the authors considered the gradient of the log- likeliicl as follows:
2x, log logp(y/fo(z1) = - o² 2x, ll A()
where A is an operator, generally non-linear, that extracts the condition from the predicted noiseless sample ê., and a is positive hyperparameter. For example, A could extract the CLIP 42 embedding from f.. and could be a target prompt embedding.
Universal Diffusion Guidance [37] and FreeDoM [88 extend the DPS approximation by proposing more general distance function D the space conditions Specifically, for y E Y, the gradient of the logarithm of the posterior distribution is given by:
Va log ((
V., Vn (y, A(co(x))))
This formulation more flexible and lets and (z)) be more complicated objects than vectors in Rdd as long we define differentiable distance function between them. In the next section, we will define the Sliced Wasserstein distance as a suitable distance between two probability measures.
2.2 Sliced Wasserstein distance
classical formulation of color transfer problem to align two probability distributions in the 3-dimensional RGB space. Specifically, the color distributions of content image and reference image can be represented as probability density functions, denoted by mo and क respectively. The objective in guided diffusion models is to match the generated sample's probability density mn with the reference Ti.
Wasserstein distances, rooted in optimal transport theory, appear to be natural for this task as they measure the cost of transporting one probability distribution match another 4.. The Wasserstein distance of order is
/
1/P
W (TO, m1)
inf (Ell((sa.)) Xxxx,
||0 - vl1" dz(x.))
(5)
Calculating W, Wp π1) for many samples can be computationally prohibitive, also a Wasserstein distance is hard to differentiate through, because its value is itself result of an optimization procedure inf over all transport plans [[ 11), i.e. over all joint distributions with marginals Ty and Ti
To alleviate this issue, the Sliced Wasserstein (SW) distance was introduced []] offering a more computationally tractable alternative by reducing dinenssinnii distributions to mi-dimensional projections where the Wasserstein distance can be computed more straightforwardly. The Sliced p- p-Wasserstein distance defined as [5] [
(1
1/p
SW (mT, TT)
WP POT1) d8
(6)
where 8d-1 the unit sphere Rd with Sgu do 1, Po is a linear projection onto a one- dimensional subspace defined by and WP is ordinary p- Wassersttei distance Eq

---

Latent space
Images
Color Distributions V (D(()))
Noise X
Denoised X
Decoded D(Xo)
control vector
prompt guided
SW
T2-1
ref, im. O
Figure General scheme of the Slices Wasserstein Guidance for latent diffusion model with decoder D) and feature extractor y.
3 Method
Below we give a detailed description of SW-Guidance algorithm. We placed some necessary theoretical fact, Proposition [ and Lemma [ at the end of this section.
The general scheme of the algorithm is illustrated Fig.
We denote IT...., To as the latent states of our diffusion sampling, where is a sample from normal distribution, Zo is noiseless sample and & (x²) is prediction of zu for given Zt. (xxo) is a decoded image from the latent space of diffusion to the real image domain. Lastly, Y is feature extractor, which in our case is the color distribution an image in RGB color space.
Algorithm Color Conditional Generation with Sliced Wasserstein Guidance
1: Initialize latent vector ~N 7), set learning rate Ate samples from the reference color distribution
2: for = T to do
3: u 4-
Initialize control vector
4: for to M do
5:
2/ + 23 +
6:
Get prediction of last latent 2. DDIM(t, Get 9o + VAB(.o) for k: = to K do
7:
b Decode latent to image Sliced Wasserstein
8:
9:
Project samples on a random direction
10:
Update loss L <- L Σ /cdf jo cdfy)
11:
end for
Update control vector u Air ((u)
13:
end for
Update latent x + He + u
15:
Get denoised latent 2 + DDIM 27)
end for
The proposed Algorithm initializes noise tensor sampled from latent normal distribution. Over T diffusion timesteps, the noise tensor iteratively refined. Following each denoising step, predicted result 2 is decoded to obtain an image 9n = (2 ). To modulate guided diffusion within
each timestep, we add an auxiliary control tensor u following [39 That is, is initialized with zeros and we set x = + Then we predict original sample 2o ) (o(uu) and compute gradient of the Sliced Wasserstein distance (SW) between the color distribution of reference image ref and the predicted (o(u) ((ā)(u)) with respect
L(u) SW (T)) Tet))
(7)
The control vector is optimized over M steps to shift the reverse diffusion process toward the reference's color distribution. This optimization accumulates gradients w.r.t u M times and minimizes the loss function L, Eq Let note that by Lemma the minimization of the loss L will lead

---

to weak convergence of generated color distribution (v) towards the reference oet with the convergence of all moments.
The loss function can incorporate first two moments (mean μ and covariance σ) of (b) and Tref
(()) = (140 - Mrer)² + (a - Fref)² + SW Teef)
(8)
The impact of adding the first two moments (see Fig. 5 along with other variants of the SW distance 9 and Energy- -aaed [1] SW distances, is studied in the
such as Generalized [8]. Distributional Experiments section.
Let u² be a shift, obtained after M steps of Eq. 7 optiiiization. Then we set xं 24 + u² and perform usual DDIM (43] denoising step for zi with classifier-free guidance to obtain 2;-1- Full algorithm for a latent diffusion model with classifier-free guidance is listed in the Appendix.
Efficient Computation of Sliced Wasserstein Let Fu and F1 be two cumulative distribution functions of 1-dimensional probability distributions To and лT. Then the Wasserstein distance of order p between xn and x has form (Rachev and Rüschendorf, 1998, Theorem 3.1.2 44
W Wp( (L) = 1
11 dy
Fo (1) (w)l
Formally, it involves differentiable estimation of inverse cumulative density functions. However, in the case of p the Proposition alllow us replace the difference of inverse CDFs by absolution difference of CDFs, making it much easier to compute
W1 W1(To, mi)
[Fo(x) Fi))) da
(10)
Moreover, since all color distributions RGB space have a compact support (unit cube), one can employ guarantees of Lemma 2 which in fact states convergence of general p-Wasserstein distances given convergence of the I-Wasserstein distance. These facts justify the selection of Wassersttei instead of general p- Wasserstein.
Differentiable Approximation CDF We approximate the cumulative distribution function (CDF) by sorting samples from the distribution. Once the samples {-;}}-1 are sorted, the CDF can be directly obtained by assigning a rank to each sorted sample. For given sample F,, its rank (i.e., its position in the sorted array) divided by the total number of samples provides the CDF value at that point. If { represents the sorted samples, the CDF at F(() is given by:
CDF(2
(11)
This sorting operation differentiable, so the CDF also differentiable. To achieve a good approxi- mation of the true underlying CDF, large number of samples is required.
Theoretical Justification We need Proposition for efficient sampling, as it allows one to avoid computing the inverse CDF.
Proposition 1. Let and G be two cumulative distribution functions. Then,
[ (1 dt= [F(x) ( G(x) -G())
dx,
(12)
where F-1 and G-1 the quantile functions (inverse CDFs) of F and G, respectively.
Lemma 2 provides the theoretical foundation for our optimization procedure for multidimensional Borel probability measures pn and μ Rd..
Lemma 2. Let pn and be Borel probability measures on the unit cube in Rd. If the numerical sequence
lim SW (pri ut) =0
(13)
n0000
then the sequence Pn converges to μ weakly, and all moments of μn converge to the moments of μ.
Proofs for Proposition and Lemmal 2 are provided in the Appendix.
5

---

InstantSiyle
iP-Adapler
RB-Modulation
Ours
Reference
InstantSlyle
IP-Adapler
RB-Modulation
Ours
Reference
gorgeous old-fsshioned lighthouse"
"llustration milltrry pop style"
Figure 3: Comparison with stylized generation methods, SDXL.. Other methods show weaker palette matching while transferring high-level features such as brush strokes and wheat fields in example with lighthouses photorealism in the second example.
Table I: Quantitative evaluation, SDXL [46]. We measure palette similarity with 2- Wasserstein distance between the color distribution of the generated image and the reference image. CLIP-IQA and CLIP-T are quality and content scores. The color transfer methods [8
19 20
52
53
54 55 56
are applied to the unconditional SDXL generations. Note, that SW-Guidance has the highest CLIP-T For visual comparisons, the Appendix.
among other stylized generation algorithms
30
39
2 Wasserstein distance
Content scores
Algorithm
mean std of mean
CLIP-IQA (51] f
CLIP-T 42
SW-Guidance SDXL (ours)
0.0297 + 0.0005
0.285 + 0.004
0.270 + 0.002
hm-mkl-hm [52
0.0543 + 0.0011
0.259 + 0.003
0.277 0.002
hm [
0.0856 + 0.0016
0.244 0.003
0.282 + 0.002
Photo WCT2 20
0.1028 = 0.0014
0.225 0.003
0.276 0.002
ModFlows
0.1125 + 0.0016
0.257 0.003
= 0.282 0.002
MKL [55]
0.1191 = 0.0017
0.238 = 0.003
0.283 = 0.002
CT [56
0,1333 + 0.0018
0.230 0.003
0,284 0,002
WCT2 18
0.1347 + 0.0017
0.179 + 0.002
0.288 + 0.002
PhotoNAS
19
0.1608 + 0.0017
0.167 + 0.002
0.279 + 0.002
InstantStyle SDXL
34
0.1758 + 0.0028
0.332 + 0.003
0.238 - 0.002
IP-Adapter SDXL
[30
0.2193 + 0.0032
0.247 + 0.002
0.214 + 0.002
Unconditional SDXL
48
0.3824 + 0.0059
0.239 + 0.003
0.294 1 0.002
RB-Modulation
39
Stable Cascade
0.3795 + 0.0133
0.323 + 0.006
0.266 + 0.003
Experiments
As successor to Universal Diffusion Guidance [37]. the proposed method is not tied to a specific architecture and can be paired with latent or pixel-space diffusion models. For our experiments we have selected Stable Diffusion 1.5 45 and Stable Diffusion XL 46 (Dreamshaper-8
[7 and
RealVisXLL V4 48 with the DDIM scheduler (43]
Test set The experiments are conducted on images generated from the first 1000 prompts taken from the ContraStyles dataset 449 Our color references 1000 photos from Unsplash Lite (50). We refer to these prompts and photos as the test set. A training set not needed for our algorithm.
Metrics To measure stylization strength, calculate the Wasserstein-2 distance between color distributions in RGB space. Two content-reds relattd metrics are based on CLIP embeddings [422. CLIP- IQA [51] is cosine similarity between generated image and pre-selected anchor vectors that define ggood-looking pictures. CLIP-T 42 is a cosine similarity between CLIP representations of a text prompt and image generated from this prompt. In other words, the CLIP-T score indicates whether a modified sampling process still follows the initial text prompt, while CLIP-IQA measures the overall quality of the pictures.
Baselines discussed earlier, the problem of color-conditional generation can solved by first creating image from text prompt and then performing color transfer with specialized color

---

transfer algorithm. Therefore, the largest family of baselines consists of algorithms of this kind applied to the output SD1.5 and SDXL: Histogram matching (hm) [53]. CT [56], MKL [55]
WCT2 18), PhotoNAS 19] PhotoWCT2 [20]. ModFlows [54]. The baseline "hm-mkl-hm" is
combination of histogram matching and MKL. taken from the library [52]. In addition, we take
three of the currently available baselines for stylized generation: IP-Adapter [30]. InstantStyle [44). and RB-Modulation [39], though stylized generation not exactly the problem we aim to solve. While recent work [58] also proposes algorithm for color conditional generation with diffusion models, we exclude direct comparisons due to absence of open-source implementation. The term Unconditional indicates that post-pres post-processing steps controls were applied. We provide CLIP-IQA and CLIP-T metrics for Unconditional SDXL and Unconditional SD1.5 as a reference.
Comparison Results The results in Table L prove that SW-Guidance achieves superior performance in color-conditional generation compared all baseline methods. particular, SW-Guidance has the minimal Wasserstein distance to the reference palette. At the same time, SW-Guidance has the highest CLIP-T among other algorithms for stylized generation (i.e. IP-Adapter, InstantStyle and RB- CB-Modulation). This indicates the ability of SW-Guidance to follow the prompt without adding irrelevant features from the reference image in contrast to other stylization methods. In terms of overall image quality SW-Guidance holds the second place according to CLIP-IQA. Qualitative comparison with stylized generation is given in Fig. 3 with additional visual examples available in the Appendix. The Appendix also contains SD-1.5 performance scores and examples comparable to those shown in Table
control
SW-Guidance
reference
SW-Guidance reference could be just a palette of colors
Figure 4: SW-Guidance combined with canny ControlNet, SD1.5.
only
onl
only only
Figure 5: Ablation studies, SDXL. The best results are obtained with the loss function by Eq (Sw only). Moments-only guidance insufficient. Please refer Table for the quantitative comparison.
7

---

SW -u-iiieenee
InstantStyle
"a woman, in rod dress
reference
woman. rod dress*
reference
Fble frar
dharection otter
Figure 6: SW-Guidance combined with depth control has more flexibility than InstantStyle. SD1.5 model, scale InstantStyle strength.
Compatibility with ControlNets SW-Guidance can be combined with other control methods to define the image layout, see Fig. for the canny control and Fig. for the depth map control. Our method supports any picture, representing a palette, as in the second row, Fig. 4
Note that stylizing algorithms, such InstantStyle, transfer not only color but also other features (see Fig. 3). making it difficult to control color separately. Fig. shows that for InstantStyle the text prompt guiding the color is ignored because it contradicts the features of the reference image (i.e denim dress). Our method is more flexible and sets a red shade which aligns with the reference palette.
Relying only on text prompts for color control is inconvenient. Moreover, color naming often connotative, and words like "lavender", "emerald" and "lime" introduce unintended content details, as shown in Fig. g Please refer to the Appendix for more examples.
With all this said, we conclude that the proposed WW-Guidance is superior in color stylization while maintaining both integrity with the textual prompt and the quality of the produced images.
4.1
Ablation study
Table Ablation study. SD-1.5, Analysis of different Sliced Wasserstein distances.
Wassersttei distance |
Content scores
Distance
mean std mean
CLIP-IQA 51
CLIP-T
Sliced Wasserstein [5]
0.0385 + 0.0006
0.2220 + 0.0027
0.2520 + 0.0017
Energy-Baun Based SW
0.0390 0.0006
0.2241 + 0.0030
0.2535 0.0017
Distributional SW 9
0.0547 = 0.0006
0.2225 + 0.0030
0.2564 0,0016
Generalized SW
g
0.0879 0.0014
0.2098 + 0.0027
0.2594 + 0.0016
Mean & Cov
0.1064 + 0.0013
0.2258 + 0.0030
0.2545 + 0.0017
Table 3: Ablation study. SDXL. The impact of adding the first two moments the SW distance (Eq which is also shown in Fig.
2- Wasserstein distance
Content scores
Distance
mean +: std of mean
CLIP-IQA (1 t
CLIP-T 42
SW only
0.0297 + 0.0005
0.285 + 0.004
0.270 + 0.002
Moments + SW
0.0305 + 0.0006
0.279 + 0.003
0,269 + 0.002
Moments only
0.1176 + 0.0016
0.276 + 0.003
0.282 + 0.002
Unconditional SDXL 48
0.3824 + 0.0056
0.239 + 0.003
0.294 + 0.002
Different Sliced Wasserstein distances Table contains scores for the tested variants of Sliced Wasserstein (SW), each assessed under K = 10 slices, M = 10 iterations per scheduler step. and Ir 100 learning rate, Let us note that Lemma 2 holds for all of them. Please find their formal definition in Appendix section. In general, we didn't 't observe any substantial difference in their content scores. We can also note that, despite the time metric is absent in the table, Distributional Sliced Wasserstein (DSW) takes more time due to inner optimization loop. This suggests that although DSW and Generalized SW are aimed to converge faster for multidimensional distributions, this advantage does not translate to our 3D color transfer task. The Energy-Based SW [I] offered

---

computationally light alternative, though it lacks any clear advantage over regular SW for this application.
Generation time The generation time dependence on M (inner steps) and K (number of slices) for SID11.5 shown in Fig. 7 For our main experiments set M 10 and K 10, which results in 30 seconds for SD-1.5 and around I minute for SDXL to generate an image on Nvidia RTX 4090 GPU. This represents an improvement compared the 2 minutes required by RB-modulation.
Mean and covariance terms The impact of adding the first two moments to the SW distance is presented in Fig. 5 and Table 3 The best results are obtained with the loss function by Eq (SW only). Mean and covariance terms (Eq(8) Moments + SW) do not increase color similarity and tend to produce images of worse quality. Moments-oni only guidance is insufficient.
Dependence on learning rate This experiment can be found in Appendix section.
Limitations and Discussion
The first important limitation of the proposed guidance is its sensitivity to the information about colors in text prompts, especially when they contradict the selected style reference. A clash between the textual and SW guidance typically results in visual artifacts, detailed textual palette descriptions should be avoided.
Secondly, combining this method with existing stylizing attention- approaches is not guaranteed work, as strong stylizing methods could also lead clash of color guidance. Ideally, other conditioning should be disentangled from the color information. This collision effect is subject for further research, As an example, we provide a joint run of InstantStyle and SW-Guidance (Fig. 9
The last point we would like to discuss is the current implementation' requirement to differentiate through U-net. Theoretically, this requirement could be avoided, but like the previous point, requires additional study.
To sum up, this paper presents SW-Guidance, novel training-free technique for color-conditional generation that can be applied to a range of denoising diffusion probabilistic models. Our study covers the SD-1.5 and SDXL architectures, and for both implementations, results in color similarity compared to color transfer algorithms and models for stylized generation. Numerically, show the ability of SW-Guidance to maintain integrity with the textual prompt and preserve the quality of the produced images. Our qualitative examples demonstrate the absence of unwanted textures and irrelevant features from the reference image.
achieved superior
Prompt guided
Wesserten
una dako ofr "AaKE
5
8
Figure 8: Text description of a color may introduce unwanted content details.
InstartStyle
SW Guidance
Rersrence
Figure 7: Ablation study for the dependence on M (inner steps) and K (number of slices) for SD-1155. We use M = 10 and K = 10, which results in 30 seconds for SD-1.5 and around minute for SDXL to generate an image on RTX 4090 GPU.
Figure 9: Limitations. Combination of SW- Guidance and InstantStyle SDXL.

---

References
[I] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
[2] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1400.1566, 2014.
[3] Andrew KC Wong and Manlai You. Entropy and distance of random graphs with application to structural pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5(PAMI-7):599-609) 1985.
[4] Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
[5] Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bemnt. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM 201 I, Ein-Gedi, Israel, May 29-June 2, 2011, Revised Selected Pupers 3. pages 435-446. Springer, 2012.
[6] Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasser- stein barycenters of measures. Journal of Mathematicaa Imaging and Vision, 51:22-45, 2015.
[7] Ishan Deshpande, Yuan-Time Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced wasserstein distance and its use for gans. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10648-10656 2019.
[8] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badcau, and Gustavo Rohde. Gener- alized sliced wasserstein distances. Advances in neural information processing systems, 32, 2019.
[9] Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional sliced-wasssesteei and applications to generative modeling. arXiv preprint vrXiv:2002.07367 2020.
[10] Khai Nguyen, Tongzheng Ren, Huy Nguyen, Litu Rout, Tan Nguyen, and Nhat Ho. Hierarchical sliced wasserstein distance. arXiv preprint arXiv:2209.1337 2022.
[11] Khai Nguyen and Nhat Ho. Energy-based sliced wasserstein distance. Advances in Neural Information Processing Systems, 36, 2024.
[12] Khai Nguyen, Shujian Zhang, Tam Le, and Nhat Ho. Sliced wasserstein with random-pate projecting directions. arXiv preprint 1xXi::2401.15889 2024.
[13] Huy Tran, Yikun Bai, Abihith Kothapalli, Ashkan Shahbazi, Xinran Liu, Rocio Diaz Martin, and Soheil Kolouri. Stereographic spherical sliced wasserstein distances. arXiv preprint arXiv:2402.023455 2024.
[14] Khai Nguyen and Nhat Ho. Hierarchical hybrid sliced wasserstein: A scalable metric for heterogeneous joint distributions. arXiv preprint arXiv:2404.11 15378. 2024.
[15] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4990- 4998, 2017.
[16] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming- Hsuan Yang. Universal style transfer via feature transforms. Advances in neural information processing systems, 30, 2017.
[17] Yijun Li, Ming- Yu Liu, Xueting Li, Ming-Hsuan Yang, and Jan Kautz. A closed-form solution to photorealistic image stylization. Proceedings of the European conference on computer vision (ECCV), pages 453-468 2018.
10

---

[18] Jaejun Yoo. Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Wor Woo Ha. Photore- alistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9036-9045, 2019.
[19] Jie An, Haoyi Xiong, Jun Huan, and Jiebo Luo. Ultrafast photorealistic style transfer via neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10443-10450, 2020.
[20] Tai- Yin Chiu and Danna Gurari. Photowet2: Compact autoencoder for photorealistic style transfer resulting from blockwise training and skip connections of high-frequency residuals. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2868 2877, 2022.
[21] Kibeom Hong, Seogkyu Jeon, Huan Yang, Jianlong Fu, and Hyeran Byun. Domain-aware universal style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14609-14617, October 2021.
[22] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598 2022.
[24] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Iya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint wrXiv:2112.1044: 2021.
[25] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar 2iv: 01618, 2022.
[26] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-200 22510, 2023.
[27] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 /96885, 2021.
[28] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[29] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296 4304, 2024.
[30] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0672., 2023.
[31] Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, and Ping Luo. Styleadapter: A single-pass lora-free model for stylized image generation. arXiv preprint arXivv2309.017770 2023.
[32] Kihyuk Sohn, Lu Jiang, Jarred Barber, Kimin Lee, Nataniel Ruiz, Dilip Krishnan, Huiwen Chang, Yuanzhen Li, Irfan Essa, Michael Rubinstein, al. Styledrop: Text-to-image synthesis of any style. Advances in Neural Information Processing Systems, 36, 2024.
[33] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4775-4785, 2024.

---

[34] Haofan Wang, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. In- stantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.022 2024.
[35] Haofan Wang, Peng Xing, Renyuan Huang, Hao Ai, Qixun Wang, and Xu Bai. Instantstyle- plus: Style transfer with content-preserving in text-to-image generation. arXiv preprint arXiv:2407.00788. 2024.
[36] Hyungjin Chung, Jeongsol Kim, Michael T Mocann, Marc L Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.1466 2022.
[37] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 843- 852, 2023.
[38] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23174-23184, 2023.
[39] Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkot- tai, and Wen-Sheng Chu. Rb-modulation: Training-free personalization of diffusion models using stochastic optimal control. arXiv preprint arXh::2405. 17401, 2024.
[40] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 68000 2020.
[41] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 1.32511111145,, 13456, 2020.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-87663 PMLR, 2021.
[43] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[44] Svetlozar T Rachev and Ludger Rüschendorf. Mass Transportation Problems: Volume 1: Theory. Springer Science & Business Media, 2006.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[46] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL
[47] Lykon. Dreamshaper-8. http:://hngggigacco.ooo/////a/ap 2023.
[48]
SG. 161222. Realvisxl v4.0. http:///aataaaaccccoe/e///////es 344487 2024.
[49]
College Park Tom Goldstein's 's Lab at University of Maryland. Contrastyles dataset. https / //huggingface. co ottp////ttsseeeooo.o.arrr/a////a group-uuld/Conntssasyyyssss 2024.
[50] Unsplash. Unsplash lite dataset 1.2.2. hatps://unsplash..c.lt
2023.
[51] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
12

---

[52] Christopher Hahne and Amar Aggoun. Plenopticam v1.0: A light-field imaging framework.
IEEE Transactions on Image Processing, 30:6757-6771, 2021, doi: 10.1109/11P2202133955777
[53] Rafael Gonzales and BA Fittes. Gray-level transformations for interactive image enhancement. Mechanism and Machine theory, 12(1):111-122) 1977.
[54] Maria Larchenko, Alexander Lobashev, Dmitry Guskov, and Vladimir Vladimirovich Palyulin. Color style transfer with modulated flows. In ICML 2024 Workshop on Structured Probabilistic Inference and Generative Modeling. 2024.
[55] François Pitié and Anil Kokaram. The linear monge-kantorovite linear colour mapping for example-based colour transfer. In 4th European conference on visual media production, pages 1-9. IET, 2007.
[56] Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer graphics and applications, 21(5):34 2001.
[57] Afifi Mahmoud. Python implementation of colour transfer algorithm based on linear monge- kantorovitch solution. luttps://github.. http:///..o.ommmmooooooofffffooo cttp/ooooouttttsffrrrrreoo twr.sserrr..... _MKL 2023.
[58] Ka Chun Shum, Binh-Son Hua, Due Thanh Nguyen, and Sai-Kit Yeung. Color alignment in diffusion. arXiv preprini trXiv:2503.06744 2025.
[59] Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
[60] Nicolas Bonnotte, Unidimensional and evolution methods for optimal transportation. PhD)
thesis, Université Paris Sud-Paris XI; Scuola normale superiore (Pise, Italie), 2013.
[61] ghoskno. Color-canny controlnet.
httges//wuggiiggeee.oo//tttttttp
laion art-en-colle ene-ee--ooolee con--cooocanny 2023.
[62] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 02(78):1-8; 2021. URL http: http://jmmlooooopppeeee wttp::pprpppp//./////////
451 .html
[63] Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry Dylov. Pytorch image quality: Metrics for image quality assessment, 2022. URL 14818
org/ abs/ 2208
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in abstract and introduction are well supported in the main part.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper contains the dedicated section, that discusses limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper does include theoretical grounds discussed in section Method. All proofs also presented in the Appendix.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides explicit explanation on how the results could be repro- duced altogether with models architecture and source code https: htap:::/aanonmous 4open
science/r/ss guidance- 3E7D
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We describe all the data used for evaluating and provide a source code. We plan to publish all the data in the case of acceptance.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This information can be found in the source code and Experiments section.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error assessment is presented comparison tables.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
14

---

Justification:
This information can be found in Experiment and Supplementary sections.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of
Answer:[Yes
Justification: We got acknowledged with the NeurlPS Code of Ethics and confirm that our research follows its guidelines.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The color style guidance generation is domain-speciffe and primarily intended for artistic and stylistic control in generative models. The method does not introduce new mechanisms for semantic manipulation, identity generation, or content creation that could be directly associated with misinformation or surveillance. As such, we do not anticipate any broader societal impacts, either positive or negative.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets included in paper are properly cited and link to them are included.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not introduce any datasets. We use available datasets and credit their sources.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Crowdsourcing is not used in this study.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study does not involve human participants subjects.
16. Declaration of LLM usage
15

---

Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The work does not any LLM for methodology development any original part.
16

---

A Sliced Wasserstein Distances
Sliced Wasserstein Distance Wasserstein distances appear be natural for our task of color transfer as they measure the cost of transporting one probability distribution to match another [4] The Wasserstein distance of order p is
1/8
inf / Wp WP( m1) [==== (mEll(Ro.5)) Xxx
||x vll dx
(14)
where [T, T1) represents the set of all joint distributions with marginals To and Ti. However, directly computing W (To, mi) is computationally expensive and difficult to differentiate through, because its value is itself result of an optimization procedure inf over all transport plans [T mi).
To overcome this issue, the sliced Wasserstein (SW) distance was introduced [5]. offering a more computationally tractable alteraative by reducing high-dimensional distributions to one-dimensional projections where the Wasserstein distance can computed more straightforwardly. The sliced p- p-Wasserstein distance is defined as [5 [
( WP POT1 d8
SW SW TT)
(15)
where Sd-1 is the unit sphere in Rd with fga-1 d0 Po is a linear projection onto one- dimensional subspace defined by 0 Radon transformation in general) and WP is an ordinary p- D-Wasserstein distance by Eq
known issue with the Sliced Wasserstein (SW) distance arises when sampling parameters for projections. As noted in (8), uniformly sampled 0 values the unit sphere S in high dimensions tend to be nearly orthogonal. This resulting in W2 (PaTT. PATI) with high probability. Consequently, these projections fail provide diseriminative information about the differences between the distributions Ty and 71-
Distributional Sliced Wasserstein Distance The Distributional Sliced Wasserstein (DSW) distance, proposed [ generalizes the SW distance by introducing a probability distribution (0) over slicing directions and defined as:
DSW, (TO, T1) =
= ssup (LL
We(Pe 1/P
(16)
where the optimization sup is performed w.r.t probability distributions over unit sphere Sd-1, with Jg. df) 1.
-ased Sliced Wasserstein Distance
The Energy-Baan Based Sliced Wasserstein (EBSW) distance, introduced in M11. provides alternative to the optimization-basee approach of DSW by defining a slicing distribution Too.m. (6) f. p) based on the projected Wasserstein distances:
am.. (.: x S(WP(PoTO. POTT)):
(17)
where f is monotonically increasing energy function (e.g., f(x) e that emphasizes directions with larger projected Wasserstein distances. Using this slicing distribution, the EBSW distance is defined as:
EBSW, (mo, u1; f)
(18)
Eg (0:f.p) [WP ( P&T1))
To improve computational efficiency, importance sampling used, with a proposal distribution (0) to sample directions and weight them according to the ratio:

---

fl fWW (WortP PoT1)) ov(0)
(19)
Generalized Sliced Wasserstein Distance The Generalized Sliced Wasserstein (GSW) distance g replaces the Radon transform with generalized Radon transform that depends defining function g(z, 8). Formally, for function I, the generalized Radon transform is defined as:
/ (z))((t ((x,0) dz, g2
Gl 0)
(20)
where 8 is the Dirac delta function. Using the generalized Radon transform, the GSW distance between two distributions To and πı is defined as:
GSW m1) = (// WP(GI=u( GI (.00) d8
1/P
(21)
where Qg
compact set feasible parameters for the function g(r, 0) (e.g., Qg sd-1 for g(2r, 0) = (x, 0)).
For empirical distributions mo and Ti. represented by samples {a.) and {g the GSW distance can be approximated as:
TI)
L N
1/v [ 01) (00())),)))'
(22)
where Vi[ni and Myjn] denote the sorted indices of (g(2;, (1)) and {o(ys 0,)), respectively, for each sampled 0j.
B Theoretical Justification
This section contains proof's of Proposition and Lemma from the main text (here they numbered as Proposition 9 and Lemma Though the statement Proposition 4 can be found in the literature, its formal treatment is omitted 4 59). Here we provide its detailed proof for Borel probability measures on IR. restricts us to non-decreasing. right-continuous cumulative distribution functions
F. Fig 10
t
Figure 10: Example of right continuous non-decreasing function.
We need Proposition 4 for efficient sampling, allows one avoid computing the inverse CDF. First we prove Lemmas and

---

Lemma 1. Let F be cumulative distribution function (CDF) on R, and let F (t) = inf IR]
F(x) f be its quantile function for
[0, 1]. Then:
{ [0, 1] F (1) و = [1E [0,1] F(a) 24
(23)
Proof. L.H.S. + R.H.S.:
Suppose t' ft [0. (t) ك a}. Then, there exists x' (t' such that x' S By the definition of the quantile function F (t'(t,, x' is the infimum of the set {ar t' S F(x)). Under the assumptions that F is right-continuous, the infimum x' belongs to the set, and therefore
F(x') 2 t'. Since F F(r) is non-decreasing and a 2 x', it follows that F(a) 2 F(x') 2 t'. Hence,
E (t E [0, F(a) 2 1).
R.H.S. L.H.S.:
Suppose t' E {t E [0 F(a)= 2 t). but & {t E [0.1] (t) S a}. i.c. t' such that I' F(a) and ((- > a. However, by the definition of x' x'= F-) is the infimum of the set {or F(x) f... Since x', cannot belong to this set, implying F(a) l', which contradicts the assumption t' S F(a). Thus, there is no in the R.H.S. that does not also belong to the L.H.S.
From these, conclude that the two sets are equal:
(L E [0, 1] F (1) s (LE [0 F(a) 2 24.
(24)
Lemma 2. Let F be cumulative distribution function (CDF) on TR. Then the quantile function Ft) = inf (I E & F(x)= 2 t}, defined for E [0, 11, is measurable with respect to the Borel
sigma algebra.
Proof. To show that (t): ((0, 1]. Bo,11) - (R, BR) is measurable, we must prove that for any Borel set B C R, the preimage:
e 1] F (t) E B} E B
(25)
The Borel sigma algebra Bg is generated by intervals of the form (-oo, b). Hence, it suffices to prove that for any b R. the set
E [0, (() E (-)0,
(26)
is measurable.
Consider the preimages of (-o0, b]:
{t = [0, 1] F (- =
= (1 [0,1] (1) <b] = /by Lemma
(27)
E [0, F(b) 21 -(0,F(M)
Since is CDF, F(b) is real number in [o. 1], and the set (t E [0, 1 F(b)) = [0, F(b) is
a Borel set in [0, 1). and thus the preimage of (-oc, bj is a measurable set in [0, 1]. Therefore the quantile function F()(() () is measurable.
Lemma 3. Let a and be two real numbers. Then:
1 [ooza Iozul du,
- bl =
(28)
where Lazu is the indicator of the set {i E R
Proof. First, suppose 2 b. Consider three cases for u:
If u > a > b, then Tazu = 0 and Tazw O, so Lazu I02ul 0.
3

---

2. a >> b > u. then lozu = 1 and 1zu 1, so
Lozn Is2ul 0.
3. If a > u u>> b, then lazu = 1 and /22 0, so
lazu I121 =1
Therefore, the integral reduces to:
J
Mozu 10zul du
|
1 du b.
(29)
b
For the case > by similar argument, integral is not zero only when:
M1azu Ibzul = 1.
and therefore, the integral reduces to
/ 1a2a Ib2ul du = {(
(
du - a.
(30)
Thus, all cases:
la bl 1oozu Ibzul du.
(31)
Proposition 4. Let F and G be cumulative distribution distribbttio,fuunciion (CDFs) on R. Then:
f (11-10---100 di d (F(s)) G(x) (100000-01000--11000-C--
dx;
(32)
where F-1 and G-1 are the quantile functions (generalized inverse CDFs) of F and G, respectively.
Proof. Note, that by Lemma 2 both F-1 and G are measurable and therefore the L.H.S exists. By
Lemma 3 its absolute value can be represented as:
/ F (t)- (t) dt=
(33)
L.
/ |F 11122 La 11)22l du dt.
Using the property of indicator functions [r-1(t)2u Ip-1 the integral becomes:
S.
1F (0)22)) 16-1(122 du dt
J
/
-I + Lo 1(0)eul du
(34)
/ [1-1=-1005 + la (0) du dt.
where the last equality is correct since function under the Lebesgue integral can be changed on a set of measure zero. Using Lemma we rewrite indicators:
[1-ESPw + Legawwl du dl
(35)
By Fubini's theorem (justified as the integrand is non-negative and measurable), we can switch the order of integration:
[ - IIsFlw) + Tesorwl du.
(36)

---

Using the Lemma again we get:
(((FF()) + Lecotu) du
(37)
G((u) - -((u]] du.
Hence, we conclude:
l -t Gt) dt = / [F(x) - G(z)] dz.
(t)
(38)
Lemma 5 (Lemma in the main text) provides the theoretical foundation for our optimization procedure for multicimensional Borel probability measures μn, and μ on F.
Lemma 5. Let un, and μs be Borel probability measures the unit cube [0, 1]d R
lim SW (pm ()) O,
(39)
T-000
then Un converges weakly to μ. and all moments of pa converge to the moments of μ.
Proof. Consider the ball ((0, R) of radius R, that contains the unit cube. Then Borel probability measure on the cube [0, 1]d can be extended to the Borel probability measure on B(0, R) by assigning measure zero any Borel set outside of the cube.
Now we can use Lemma 1.4 from [60], which states that for the I-Wasserstein distance W1 there exists constant C. > 0 such that for all Borel probability measures μl, on B Rt
0 S Wi (14, v) & C. R SW1 v) u
(40)
Since μn, and μ are supported on the unit cube in Rd, we take R d, which is sufficient radius to bound the unit cube. From the assumption that lim SW SWi(un
O, we have:
lim 12+000
Ca Rafx
SW WWi( u)) uuatt = O.
(41)
Using the squeeze Theorem for (40) follows that:
lim Wi(Mnn (4) = 0.
(42)
By Definition 6,8 (iv) and Theorem 6.9 of 4. the convergence WI - implies that μl converges weakly μ. Specifically, for any xo B(, Rt) and all continuous functions &p with |g|| < C (1 x)), C C R one has
lim / p()) - (())d((z). 1=000
(43)
For our case d( .)) S 2R, so p bounded, and integration over the B(0, R) could be replaced with integration the unit cube by construction of our extension of pn, and μ.
Given (finite) multi-index (a1, .2,..., ad), one can define the moment:
ma -f 1''22" 22
2 ad dp(x).
(44)
Polynomial functions ((x)) = x are bounded and continuous on the unit cube because x; S for all i (1....,d) d), ensuring all terms T < 1. Thus, weak convergence implies that for all multi-indices ā,
/ xa dun(x))= = du((z),
lim #00
(45)
i.c., all moments of pn converge to the corresponding moments of component-wise.
5

---

Algorithm 2 Color Conditional Generation with Sliced Wasserstein Guidance for latent text-to-image diffusion
Require:
DDIM: Diffusion DDIM scheduler
88: UNet model
D: Decoder of the Variational Autoencoder
E: Encoder of the Variational Autoencoder
T: Text embeddings for conditioning
Let Reference image
Y Guidance scale factor
M: Number of optimization steps
Initialize ~ (0,
for in .0..... T 1) do
2:
<- 0 (tensor with same shape 2z)) 3: for j in (1....,MI do
4:
x/ +- It
5:
e + sp s0 T) i0 f- (,,,(,,L,x1))
6:
7:
Igen -- D(0o)
8:
P gen pixels from image(4gau)
9:
K <- 10
Number of slices
10:
for l in {L,..., K} do
11:
R + rand rotation_ rotation_matrix)
12:
PR + PT
g
pR f- pd R
14:
for d in (1.....3) do
15:
Trot f pialis d]
16:
lro + Pilli d]
cdf, get_ cet__cd((FFo)
18:
edf, y + get cdf(ymx)
+ mean cddf,- cdf,)
20:
end for
end for
22:
9u f- V (..((u) 2a
23:
gu f sid g
24:
u +- u Agn
25:
end for
26:
Ii f- Jt - u
27:
Econd +- 80 (x 5)
28:
Cuncond + 50 12 t, (8)
Eguided +- Guncond (Econd Cuncond)
30:
Tt. < DDIM Eguided, t, xx)
end for

---

Table 4: Texl-to-image generation conditioned on reference color distribution. Quantitative evaluation, SD1.5 2-Wasserstein distance between the color distributions measures color similarity, CLIP-IQA and CLIP-T are quality and content scores. All color transfer methods [8] 19 are applied to the Unconditional SD1.5 generations.
20
52
53
55 56
2- Wasseerstein distance
Content scores
Algorithm
mean + std of mean
CLIP-IQA [5]
CLIP-T
t
SW-Guidance SD-1.5 (ours)
0.0328 + 0.0003
0.2221 + 0.0029
0.2624 + 0.0017
hm-mkl-hm 52
0.0572 + 0.0011
0.2013 + 0.0030
0.2656 + 0.0017
hm [53
0.0896 + 0.0019
0.2054 + 0.0029
0.2700 0.0016
PhotoWCT2 20
0.1085 0.0016
0.1796 + 0.0026
0.2621 0.0016
ModFlows [54]
0.1182 + 0.0015
0.2035 + 0.0030
0.2640 + 0.0016
Colorcanny
ControlNet SD-1.5 61
0.1183 + 0.0016
0.1953 + 0.0025
0.2600 + 0.0018
MKL 55
0.1274 0.0018
0.1880 + 0.0028
0.2700 = 0.0016
CT [56
0.1412 + 0.0019
0.1826 + 0.0027
0.2713 = 0.0016
WCT2 18
0.1425 + 0.0018
0.1819 + 0.0026
0.2761 + 0.0016
PhotoNAS 19
0.1724 + 0.0017
0.2878 + 0.0027
0.2590 + 0.0015
InstantStyle SD-1.5 34
0.2802 0.0043
0.1891 + 0.0020
0.2554
+ 0.0018
Unconditional SD-1.5
0.4062 + 0.0063
0.2010 + 0.0023
0.2837 + 0.0016
C Additional results
Dependence on learning rate The effect of learning rates on the performance of sliced Wasserstein- based guidance is given in Fig- [6] The learning rate has significant impact the 2-Wasserstein distance, with an optimal value of 0.04, beyond which the loss plateaus and then increases. In contrast, the CLIP-IQA and CLIP-T metrics exhibit linear relationships with respect to the learning rate, suggesting no minimum or optimal value within the range tested.
Text prompts to control the color Using text prompts for controlling the color has several major issues. The first row Fig. 15] shows that the red color specified by the prompt often ignored. The second of Fig. 15 shows how the same prompt applied another control image produces completely different color distribution. also introduces content details due to connotative words like "denim", "warm" and "soft". Removing these words alters the colors, making the prompt design tedious. Please note, that color naming is often connotative, and words like "bloody red" and "lime" will introduce content details.
Content Diversity Evaluation To evaluate the content diversity of the generated images, we computed the FID between unconditional SDXL generations and those obtained using various style guidance methods. To mitigate potential effects color distribution alignment on the FID, all evaluations were conducted after conversion to grayscale histogram normalized images. The results on our generated dataset are summarized in Table
Table 5: FID scores between unconditional SDXL generations and stylized outputs.
Method Used with SDXL,
FID Score (vs. Unconditional)
Mean/Covariance Matching Only
53.16
SW-Guidance (Ours)
58.40
InstantStyle
58.95
IP-Adapter
71.06
RB Modulation
72.75
The results show that SW-Guidance maintains content diversity comparable to other state-of-the-arr stylization methods. While there is slight FID increase compared to simple moment matching (which provides weaker color control), our method preserves substantially more diversity than stronger stylization techniques such IP-Adapter and RB Modulation.
7

---

D Experimental Details
The experiments were conducted on images generated by SD-1.5 (Dreamshaper-8) and SDXL (RealVisXL-V4) using the first 1000 ContraStyles prompts 49 No negative prompts negative embeddings were used.
We fixed the CFG scale to and the resolution to 768x768 for SDXL. For SD-1.5, the CFG scale was set 8 and the resolution 512x512. Both the SDXL and SD pipelines used the DDIM scheduler with 30 inference steps. Images for RB- AB-Modulation were produced by Stable Cascade with resolution of 1024x 1024 and a total of 30 inference steps (20 for stage and 10 for stage B). Method-specific settings are provided below.
Baselines For InstantStyle, the SDXL and SDD11.5 scales set to 1.0. For IP-Adapter, the SDXL scale was set to 0,5 because higher scales tended to ignore the text prompt, producing variations of reference image. The Colorcanny ControlNet for SD-1.5 had a conditioning scale of 1.0. For SW-Guidance, the SD-1.5 learning rate was Ir = 0.04. In the SDXL version of SW-Guidance, we did not apply gradient normalization (line 23, Algorithm 2 and set the constant Ir 0.01 10 100.
For evaluation, we used publicly available models and algorithms (i.c., none of them were re-trained re-implemented). We ran color transfer baselines with the default settings provided by the authors.
We observed that PhotoNAS demonstrated a dependency on the resolution input images. Specifi- cally, the method was optimized for 512x512 inputs and exhibited noticeable variations in perfor- mance, including high-frequency defects when images of different resolutions were used. Therefore, the evaluations for SDXL and DreamShaper were different, as SDXL outputs images in higher resolutions.
Metrics The 2- Wasserstein distance was estimated with 3000 randomly sampled points using the "emd" function from the POT library [62]. The CLIP-IQA metric implementation was taken from the "piq" Python library [63]. The CLIP-T metric was calculated using the model "openai/clip-vit vit--lrge-- patch14" with an embedding dimension of 768.
The experiments were conducted on a single workstation equipped with two Nvidia RTX 4090 GPU accelerators and 256 GB of RAM.
Hardware
Prompts for illustrations
Fig. (main text):
1. Astronaut in jungle, detailed, 8k
2. A cinematic shot of cute little rabbit wearing jacket and doing thumbs up
3. extremely detailed illustration of steampunk train the station, intricate details, perfect environment
Fig. (main text):
1. Sunflower Paintings Sunflowers Painting by Chris Me Morrow Tuscan Sunflowers Fine Art...
2. b8547793944 Formal dress suit men male slim wedding suits for men double breasted mens suits wine red costume ternos masculino fashion 2XL
3. martino leather chaise sectional sofa 2 piece apartment and sets from china interio tucson dining room rustic furniture with home the company
4. 1125x2436 Rainy Night Man With Umbrella Scifi Drawings Digital Art
Fig. 17 72 and Fig. 17 (Appendix):
1. Woman with Parasol Madame Monet and Her Son Image: Monet woman with parasol right
2. """Iceland: Through an Artist's Eyes part Rainy Day Adventures original fine art by Karen Margulis"
3. Parthenon Poster featuring the digital art Parthenon Of Nashville by Honour Hall
4. How To Make A Caramel Frappuccino At Home

---

5. New York Central Building, Park Avenue, 1930, Vintage Poster, by Chesley Bonestell
Fig. 13
:ppendixx:
1. Francis Day- The Piano Lesson Frederick Childe Hassam The Sonata George Bellows- Emma at the Piano Theodore Robinson Girl At Piano Pierre-Auguste Renoir The Piano Lesson Louise Abbema At the Piano Gustave...
2. Illustration pour Girl retro military pilot pop art retro style. The army and air force. A woman the army image libre droit
3. Victor Tsvetkov The Bicycle Ride 1965 Russian Painting, Russian Art, Figure Painting, Bicycle Painting, Bicycle Art, Socialist Realism, Soviet Art, Illustration Art, Illustrations
4. There Was A Time Milwaukee, Wisconsin Horizons by Phil Koch USA"". """
5. Poster featuring the painting Monet Wedding by Clara Sue Beym

---

Unconditional
InstantStyle
IP-Adapter
RB-Modulation
Ours
Reference
Cafea
Figure 11: Text-to- Text-to-image generation conditioned reference color distribution. Comparison with stylized generation methods. Examples from the test set. All images are generated by RealVisXL except of RB-Modulation running on Stable Cascade. Other methods have greater mismatch in color distributions and also often transfer some composition details such as: a forest (first row), field of flowers (second row), bouquet (third row), mountains (fourth row), cloudy sky and mountains (last row).
10

---

Content (SD1.5)
MKL
PhotoWCT2
Modflows
hm
hmm-mk-hhm
Ours
Style
CA
Figure 12: Text-to-image generation conditioned on a reference color distribution. Qualitative comparison with color transfer methods for SD-1.5. Examples from the set. Please refer the Table 4 for the quantitative comparison.

---

Unconditional
ModFlows
hm-mkk-hm
hm
Ours
Reference
Figure 13: Text-to- Text-to-image generation conditioned reference color distribution. Comparison with color transfer methods. Examples from the test set. Color transfer methods (ModFlows, hm and hm-mkl-hm) are applied to the Unconditional RealVisXL. generations. Images produced by color transfer methods have greater mismatch in color distributions with the reference when compared to SW-Guidance.
12

---

control
SW-Guidance + ControlNet
reference
Figure 14: SW-Guidance combined with depth and canny controls.
control
Prompt ControlNet
prompt
cozy shades of soft off-white, red, denim blue with warm brown tones
cozy shades of saft off-white, red, denim blue with warm brown tones
shades
cozy, warm
denim
palette off-whits, red, blue and brown tones
Note: The absence connotative terms like 'cozy." 'denim, and Warm' efters color distribution.
Figure 15: Text prompt mimicking the color distribution of Fig
13

---

0.0600
2-Wassersten
2-Wasserstein Distance
0.0575
0.0550
0.0525
0.0500
0.240
CLIP-IQA Score Score
CLIP-IQA
0.235
0.230
0.225
0.26
CUPPT Score
CLIP-T
0.24
0.22
0.02
0.04
0.06
0.08
0.10
0.12
Leaming Rate
Figure 16: The performance metries dependence on the learning rate for SD-1.5.
(SDXL)
hm-mkk-hm
Stylc
CAIT
Figure 17: Text-to-image generation conditioned on reference color distribution. Qualitative comparison with color transfer methods for SDXL. Please refer to the Table in the main text for the quantitative comparison.
14


Reviews:
[{"review_id": "le68lZTlTs", "reviewer": "Reviewer_SYqP", "summary": "The authors propose SW-Guidance, a training-free and optimization/guidance-based method for color distribution conditioned image generation with T2I diffusion models. Particularly, since the color distribution of an image is defined by a CDF, the authors use the sliced Wasserstein distance to compare the color CDFs of the generated and reference images, defining a loss which is optimized w.r.t. the latent for every denoising step similar to Universal Guidance.", "questions": "1. Why was the 2-Wasserstein distance between CDFs chosen as the quantitative evaluation metric? Since the method directly optimizes 1-Wasserstein distance, it can seem like the method is directly optimizing for the quantitative evaluation metric, which makes it less good of a metric.\n2. From the examples shown in Figure 13 and 17, it seems like SW-Guidance slightly changes the content and structure of the generated image compared to vanilla T2I generation from the same seed, which makes sense for guidance. However, does SW-Guidance limit the diversity of T2I generation, where the color palette may encourage the output image to have a certain style, structure, or texture? I would love to see the authors investigate this possibly with FID (lower means more diverse generation) as this seems to echo the quality/category adherence and diversity tradeoff of classifier-free guidance. To possibly reduce effects of matching color distributions on FID, the authors can evaluate images after they have been converted to grayscale and/or passed through histogram normalization.\n3. These two questions are purely for discussion, but how does having to differentiate through the SDXL VAE affect the method’s inference time and VRAM usage, and would it be possible to find some way to compute color CDFs in the VAE latent space? Moreover, what are some other conditional generation applications/objectives the sliced Wasserstein guidance approach can be applied to? This can help illuminate the significance of the work.\n\nAddressing weaknesses 1 and 2, clarifying question 1, and providing some experiments and discussions on question 2 can push my score from a 4 to a 5. Discussions on question 3 would be well-appreciated too!", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "1. It would be great if the authors acknowledged existing optimization/guidance-based methods for T2I diffusion controllable generation, such as Diffusion Self-Guidance [1] and FreeControl [2], especially since both works enable scene appearance control which is adjacent to color control.\n2. It would also be great to see SW-Guidance compared to naive approaches to color transfer/stylization with a baseline of a simple loss function such as the following, which can more visually show the advantages of SW-Guidance’s loss function design: 1) L2 loss between the mean and standard deviation of intermediate diffusion features (which these two aforementioned methods use), or 2) L1/L2 loss between the CDFs of the generated and reference images without random rotations which come from the sliced Wasserstein distance.\n3. This is more of a suggestion: Algorithm 2 is essentially a much more detailed version of Algorithm 1. It would be nice to acknowledge Algorithm 2 when introducing Algorithm 1 as otherwise one has to dig into the appendix to realize that there is a more detailed version of the proposed method.\n\n[1] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, Aleksander Holynski. “Diffusion Self-Guidance for Controllable Image Generation.” NeurIPS 2023, https://arxiv.org/abs/2306.00986. \n\n[2] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, Bolei Zhou. “FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition.” CVPR 2024, https://arxiv.org/abs/2312.07536.", "was_revised": true, "final_justification": "The authors have addressed most of my questions and concerns, and I especially appreciate the added experiment to show how SW-Guidance is able to preserve output diversity compared to existing baselines. I urge the authors to incorporate the rebuttal into the final version of the paper. After reading other reviewers’ comments and the authors’ responses, I raise my final rating to Accept.", "rebuttal": "We thank Reviewer SYqP for the thoughtful feedback and constructive suggestions. \n\n\n**Acknowledgment of existing optimization/guidance-based methods like Diffusion Self-Guidance and FreeControl.** We appreciate this suggestion and agree that discussing these relevant works will better contextualize our paper. We note that while these methods excel at providing spatial and structural control, our work specifically targets the matching of an image's global color distribution, which is a distinct goal within the broader field of controllable generation. Also it is worth noting that color information is entangled with spatial features within attention maps, which makes it hard to control only color of the generated image without affecting other features.\n\n**On comparisons with other loss functions.** This is an excellent suggestion that will strengthen our ablation study. The SW distance and its variations guarantee weak convergence of the generated color distribution to the reference one. There is no such property for L2 loss on mean/std and L1/L2 loss on CDFs without random rotations.\nWe acknowledge that our paper did not consider attention-based or feature-map-based methods and thus we defer this investigation of applying loss on features for later investigation. Regarding showing that sliced wasserstein performs better we refer to ablation study, we compare our method with matching first moments of distributions.\n**Algorithm Presentation** We will add a reference to the more detailed Algorithm 2 from the appendix when we first introduce the main algorithm in the body of the paper.\n\n**Why was the 2-Wasserstein distance chosen for evaluation when the method optimizes for the 1-Wasserstein distance?**. We chose the 2-Wasserstein distance as it is a standard and widely used metric in the color transfer literature and in the broader field of density matching. However, to provide a more comprehensive and unbiased evaluation, we will expand our analysis to include additional metrics. As suggested by the reviewer in a later question, we will add FID scores. \n**Does SW-Guidance limit the diversity of T2I generation?** To provide a quantitative answer, we have followed the reviewer's suggestion and computed the FID scores between unconditional SDXL generations and those from various style guidance methods. The results, calculated on our generated dataset, are as follows:\n\n| Method Used with SDXL | FID Score (vs. Unconditional) |\n| :--- | :--- |\n| Mean/Covariance Matching Only | 53.16 |\n| **SW-Guidance (Ours)** | **58.40** |\n| InstantStyle | 58.95 |\n| IP-Adapter | 71.06 |\n| RB Modulation | 72.75 |\n\nThey show that SW-Guidance maintains content diversity on par with other state-of-the-art stylization methods. While there is a slight increase in FID compared to simple moment matching (which provides weaker color control), our method is significantly better at preserving diversity than stronger stylization methods like IP-Adapter and RB Modulation. \n\n**What is the computational overhead, and could CDFs be computed in latent space?**\nThis is an important practical consideration. We will add a more detailed analysis of the computational overhead in addition to ablation in Fig 7. For a standard SDXL generation used a peak of **13301 MiB** of VRAM on our evaluation hardware, whereas generation with SW-Guidance used **15641 MiB**.\n\nRegarding the use of latent-space CDFs, we agree this is an interesting direction. As color is an explicit property of the pixel space, computing exact CDFs requires decoding the image. However, one could explore lightweight, learnable models that approximate color statistics from latent features, which could improve efficiency in future work.\nFurthermore, we explored ways to mitigate overhead by avoiding backpropagation through the expensive U-Net. One potential direction we investigated involves leveraging the VAE decoder directly on the noised latent z_t at an intermediate timestep. While decoding a noised latent does not produce a semantically meaningful image, our findings suggest it still preserves some coarse color information to provide a useful guidance signal. However our preliminary study in this direction showed that in this case guidance effectively applied only on the later generation steps, which reduces color similarity and coherence of the generated image."}, {"review_id": "zGTsKACwXf", "reviewer": "Reviewer_B7v7", "summary": "Color Conditional Generation with Sliced Wasserstein Guidance adds a training free technique to generative (text--to-image) diffusion models to control the color palette of the generation using a reference image. Using a differentiable version of the general form of the Sliced Wasserstein distance to condition the diffusion process style or content are successfully disentangled from color. Specifically, the Diffusion Posterior Sampling scheme as formulated in prior works Universal Diffusion Guidance and FreeDoM is used to incorporate the Sliced Wasserstein distance as a measure between two probability distributions into the conditioned diffusion process. The probability distributions correspond to the color distributions of generated image and color reference. While the regular prompt-based conditioning is still applied in latent space, the color guidance is added on the decoded image. The 1-Sliced Wasserstein distance is used as it is computationally lighter, implemented as a random 1D projection and L1 distance between the CDFs.\nThe method is evaluated on Stable Diffusion (SD) 1.5 and SDXL. with prompts from the ContraStyles dataset and references from Unsplash Lite. Qualitative results look promising with a superior disentanglement of color and style compared to other methods. Quantitative measurements look favorable for SD 1.5 and a little bit worse on SDXL with state-of-the-art performance in the distribution metric.", "questions": "- It seems like the method might be difficult to scale to newer diffusion architectures. Would it also work with e.g. a recitifed flow model? What are the adjustments necessary?\n- How do papers like [1, 2]  relate to the claim that textual input is impractical for color control? It seems like adding RGB tuples to text prompts works for e.g. StableDiffusion 1.4. Adding [2] to the related works section would make sense, I think.\n- What do the results of the CLIP-T in Table 1 really mean? The text mentions the highest score for the SW Guidance which doesn't seem to be the case in the table here. A bit more discussion of the CLIP results could be interesting here.\n- What is happening in l.115? The sentence seems broken.\n- l. 185, 186: Isn't a good performance on the 2-Wasserstein distance expected after optimization for the 1-Wasserstein distance?\n\n1: Bordin et al. Fine color guidance in diffusion models and its application to image compression at extremely low bitrates. ArXiv 2024.\n2: Butt et al. ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglement. ECCV 2024", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The combination of the Sliced Wasserstein Distance, mostly known from traditional statistical image optimization tasks with the generative image diffusion paradigm is an original and novel idea.\n- Nice documentation of the method's limitations.\n- The paper is overall nicely written. Interesting choice to incorporate the related works fully into the introduction (which is somewhat uncommon for computer science papers these days) but it works well.\n- The method seems flexible and can, for example, be combined with the ControlNet architecture, as shown.\n\nWeaknesses:\n- The method needs to backpropagate through the entire U-Net of the diffusion model. Especially for larger (more recent) diffusion networks, this circumstance will make the approach infeasible. Showing a way to overcome this limitation would give this paper a significantly higher impact, I believe. \n\nOverall, the paper presents an original idea paired with a useful application. There are some minor questions left (see questions), but the method could be a good fit for the NeurIPS conference.", "was_revised": true, "final_justification": "With the corrections and improvements made that the authors announced in the rebuttal, the paper would be a relevant addition to the conference program. The proposed perceptual study certainly is a good addition.\nStill the scope of the work remains limited due to the paper mostly concentrating on Unet-based diffusion models and the need to backpropagate through the network. Therefore, my recommendation to accept the paper still holds but is not increased to a strong accept.", "rebuttal": "We sincerely thank Reviewer B7v7 for the positive evaluation. \n\n**Generality** We note that our guidance mechanism is architecturally agnostic and works with any generative model that is differentiable. For rectified flow models, the adaptation would be straightforward: our guidance term would be applied to the predicted clean data, `x_0`, in the same manner as it is in our current DDIM-based implementation. \n\n**How do papers like Bordin et al. and Butt et al. relate to the claim that text is impractical for color control?**\nThank you for pointing out these relevant works; we will add them to our related work section. We agree that text-based color control has its uses, but our claim is that it lacks the precision and fidelity required for matching complex color palettes. Its primary limitations, which our method overcomes, are:\n-   **Limited Precision:** Text is often insufficient for describing a full, nuanced color distribution.\n-   **Difficulty Specifying Distributions:** Text struggles to specify the relative amounts and interplay of multiple colors.\n\nWhile the approach by Bordin et al. is interesting, it focuses on spatially-aware color control (e.g., placing specific colors in specific regions), which is a different goal from our objective of matching a *global* color distribution and limits diversity of generation. In the paper we also discuss and compare to the similar approach of guiding with spatial color information [1], please refer to the Table 4 in Supplementary.\n\n[1] ghoskno. Color-canny controlnet. https://huggingface.co/datasets/ghoskno/ laion-art-en-colorcanny, 2023.\n\n**Results of the CLIP-T** Thank you for highlighting the lack of clarity; we will revise the text to be more precise. Our claim is that SW-Guidance achieves the highest CLIP-T score among methods that perform color/style conditioning, such as IP-Adapter, RB-Modulation and InstantStyle.\n| Method                         | CLIP-T Score ↑       |\n|--------------------------------|-----------------------|\n| SW-Guidance SDXL (Ours)        | 0.270 ± 0.002         |\n| RB-Modulation StableCascade    | 0.266 ± 0.003         |\n| InstantStyle SDXL              | 0.238 ± 0.002         |\n| IP-Adapter SDXL                | 0.214 ± 0.002         |\n\n**l.115** Thank you for catching this. We will correct this typo in the revised manuscript, which is supposed to be a discussion of the algorithm pictured in Fig2.\n\n**Isn't a good performance on the 2-Wasserstein distance expected after optimization for the 1-Wasserstein distance?** We use 2-Wasserstein as a standard evaluation metric in the field.  We will clarify this relationship in the paper."}, {"review_id": "QjuIdklnxs", "reviewer": "Reviewer_sChs", "summary": "The paper introduces SW-Guidance, a training-free control technique that steers a text-to-image diffusion sampler so that the generated image’s color distribution matches a given reference distribution provided as image. Building on Universal Diffusion Guidance, the key idea is to use at denoising steps a gradient signal from the Sliced 1-Wasserstein (SW) distance between the reference image and the current prediction.  Experiments are conducted on Stable Diffusion 1.5 and SD-XL and a subset of the ContraStyles dataset. The introduced method is compared to other histogram matching algorithms as well as established methods such as IP-Adapter and InstantStyle. Additionally, the authors showcase the compatibility of the introduced method in combination with ControlNet, introducing further (non-color) conditioning such as edge maps.", "questions": "- You note the need to differentiate through the UNet as a limitation. Have you experimented with rectified flow transformers such as Flux? Is Universal Guidance, and therefore your proposed approach, directly transferable to other architectures?", "limitations": "Yes, limitations are adequately addressed", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- While lightweight in the sense that no training or additional modules are required, the introduced method outperforms current approaches.\n- The method is well introduced, and the paper is generally well written.\n- Extensive experiments, including a wide range of baselines.\n- Thorough ablations of hyperparameters and analysis of factors such as generation time.\n- Limitations are well discussed.\n- One of the limitations of the proposed approach is that the color steering alters the image content compared to unconditioned generation. However, the paper demonstrates that the method is compatible with ControlNets, which may mitigate this limitation.\n- Publicly available code.\n\n**Weaknesses**\n\n- Compatibility with ControlNets is only demonstrated qualitatively.\n- Inference time overhead is relatively high.\n- The application appears to be quite niche, and the technical contribution is limited. Essentially, the proposed method combines Universal Guidance with the Sliced 1-Wasserstein distance.\n - The authors only consider UNet-based diffusion models and do not explore diffusion transformers or rectified flow transformers.", "was_revised": true, "final_justification": "The clarification regarding architectural limitations, the discussion of applicability beyond color conditioning, and the promised additional analysis on inference overhead are appreciated and would strengthen the paper. However, as no concrete evidence or results have been provided for any of these points, I will maintain my score of 4 (borderline accept).", "rebuttal": "We thank Reviewer sChs for the feedback.\n\n\n **On Niche Application** Our method can be applied to other generation problems conditioned on a distribution. An example of such a task is text-to-speech generation conditioned on a person’s voice. Audio encoders WavLM [2] and FACodec [3] encode patches of 20 ms, producing a distribution of speaker embeddings from a long recording. This distribution captures richer characteristics of the speaker’s voice than a single embedding. \n\nWhile we build upon existing ideas, we propose using the differentiable Sliced Wasserstein distance for guidance in diffusion models. This is a non-trivial adaptation that requires careful theoretical and empirical validation. Our work provides a practical, training-free solution that effectively isolates color, a common and important creative need.\n\n[2] S. Chen et al., arXiv:2110.13900. \n\n[3] Z. Ju et al., arXiv:2403.03100.\n\n **Inference Overhead** We thank the reviewer for raising these important points. We will expand our analysis to provide a more detailed breakdown of the computational overhead, as promised to other reviewers, in addition to ablation study in Fig 7. For a standard SDXL generation used a peak of **13301 MiB** of VRAM on our evaluation hardware, whereas generation with SW-Guidance used **15641 MiB**.\n\nFurthermore, we explored ways to mitigate overhead by avoiding backpropagation through the expensive U-Net. One potential direction we investigated involves leveraging the VAE decoder directly on the noised latent z_t at an intermediate timestep. While decoding a noised latent does not produce a semantically meaningful image, our findings suggest it still preserves some coarse color information to provide a useful guidance signal. However our preliminary study in this direction showed that in this case guidance effectively applied only on the later generation steps, which reduces color similarity and coherence of the generated image.\n\n**On Architecture Limitations** Regarding architectural generalizability, our method is not fundamentally limited to U-Nets. The core principle of our guidance is agnostic to the underlying network architecture. It requires only two conditions:\n1.  A differentiable image prediction process.\n2.  The ability to inject a guidance signal into the sampling loop.\n\nThese conditions are met by most modern generative architectures, including Diffusion Transformers (DiT) and Rectified Flow models (e.g., Flux). In all these cases, our SW-Guidance loss would be applied to the model's prediction of the clean image (`x_0`), making the adaptation straightforward."}, {"review_id": "xpDmjjp6Mw", "reviewer": "Reviewer_88qU", "summary": "This paper proposes SW-Guidance, a training-free method for color-conditional image generation that modifies the sampling process of diffusion models using Sliced 1-Wasserstein distance to match reference color distributions. The approach demonstrates superior color matching while maintaining semantic coherence with text prompts compared to existing color transfer and stylized generation methods. The authors provide theoretical justification through Proposition 1 and Lemma 2, and extensive experiments validate their claims across SD-1.5 and SDXL architectures.", "questions": "1.While the paper provides extensive quantitative evaluation, the absence of human perceptual studies (e.g., preference tests or perceptual color matching assessments) leaves open questions about subjective quality and practical utility compared to existing methods.\n\n2. The evaluation omits comparison with several recent state-of-the-art approaches in style transfer and prompt-aware stylization, making it difficult to fully assess the method's relative advantages.\n[1]. SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer\n\n3. While the paper evaluates traditional color transfer techniques (e.g., WCT2, PhotoWCT2) and diffusion-based stylization (e.g., IP-Adapter, InstantStyle), it omits critical discussion of ​​attention-based image editing methods​​ that achieve style alignment via feature adaptation or attention manipulation. A deeper comparative analysis would strengthen the paper by addressing:\n[1] Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing.\n[2] PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention.\n\n4. The caption text in Figure 4 appears noticeably blurred, which compromises the figure's effectiveness.\n\n5. typos issues in line 200.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\nThis work incorporates differentiable Sliced Wasserstein distance into diffusion model conditioning, providing a mathematically grounded solution to color-specific generation. The evaluation includes rigorous quantitative comparisons, showing statistically significant improvements. The method demonstrates compatibility with ControlNets and architectural agnosticism, working with both SD-1.5 and SDXL. The mathematical foundations are well-justified, with complete proofs provided in the Appendix.\n\nWeaknesses:\n1.While the paper provides extensive quantitative evaluation, the absence of human perceptual studies (e.g., preference tests or perceptual color matching assessments) leaves open questions about subjective quality and practical utility compared to existing methods.\n\n2. The evaluation omits comparison with several recent state-of-the-art approaches in style transfer and prompt-aware stylization, making it difficult to fully assess the method's relative advantages.\n[1]. SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer\n\n3. While the paper evaluates traditional color transfer techniques (e.g., WCT2, PhotoWCT2) and diffusion-based stylization (e.g., IP-Adapter, InstantStyle), it omits critical discussion of ​​attention-based image editing methods​​ that achieve style alignment via feature adaptation or attention manipulation. A deeper comparative analysis would strengthen the paper by addressing:\n[1] Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing.\n[2] PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention.", "was_revised": true, "final_justification": "The authors have addressed part of my concerns. I raise my initial score to borderline Accept.", "rebuttal": "We thank Reviewer 88qU for the review. We understand the concerns raised, and we appreciate the opportunity to address them directly. \n\n **On the Absence of Human Perceptual Studies**  To address this gap, we conduct a human evaluation study for the final version of the paper. Although we had limited time and could not conduct a large-scale user study, we designed a small experiment in which each participant was asked to vote for the image that best matched a given color style and textual description. Each participant rated 10 image sets : \n| Rank | Method               | Votes) |\n|------|----------------------|---------------------|\n| 1    | SW-Guidance   | 60                |\n| 2    | InstantStyle  | 15              |\n| 3    |  IP-Adapter    | 8                 |\n| 4    |  RB Modulation | 5                 |\n\n\n\n**On Missing Recent SOTA Comparisons** We will update our related work section to include a detailed discussion of SCSA, as well as the attention-based and flow-transformer editing methods suggested.\nWe note that while these methods are powerful for general style transfer or subject-driven editing, they often operate by manipulating internal features and are not explicitly designed to match a specific global color distribution with the precision that SW-Guidance offers. Our approach is distinct in its objective function, which directly minimizes the distance between color distributions in the pixel space, offering a different and more direct form of control. \n\n**On Technical and Presentation Issues** We sincerely apologize for blurred images and other oversights. The blurred figure was an unfortunate artifact of compressing our figures to meet the submission file size restrictions."}, {"review_id": "HK1i7d29xx", "reviewer": "Reviewer_rVkA", "summary": "The paper proposes SW-Guidance, a training-free method for color-conditional image generation that modifies the sampling process of diffusion models using the differentiable Sliced Wasserstein distance to align the color distribution of generated images with that of a reference image. Unlike existing approaches that entangle style and color or rely on prompt engineering, SW-Guidance offers direct control over color without transferring unwanted textures or stylistic features. It achieves state-of-the-art performance in color similarity and semantic alignment with text prompts, as demonstrated through both quantitative metrics and qualitative results. The method is compatible with popular diffusion models like Stable Diffusion 1.5 and SDXL, and supports integration with structural controls like ControlNet.", "questions": "1. The method relies heavily on quantitative metrics such as 2-Wasserstein distance and CLIP-based scores. However, these may not fully align with human judgment, especially for color similarity. Could the authors include a small-scale human evaluation to validate whether their method's color outputs are perceptually preferred over baselines? Even a limited user study would provide valuable insight into real-world perceptual alignment.\n\n2. The current experiments primarily focus on palettes with relatively uniform color characteristics. In practice, reference images often contain complex or mixed color themes. Can the authors provide either qualitative examples or additional results demonstrating SW-Guidance's robustness in cases where the reference image has multi-modal or ambiguous color distributions (e.g., natural scenes with mixed lighting or object diversity)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. The paper presents a theoretically grounded, training-free method for integrating Sliced Wasserstein distance into the diffusion sampling process. This cleanly isolates color as a conditioning target without affecting texture or structure, representing a novel and modular extension to guidance methods.\n\n2. Through comprehensive experiments on both SD1.5 and SDXL, the authors demonstrate superior performance in color fidelity, text alignment (CLIP-T), and visual quality. The inclusion of both stylization and color transfer baselines, along with ablation studies, supports the robustness of the approach.\n\n3. The method is fully training-free, easy to plug into existing diffusion pipelines, and supported by open-source code. Its compatibility with structural guidance (e.g., ControlNet) enhances its applicability to creative and controllable image generation scenarios.\n\nWeakness\n\n1. While the paper reports strong quantitative results, it does not provide any human studies or perceptual assessments. This limits insight into real-world alignment with human visual preferences, especially for color.\n\n2. The approach is evaluated on global color distributions, but does not explore real-world challenges such as multi-region palettes, spatial color layouts, or robustness to diverse lighting. This narrows its demonstrated applicability in practical design workflows.\n\n3. Although the paper mentions inference time, it lacks detailed analysis of gradient computation overhead (e.g., memory, runtime scaling with image size or slice count), which could impact adoption in latency-sensitive or large-scale settings.", "was_revised": true, "final_justification": "The authors have adequately addressed all my concerns, including perceptual validation and inference analysis. The planned additions (e.g., qualitative results for complex palettes) further strengthen the submission. I maintain my initial score of Accept.", "rebuttal": "We sincerely thank Reviewer rVkA for the positive feedback.\n\n**On Human Studies and Perceptual Assessment**  As we have noted in our responses to other reviewers, we believe a human perceptual study is a critical next step for validating our work. We conduct a human evaluation and will add it to the final version of the paper.  Although we had limited time and could not conduct a large-scale user study, we designed a small experiment in which each participant was asked to vote for the image that best matched a given color style. Each participant rated 10 image sets: \n| Rank | Method               | Votes) |\n|------|----------------------|---------------------|\n| 1    | SW-Guidance   | 60                |\n| 2    | InstantStyle  | 15              |\n| 3    |  IP-Adapter    | 8                 |\n| 4    |  RB Modulation | 5                 |\n\n\n**On Robustness to Complex Color Distributions** Please refer to Figure 4, that demonstrates applicability to the sharp palette. While a full quantitative analysis of these cases is extensive, we will add a new qualitative section to the appendix of our revised paper, showcasing SW-Guidance's performance multi-modal or ambiguous color distributions. \nWe also agree that region-specific color control is an interesting and adjacent problem, as noted by the reviewer. One could apply SW-Guidance for a specific region of the generated image, the only adjustment which is needed is the computations of several SW distances for every region and reference color distribution.\n\n **Inference Overhead** We thank the reviewer for raising these important points. We will expand our analysis to provide a more detailed breakdown of the computational overhead, as promised to other reviewers, in addition to ablation study in Fig 7. For a standard SDXL generation used a peak of **13301 MiB** of VRAM on our evaluation hardware, whereas generation with SW-Guidance used **15641 MiB**.\nFurthermore, we explored ways to mitigate overhead by avoiding backpropagation through the expensive U-Net. One potential direction we investigated involves leveraging the VAE decoder directly on the noised latent z_t at an intermediate timestep. While decoding a noised latent does not produce a semantically meaningful image, our findings suggest it still preserves some coarse color information to provide a useful guidance signal. However our preliminary study in this direction showed that in this case guidance effectively applied only on the later generation steps, which reduces color similarity and coherence of the generated image."}]

Author rebuttal:

Meta-review: This paper introduces SW-Guidance, a training-free method for controlling the color palette of images generated by diffusion models. The main idea is to guide the sampling process with a loss based on the Sliced 1-Wasserstein distance, which directly compares the color distribution of the generated image to that of a reference palette. The authors claim this technique effectively disentangles color from content, producing images that match the reference colors while staying true to the text prompt.

The paper's primary strengths are its novelty, strong performance, and thorough evaluation. The main idea of using Sliced Wasserstein distance for guidance was noted as original and theoretically sound (B7v7, rVkA). It is a lightweight, training free method, and outperforms more complex baselines in color matching while maintaining text coherence (sChs, rVkA). This was supported by extensive experiments and clear ablations (sChs, SYqP, rVkA), and the paper was consistently described as well-written and easy to follow (B7v7, sChs).

Initially, reviewers' concerns were less about the method's core function and more about its context and practical limits. Reviewers noted gaps in the evaluation, e.g., human perceptual studies (88qU, rVkA) and comparison against a broader set of controllable generation methods (SYqP, 88qU). The most significant technical weakness raised was scalability and overhead, as the required backpropagation through the entire U-Net is costly (B7v7, sChs).

The rebuttal was strong and resolved many issues. Still, the method's scalability and computational overhead remains the main unresolved weakness. The need to backpropagate through the entire U-Net is a limitation for future, larger architectures (B7v7, sChs).

Overall, the AC finds that this paper to introduce a novel, elegant, and effective training free method for color control that earned a strong consensus for acceptance from all five reviewers. While the computational overhead is a fair limitation on future scalability, it does not undermine the significance of the present work. A spotlight is recommended as the core idea of using Sliced Wasserstein guidance is a clever and principled technique with the potential to inspire similar distributional control methods in other domains, making it a work of broader interest.
Decision: Accept (spotlight)