Title: Color Conditional Generation with Sliced Wasserstein Guidance
Venue: NeurIPS 2025
Accepted: yes
Keywords: color conditional generation

diffusion models

sliced Wasserstein distance

optimal transport
Abstract: We propose SW-Guidance, a training-free approach for image generation conditioned on the color distribution of a reference image. While it is possible to generate an image with fixed colors by first creating an image from a text prompt and then applying a color style transfer method, this approach often results in semantically meaningless colors in the generated image. Our method solves this problem by modifying the sampling process of a diffusion model to incorporate the differentiable Sliced 1-Wasserstein distance between the color distribution of the generated image and the reference palette. Our method outperforms state-of-the-art techniques for color-conditional generation in terms of color similarity to the reference, producing images that not only match the reference colors but also maintain semantic coherence with the original text prompt. 
Our source code is available at https://github.com/alobashev/sw-guidance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SW-Guidance, a training-free method for color-conditional image generation that modifies the sampling process of diffusion models using the differentiable Sliced Wasserstein distance to align the color distribution of generated images with that of a reference image. Unlike existing approaches that entangle style and color or rely on prompt engineering, SW-Guidance offers direct control over color without transferring unwanted textures or stylistic features. It achieves state-of-the-art performance in color similarity and semantic alignment with text prompts, as demonstrated through both quantitative metrics and qualitative results. The method is compatible with popular diffusion models like Stable Diffusion 1.5 and SDXL, and supports integration with structural controls like ControlNet.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper presents a theoretically grounded, training-free method for integrating Sliced Wasserstein distance into the diffusion sampling process. This cleanly isolates color as a conditioning target without affecting texture or structure, representing a novel and modular extension to guidance methods.\n\n2. Through comprehensive experiments on both SD1.5 and SDXL, the authors demonstrate superior performance in color fidelity, text alignment (CLIP-T), and visual quality. The inclusion of both stylization and color transfer baselines, along with ablation studies, supports the robustness of the approach.\n\n3. The method is fully training-free, easy to plug into existing diffusion pipelines, and supported by open-source code. Its compatibility with structural guidance (e.g., ControlNet) enhances its applicability to creative and controllable image generation scenarios.\n\nWeakness\n\n1. While the paper reports strong quantitative results, it does not provide any human studies or perceptual assessments. This limits insight into real-world alignment with human visual preferences, especially for color.\n\n2. The approach is evaluated on global color distributions, but does not explore real-world challenges such as multi-region palettes, spatial color layouts, or robustness to diverse lighting. This narrows its demonstrated applicability in practical design workflows.\n\n3. Although the paper mentions inference time, it lacks detailed analysis of gradient computation overhead (e.g., memory, runtime scaling with image size or slice count), which could impact adoption in latency-sensitive or large-scale settings.\n\nQUESTIONS:\n1. The method relies heavily on quantitative metrics such as 2-Wasserstein distance and CLIP-based scores. However, these may not fully align with human judgment, especially for color similarity. Could the authors include a small-scale human evaluation to validate whether their method's color outputs are perceptually preferred over baselines? Even a limited user study would provide valuable insight into real-world perceptual alignment.\n\n2. The current experiments primarily focus on palettes with relatively uniform color characteristics. In practice, reference images often contain complex or mixed color themes. Can the authors provide either qualitative examples or additional results demonstrating SW-Guidance's robustness in cases where the reference image has multi-modal or ambiguous color distributions (e.g., natural scenes with mixed lighting or object diversity)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SW-Guidance, a training-free method for color-conditional image generation that modifies the sampling process of diffusion models using Sliced 1-Wasserstein distance to match reference color distributions. The approach demonstrates superior color matching while maintaining semantic coherence with text prompts compared to existing color transfer and stylized generation methods. The authors provide theoretical justification through Proposition 1 and Lemma 2, and extensive experiments validate their claims across SD-1.5 and SDXL architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\nThis work incorporates differentiable Sliced Wasserstein distance into diffusion model conditioning, providing a mathematically grounded solution to color-specific generation. The evaluation includes rigorous quantitative comparisons, showing statistically significant improvements. The method demonstrates compatibility with ControlNets and architectural agnosticism, working with both SD-1.5 and SDXL. The mathematical foundations are well-justified, with complete proofs provided in the Appendix.\n\nWeaknesses:\n1.While the paper provides extensive quantitative evaluation, the absence of human perceptual studies (e.g., preference tests or perceptual color matching assessments) leaves open questions about subjective quality and practical utility compared to existing methods.\n\n2. The evaluation omits comparison with several recent state-of-the-art approaches in style transfer and prompt-aware stylization, making it difficult to fully assess the method's relative advantages.\n[1]. SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer\n\n3. While the paper evaluates traditional color transfer techniques (e.g., WCT2, PhotoWCT2) and diffusion-based stylization (e.g., IP-Adapter, InstantStyle), it omits critical discussion of ​​attention-based image editing methods​​ that achieve style alignment via feature adaptation or attention manipulation. A deeper comparative analysis would strengthen the paper by addressing:\n[1] Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing.\n[2] PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention.\n\nQUESTIONS:\n1.While the paper provides extensive quantitative evaluation, the absence of human perceptual studies (e.g., preference tests or perceptual color matching assessments) leaves open questions about subjective quality and practical utility compared to existing methods.\n\n2. The evaluation omits comparison with several recent state-of-the-art approaches in style transfer and prompt-aware stylization, making it difficult to fully assess the method's relative advantages.\n[1]. SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer\n\n3. While the paper evaluates traditional color transfer techniques (e.g., WCT2, PhotoWCT2) and diffusion-based stylization (e.g., IP-Adapter, InstantStyle), it omits critical discussion of ​​attention-based image editing methods​​ that achieve style alignment via feature adaptation or attention manipulation. A deeper comparative analysis would strengthen the paper by addressing:\n[1] Unveil Inversion and Invariance in Flow Transformer for Versatile Image Editing.\n[2] PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention.\n\n4. The caption text in Figure 4 appears noticeably blurred, which compromises the figure's effectiveness.\n\n5. typos issues in line 200.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SW-Guidance, a training-free control technique that steers a text-to-image diffusion sampler so that the generated image’s color distribution matches a given reference distribution provided as image. Building on Universal Diffusion Guidance, the key idea is to use at denoising steps a gradient signal from the Sliced 1-Wasserstein (SW) distance between the reference image and the current prediction.  Experiments are conducted on Stable Diffusion 1.5 and SD-XL and a subset of the ContraStyles dataset. The introduced method is compared to other histogram matching algorithms as well as established methods such as IP-Adapter and InstantStyle. Additionally, the authors showcase the compatibility of the introduced method in combination with ControlNet, introducing further (non-color) conditioning such as edge maps.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- While lightweight in the sense that no training or additional modules are required, the introduced method outperforms current approaches.\n- The method is well introduced, and the paper is generally well written.\n- Extensive experiments, including a wide range of baselines.\n- Thorough ablations of hyperparameters and analysis of factors such as generation time.\n- Limitations are well discussed.\n- One of the limitations of the proposed approach is that the color steering alters the image content compared to unconditioned generation. However, the paper demonstrates that the method is compatible with ControlNets, which may mitigate this limitation.\n- Publicly available code.\n\n**Weaknesses**\n\n- Compatibility with ControlNets is only demonstrated qualitatively.\n- Inference time overhead is relatively high.\n- The application appears to be quite niche, and the technical contribution is limited. Essentially, the proposed method combines Universal Guidance with the Sliced 1-Wasserstein distance.\n - The authors only consider UNet-based diffusion models and do not explore diffusion transformers or rectified flow transformers.\n\nQUESTIONS:\n- You note the need to differentiate through the UNet as a limitation. Have you experimented with rectified flow transformers such as Flux? Is Universal Guidance, and therefore your proposed approach, directly transferable to other architectures?\n\nLIMITATIONS:\nYes, limitations are adequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nColor Conditional Generation with Sliced Wasserstein Guidance adds a training free technique to generative (text--to-image) diffusion models to control the color palette of the generation using a reference image. Using a differentiable version of the general form of the Sliced Wasserstein distance to condition the diffusion process style or content are successfully disentangled from color. Specifically, the Diffusion Posterior Sampling scheme as formulated in prior works Universal Diffusion Guidance and FreeDoM is used to incorporate the Sliced Wasserstein distance as a measure between two probability distributions into the conditioned diffusion process. The probability distributions correspond to the color distributions of generated image and color reference. While the regular prompt-based conditioning is still applied in latent space, the color guidance is added on the decoded image. The 1-Sliced Wasserstein distance is used as it is computationally lighter, implemented as a random 1D projection and L1 distance between the CDFs.\nThe method is evaluated on Stable Diffusion (SD) 1.5 and SDXL. with prompts from the ContraStyles dataset and references from Unsplash Lite. Qualitative results look promising with a superior disentanglement of color and style compared to other methods. Quantitative measurements look favorable for SD 1.5 and a little bit worse on SDXL with state-of-the-art performance in the distribution metric.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The combination of the Sliced Wasserstein Distance, mostly known from traditional statistical image optimization tasks with the generative image diffusion paradigm is an original and novel idea.\n- Nice documentation of the method's limitations.\n- The paper is overall nicely written. Interesting choice to incorporate the related works fully into the introduction (which is somewhat uncommon for computer science papers these days) but it works well.\n- The method seems flexible and can, for example, be combined with the ControlNet architecture, as shown.\n\nWeaknesses:\n- The method needs to backpropagate through the entire U-Net of the diffusion model. Especially for larger (more recent) diffusion networks, this circumstance will make the approach infeasible. Showing a way to overcome this limitation would give this paper a significantly higher impact, I believe. \n\nOverall, the paper presents an original idea paired with a useful application. There are some minor questions left (see questions), but the method could be a good fit for the NeurIPS conference.\n\nQUESTIONS:\n- It seems like the method might be difficult to scale to newer diffusion architectures. Would it also work with e.g. a recitifed flow model? What are the adjustments necessary?\n- How do papers like [1, 2]  relate to the claim that textual input is impractical for color control? It seems like adding RGB tuples to text prompts works for e.g. StableDiffusion 1.4. Adding [2] to the related works section would make sense, I think.\n- What do the results of the CLIP-T in Table 1 really mean? The text mentions the highest score for the SW Guidance which doesn't seem to be the case in the table here. A bit more discussion of the CLIP results could be interesting here.\n- What is happening in l.115? The sentence seems broken.\n- l. 185, 186: Isn't a good performance on the 2-Wasserstein distance expected after optimization for the 1-Wasserstein distance?\n\n1: Bordin et al. Fine color guidance in diffusion models and its application to image compression at extremely low bitrates. ArXiv 2024.\n2: Butt et al. ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglement. ECCV 2024\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose SW-Guidance, a training-free and optimization/guidance-based method for color distribution conditioned image generation with T2I diffusion models. Particularly, since the color distribution of an image is defined by a CDF, the authors use the sliced Wasserstein distance to compare the color CDFs of the generated and reference images, defining a loss which is optimized w.r.t. the latent for every denoising step similar to Universal Guidance.\n\nSTRENGTHS AND WEAKNESSES:\n1. It would be great if the authors acknowledged existing optimization/guidance-based methods for T2I diffusion controllable generation, such as Diffusion Self-Guidance [1] and FreeControl [2], especially since both works enable scene appearance control which is adjacent to color control.\n2. It would also be great to see SW-Guidance compared to naive approaches to color transfer/stylization with a baseline of a simple loss function such as the following, which can more visually show the advantages of SW-Guidance’s loss function design: 1) L2 loss between the mean and standard deviation of intermediate diffusion features (which these two aforementioned methods use), or 2) L1/L2 loss between the CDFs of the generated and reference images without random rotations which come from the sliced Wasserstein distance.\n3. This is more of a suggestion: Algorithm 2 is essentially a much more detailed version of Algorithm 1. It would be nice to acknowledge Algorithm 2 when introducing Algorithm 1 as otherwise one has to dig into the appendix to realize that there is a more detailed version of the proposed method.\n\n[1] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, Aleksander Holynski. “Diffusion Self-Guidance for Controllable Image Generation.” NeurIPS 2023, https://arxiv.org/abs/2306.00986. \n\n[2] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, Bolei Zhou. “FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition.” CVPR 2024, https://arxiv.org/abs/2312.07536.\n\nQUESTIONS:\n1. Why was the 2-Wasserstein distance between CDFs chosen as the quantitative evaluation metric? Since the method directly optimizes 1-Wasserstein distance, it can seem like the method is directly optimizing for the quantitative evaluation metric, which makes it less good of a metric.\n2. From the examples shown in Figure 13 and 17, it seems like SW-Guidance slightly changes the content and structure of the generated image compared to vanilla T2I generation from the same seed, which makes sense for guidance. However, does SW-Guidance limit the diversity of T2I generation, where the color palette may encourage the output image to have a certain style, structure, or texture? I would love to see the authors investigate this possibly with FID (lower means more diverse generation) as this seems to echo the quality/category adherence and diversity tradeoff of classifier-free guidance. To possibly reduce effects of matching color distributions on FID, the authors can evaluate images after they have been converted to grayscale and/or passed through histogram normalization.\n3. These two questions are purely for discussion, but how does having to differentiate through the SDXL VAE affect the method’s inference time and VRAM usage, and would it be possible to find some way to compute color CDFs in the VAE latent space? Moreover, what are some other conditional generation applications/objectives the sliced Wasserstein guidance approach can be applied to? This can help illuminate the significance of the work.\n\nAddressing weaknesses 1 and 2, clarifying question 1, and providing some experiments and discussions on question 2 can push my score from a 4 to a 5. Discussions on question 3 would be well-appreciated too!\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}