RB-Modulation: Training-Free Stylization using Reference-Based Modulation

Authors: Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, Wen-Sheng Chu
Primary area: generative models
Keywords: Inverse Problems, Generative Modeling, Diffusion Models, Posterior Sampling, Optimal Control, Test-time Optimization

Abstract:
We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play solution for training-free personalization of diffusion models.
Existing training-free approaches exhibit difficulties in (a) style extraction from reference images in the absence of additional style or content text descriptions, (b) unwanted content leakage from reference style images, and (c) effective composition of style and content. 
RB-Modulation is built on a novel stochastic optimal controller where a style descriptor encodes the desired attributes through a terminal cost. 
The resulting drift not only overcomes the difficulties above, but also ensures high fidelity to the reference style and adheres to the given text prompt. 
We also introduce a cross-attention-based feature aggregation scheme that allows RB-Modulation to decouple content and style from the reference image.
With theoretical justification and empirical evidence, our test-time optimization framework demonstrates precise extraction and control of *content* and *style* in a training-free manner. 
Further, our method allows a seamless composition of content and style, which marks a departure from the dependency on external adapters or ControlNets. See project page:  https://rb-modulation.github.io/ for code and further details.

Decision: accept (oral)
Reviewer scores: 10, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730095023913

thread_type:
official_review

<thread_object>
creator: reviewer_ytqn

<content>
summary:
The authors propose RB-Modulation, a method that allows training free stylization and content-style composition with diffusion models. The authors reframe the personalization task as an optimal control problem, and propose an additional Attention Feature Aggregation (AFA) module that performs content/style disentanglement. The authors include extensive qualitative/quantitative results that demonstrate the effectiveness of their proposed method.

strengths:
1. AFAIK, personalization with diffusion models are inherently tricky in the sense that training based methods take a long time or training free methods lack in fidelity or fail in content/style disentanglement. The proposed RB-Modulation takes on a training free method that achieves both content/style disentaglement in a training free manner.

2. Evaluations are comprehensive and the proposed method surpasses previous SOTA methods. Qualitative results are especially impressive.

3. The paper is well written and easy to follow.

weaknesses:
1. The choice of the baseline model (StableCascade) seems like a design choice. Authors might want to provide additional results on more widely adapted baseline models (e.g. SDXL), since this can show whether the inherent high fidelity results stems from the potency of StableCascade or their proposed method.
2. Most of the qualitative results seem to be on generated reference style/content images. The proposed method seems to be agnostic to whether the reference image is generated or not. Results with non-generated reference images would be appreciated.
3. Authors use Consistent Style Descriptor (CSD) to extract style features. Are there alternatives for this module? What was the main reasoning behind this choice?
4. For content style composition, it seems like the content is also processed in the same manner as style. Is this optimal in terms of content identity preservation? In algorithm 1 and 2, it seems like the image latent is only updated by the loss w.r.t the style descriptor loss. Would this not result in information loss about the identity of the content?
5. In figure 1, the resulting images do not seem to hold the content identity of the reference content image, but instead seems to heavily rely on the given text description. Using examples from figure 2, additional content-style composition results on including/excluding $K_c, V_c$ would be appreciated.

questions:
1. Does RB-Modulation also work for recent flow based models? Does the standard reverse SDE for the OU process hold for flow based models, or will the optimal controller have to be redefined?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535285684

thread_type:
official_review

<thread_object>
creator: reviewer_icfz

<content>
summary:
The paper presents am encoder-based (i.e. fine-tuning free) method for style personalization/ customization in text2image diffusion models.  The approach (RB-Modulation) is to consider the reverse dynamics of diffusion as a stochastic control problem with a terminal cost objective that aligns extracted style features with the exemplar style features using a pre-existing  style descriptor [Somepalli, CVPR 2024].   A novelty of the approach is the ‘AFA module’ proposed which separates style and content using cross-attention to mitigate content leakage from the reference image.  The authors claim but do not comprehensively show this allows controlled integration of style and content elements.

strengths:
The formulation of the feed-forward instance personalization problem using a ‘optimal controller’ is novel over the majority of prior works in this category that focus up adaptation.  However the practicalities of the problem seem to demand iterative reverse/feed-forward steps which negates a lot of the benefit of theoretical arguments and underlying assumptions presented.

weaknesses:
Deeper comparisons could have been made to the NST literature which bear some relevance both in task and in approach particularly the recent diffusion based approaches.  Specifically the paper bears some similarity with prior works that partly reverse the diffusion process and then run it ‘forward’ again with style conditioning either from CLIP or from style descriptors.  For example ‘Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models’ [Wu et CVPR 2023) and PARASOL [Canet-Tarres et al., CVPR 2024] which uses ALADIN style descriptors for conditioning as does DIFF-NST [Ruta et al., ECCVW 2024].  Although the literature survey is quite broad in the areas of NST and style conditioned diffusion, there is limited focus on prior descriptor-based stylization work including these works and how the proposed approach fundamentally differs especially considering the use of cross-attention in AFA versus these works.  

The paper claims extensive experiments covering stylization and content-style composition as one of their three contributions.  It would have been helpful to see experiments exhibiting more nuanced control over style i.e. fine-grained variations of similar style, to exercise the controllability offered via the style descriptor conditioning.  Conditioning on text is sufficient to give coarse grain control over style (e.g. neon may be specified versus a descriptor for neon, as shown in the examples).  The fine-grained control offered by a continuous style descriptors in a feed-forward framework seems the main benefit but is not discussed or explored.  Similarly a matrix-like experiment showing descriptor interpolation and varying weights on the disentangled content/style would have been helpful to show the practical use/controllability of the approach.

questions:
Please see the above questions in the weakness section.  Overall this appears a technically sound paper but that does not fully contextualize its contribution in the literature or fully evidence its practicality.  The authors can consider addressing this in the rebuttal.

POST-REBUTTAL:  Additional works raised in this review have been discussed within Section 2 of the main paper to contextualize the contribution.  Additional experiments have been added to the supplemental material to show the visual results of continuous variation of style strength and of style interpolation.  I have raised my initial score accordingly to reflect a recommendation to accept the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730787774537

thread_type:
official_review

<thread_object>
creator: reviewer_nthn

<content>
summary:
This paper introduces reference-based modulation (RB-Modulation) for training-free personalization of diffusion models. The modulation builds on concepts from stochastic optimal control to modulate the drift field of reverse diffusion dynamics, incorporating desired attributes (e.g., style or content) via a terminal cost. Besides, the author also proposes Attention Feature Aggregation (AFA) module to decouple content and style in the cross-attention layers. The qualitative and quantitative results verify its effectiveness.

strengths:
1. The idea is novel. The author provides the first training-free personalization framework using stochastic optimal control. 
2. The author provides theoretical justifications connecting optimal control and reverse diffusion dynamics. 
3. The qualitative and quantitative results are very promising for not only stylization, but also content-style composition. The author also conducts user study to further verify its superiority.

weaknesses:
1. As the title said, the method aims for personalization. Therefore, I recommend the author add some comparison with classical personalization method, not just style transfer methods.
2. The section on ablation study in the paper is too brief. Considering that the AFA and SOC models are central to the article, I recommend that the authors include numerical comparisons and more qualitative comparisons.
3. For the content-style composition experiment, I recommend the author to provide more complicated ref content (dog, sloth, cat are simple cases, or you can change the color of the ref to see whether the proposed method has achieved better content-style composition) to verify its effectiveness.
4. For situations that require ControlNet to control the layout, can the method generalize well to these scenarios?

questions:
Please refer to the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730979926511

thread_type:
official_review

<thread_object>
creator: reviewer_ufzj

<content>
summary:
This paper borrows some concepts from optimal control and applies them to the diffusion models, which are training-free. The method is SOTA for the problems attempted on image stylization and composition.

strengths:
+ The paper presents theoretical underpinnings from optimal control. It uses those ideas to solve the training-free stylization problems similar to Deep Image Prior, which is somehow not cited in the paper.

+ The work has several merits, especially in the problem of prompt-based image stylization in the training-free framework, though a pre-trained diffusion model is used for generation. 

+ The title RB-Modulation is pretty weird and does not suit the paper's excellent contribution. This must be revised to put the paper in the correct research context. 

+ The idea and theoretical contribution are quite good, and this work could lead to more interest in similar works.

weaknesses:
- Title needs revision

- Missing references such as Deep Image Prior need to be cited.

questions:
* Sorry to quote from your limitation. However, I am curious to know how feature descriptors can help this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732767944391

thread_type:
public_comment

<thread_object>
creator: ~yuchen_liang4

<content>
title:
Missing Reference 1/2

comment:
Dear Authers and Reviewers,

I think this article is just a special case of "FreeDom: Training-Free Energy-Guided Conditional Diffusion Model" published at ICCV 2023. FreeDom is a well-known  work in this field which was completed a year ago. Frankly, the only innovation I see in this article is the proximal algorithm. It's hard to believe that the authors didn't notice FreeDom's work when they researched the literature.

Best, 

Yuchen
</content>
</thread_object>

</thread>
<thread>
creation date:
1732791536542

thread_type:
public_comment

<thread_object>
creator: ~yuchen_liang4

<content>
title:
Miss  Reference 2/2

comment:
Dear Authers and Reviewers,

I find that the proximal algorithm 2 proposed in this paper is also an existing method in the field of image processing. 
It's a technique that's been used many times on other missions. In particular, "Manifold preserving guided diffusion" has tried this technique on FreeDom including stylization  task.

H. Chung, S. Lee, and J. C. Ye, “Decomposed diffusion sampler for accelerating large-scale inverse problems,” arXiv preprint arXiv:2303.05754, 2023.

Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte, and L. Van Gool, “Denoising diffusion models for plug-and-play image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1219–1229.

Chung H, Ye J C, Milanfar P, et al. Prompt-tuning latent diffusion models for inverse problems[J]. arXiv preprint arXiv:2310.01110, 2023.

Y. He, N. Murata, C.-H. Lai, Y. Takida, T. Uesaka, D. Kim, W.-H. Liao, Y. Mitsufuji, J. Z. Kolter, R. Salakhutdinov et al., “Manifold preserving guided diffusion,” in The Twelfth International Conference on Learning Representations, 2023.

Best,

Yuchen
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the task of image Stylization using Reference-Based Modulation (RB-Modulation). The proposed approach is based on the proposed stochastic optimal controller, resulting in drift term ensuring high fidelity to reference image style and content in the text prompt. Cross-attention-based feature aggregation was proposed for decomposing the style and content from the reference image. The experimental results show the good control of the content and style composition in the generated image. This work received overall positive comments from the reviewers, with final scores of 10, 8, 6, 8. The proposed RB-Modulation approach is training-free, demonstrating sufficient novelty and effectiveness in controlling the content-style composition with diffusion models. It can be accepted based on these contributions and the consensus of acceptance recommendations from reviewers.