RB-Modulation: Training-Free Stylization using Reference-Based Modulation
Authors: Litu Rout
Yujia Chen
Nataniel Ruiz
Abhishek Kumar
Constantine Caramanis
Sanjay Shakkottai
Wen-Sheng Chu
Venue: ICLR 2025 Oral (2025)
Keywords: Inverse Problems
Generative Modeling
Diffusion Models
Posterior Sampling
Optimal Control
Test-time Optimization
Primary area: generative models
TL;DR: We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play test-time optimization method for training-free personalization (stylization and content-style composition) of diffusion models using stochastic optimal control.

Abstract: We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play solution for training-free personalization of diffusion models.
Existing training-free approaches exhibit difficulties in (a) style extraction from reference images in the absence of additional style or content text descriptions, (b) unwanted content leakage from reference style images, and (c) effective composition of style and content. 
RB-Modulation is built on a novel stochastic optimal controller where a style descriptor encodes the desired attributes through a terminal cost. 
The resulting drift not only overcomes the difficulties above, but also ensures high fidelity to the reference style and adheres to the given text prompt. 
We also introduce a cross-attention-based feature aggregation scheme that allows RB-Modulation to decouple content and style from the reference image.
With theoretical justification and empirical evidence, our test-time optimization framework demonstrates precise extraction and control of *content* and *style* in a training-free manner. 
Further, our method allows a seamless composition of content and style, which marks a departure from the dependency on external adapters or ControlNets. See project page:  https://rb-modulation.github.io/ for code and further details.

Published as a conference paper at ICLR 2025
RB RB-MODULATION: TRAINING-FREE STYLIZATION USING REFERENCE-BASEL MODULATION
Litu Rout Yujia Chen'
Nataniel Ruiz'
Abhishek Kumar
Constantine Caramanis Sanjay Shakkottai²
Wen-Sheng Chu
Google 2 UT Austin 3 Google DeepMind
[litu. rcutu.rout, constantine, htt.:aateee...........aaaasttaae wtt..yhhtattttaeeeeeeeaaaaa edu [liturcut, yufiachen, natanielruiz, r abhishk, xschul@googgee.com
ABSTRACT
We propose Reference- Based Modulation (RB-Modulation), a new plug-and-play solution for training-free personalization of diffusion models. Existing training- free approaches exhibit difficulties in (a) style extraction from reference images in the absence of additional style or content text descriptions, (b) unwanted content leakage from reference style images, and (c) effective composition of style and content. CB-Modulation is built on novel stochastic optimal controller where style descriptor encodes the desired attributes through a terminal cost. The result- ing drift not only overcomes the difficulties above, but also ensures high fidelity to the reference style and adheres to the given text prompt. We also introduce a cuss-atteetioon-asee feature aggregation scheme that allows RB-Modulation decouple content and style from the reference image. With theoretical justifica- tion and empirical evidence, our test-time optimization framework demonstrates precise extraction and control of content and sryle in training-free manner. Fur- ther, our method allows seamless composition of content and style, which marks a departure from the dependency external adapters or ControlNets. See project page https:///// / /rb modo-moollll- lation. g trithub... /o/ for code and further details.
INTRODUCTION
Text-to-image (T2I) generative models (Ramesh et al., 2021; Rombach et al., 2022: Saharia et al., 2022) have excelled in crafting visually appealing images from text prompts. These T2I models are increasingly employed in creative endeavors such visual arts (Xu et al., 2024), gaming (Pearce et al., 2023), personalized image synthesis (Ruiz et al., 2023; Huang et al., 2024a; Hu et al., 2021; Shah et al., 2023), stylized rendering (Sohn al., 2023; Hertz et al., 2023; Wang et al., 2024a; Jeong et al., 2024), and image inversion or editing (Ulyanov et al., 2018; Delbracio & Milanfar, 2023; Rout al., 2023b; 2024; Mokady et al., 2023). Content creators often need precise control over both the content and the style of generated images to match their vision. While the content of an image can be conveyed through text, articulating an artist's 's unique style characterized by distinct brushstrokes, color palette, material, and texture is substantially more nuanced. This has led to research on personalization through visual prompting (Sohn et al., 2023; Hertz et al., 2023; Wang et al., 2024a).
Recent studies have focused on finetuning pre-trained T2I models to learn style from a set of refer- ence images (Gal et al., 2022; Ruiz et al., 2023; Sohn et al., 2023; Hu et al., 2021). This involves optimizing the model's text embeddings, model weights, or both, using the denoising diffusion loss. However, these methods demand substantial computational resources for training or finetuning large-scale foundation models, thus making them expensive to adapt to new, unseen styles. Fur- thermore, these methods often depend on human-curated images of the same style, which is less practical and can compromise quality when only single reference image is available.
In training-free free stylization, recent methods (Hertz et al., 2023; Wang et al., 2024a; Jeong et al., 2024) manipulate keys and values within the attention layers using just reference style image. These methods face challenges in both extracting the style from the reference style image and ac- curately transferring the style to target content image. For instance, during the DDIM inversion step (Song et al., 202 la) utilized by StyleAligned (Hertz et al., 2023), fine-grained details tend to be compromised. To mitigate this issue, InstantStyle (Wang et al., 2024a) incorporates features from
"This work done during internship at Google.

---

Published as a conference paper at ICLR 2025
Reference
Reference style
A guitar
A piano
A butterfly
Askyscraper
lighthouse
kangaroo
dwarl
dragon
An
Figure I: Given single reference image (rounded rectangle), our method RB-Modulation offers blug-and-play solution for (a) stylization, and (b) content-style composition with various prompts while maintaining sample diversity and prompt alignment. For instance, given reference style image (e.g., "melting golden 3d rendering style") and content image (e.g.. "a dog"), our method adheres to the desired prompts without leaking contents (c.g., flower) from the reference style image and without being restricted to the fixed pose or layout of the reference dog image.
the reference style image into specific layers previously trained IP-Adapter (Ye et al., 2023). However, identifying the exact layer for feature injection in model is complex and universally applicable across models. Also, feature injection can cause content leakage from the style image into the generated content. Moving on to content-style composition, InstantStyle (Wang al., 2024a) employs a ControlNet (Zhang et al., 2023) (an additionally trained network) to preserve image lay- out, which inadvertently limits its diversity.
We introduce Siference-Basser Modulation (RB-Modulation), novel approach for stylization and composition that eliminates the need for training or finetuning diffusion models (e.g. Control- Net (Zhang et al., 2023) adapters (Ye et al., 2023; Hu et al., 2021)). Our work reveals that the reverse dynamies in diffusion models can formulated as stochastie optimal control problem. By incorporating style features into the controller's terminal cost, we modulate the drift field in diffu- sion model's reverse dynamics, enabling training-free personalization. Unlike conventional attention processors that often leak content from the reference style image, propose enhance the image fidelity via an Attention Feature Aggregation (AFA) module that decouples content from reference style image. We demonstrate the effectiveness of our method in stylization (Hertz al., 2023; Wang et al., 2024a; Jeong et al., 2024) and style+content composition, as illustrated in Figure (a) and (b), respectively. Our experiments show that RB-Modulation outperforms current SoTA methods (Hertz et al., 2023; Wang et al., 2024a) in terms of human preference and pompt--llggmmen metrics.
Our contributions are summarized follows:
We present reference-based modulation (BB-Modulation) a novel stochastic optimal con- trol based test-time optimization framework that enables training-free personalized style and content control, with a new Attention Feature Aggregation (AFA) module to maintain high fidelity the reference image while adhering to the given prompt (g4).
We provide theoretical justifications connecting optimal control and reverse diffusion dy- namics. We leverage this connection to incorporate desired attributes (e.g., style) in our controller's terminal cost and personalize T2I models in training-free manner ($5).
We perform extensive experiments covering stylization and content-style composition, demonstrating superior performance over SoTA methods in human preference metrics (56).
RELATED WORK
Personalization of T2I models: T2I generative models (Rombach et al., 2022; Podell et al., 2023; Pernias et al., 2024) generate high quality images from text prompts. Their text-following
2

---

Published as a conference paper at ICLR 2025
ability has unlocked new avenues in personalized content creation, including text-guided image editing (Mokady et al., 2023; Rout et al., 2024). solving inverse problems (Rout et al., 2023b; 2024), concept-driven generation (Ruiz et al., 2023; Tewel et al., 2023; Kumari et al., 2023; Chen et al., 2024), personalized outpainting (Tang et al., 2023), et al,,, 2024a; Wang et al., 2024b), and stylized synthesis (Sohn et al., 2023; Wang al., 2024a; Hertz et al., 2023; Shah et al., 2023). To tailor T2I models for a specific style (e.g., painting) or content (e.g., object), existing methods follow one of two recipes: (1) full finetuning (FT) or parameter efficient finetuning (PEFT) and (2) training-free, which we discuss below.
(Ruiz et al., 2024; Huang
Finetuning T21 models for personalization: FT (Ruiz et al., 2023: Everaert et al., 2023) and PEFT (Kumari et al., 2023; Hu et al., 2021; Sohn et al., 2023; Shah et al., 2023) methods excel at capturing style or object details when the underlying T21 model can be finetuned on a few (typically 4) reference images for few thousand iterations. PARASOL AAASOL..Tarree (Tarrés et al., 2024) requires supervised data via a cross-modal search to train both the denoising U-Net and projector network. Diff-NST (Ruta et al., 2023) trains the attention processor by targeting the "V' values within the denoising U-Net. The curation of supervised data and resource-intensive finetuning for every style or content makes these methods challenging for practical usage.
Training-free methods for personalization: Training-free free personalization methods are preferable to finetuming methods given the vastly faster time of execution. In StyleAligned (Hertz et al., 2023), a reference style image and a text prompt describing the style are used to extract style features via DDIM inversion (Song al., 2021a). Target queries and keys are then normalized using adaptive instance normalization (Huang & Belongie, 2017) based on reference counterparts. Finally, refer- ence image keys and values are merged with DDIM-inverted latents in self-attention layers, which tends to leak content information from the reference style image (Figure 2). Moreover, the need for textual description in the DDIM inversion step can degrade its performance. DiffusionDisen- tanglement (Wu et al., 2023) aims to reduce the approximation error in DDIM inversion by jointly minimizing a perceptual loss and a directional CLIP loss, which is prone to content leakage (Wang et al., 2024a). Swapping Self-Attentin replacing the target keys and values in self-attention layers with those from a reference style image. It still relies on DDIM inversion to cache keys and values of the reference style, which tends to compromise fine-grained details (Wang et al., 2024a). Both StyleAligned (Hertz et al., 2023) and SSA (Jeong al., 2024) require two reverse processes to share their attention layer features and thus demand significant memory. InstantStyle (Wang et al., 2024a) injects reference style features into specific cross-attention layers of IP-Adapter (Ye al., 2023), addressing two key limitations: DDIM inversion and memory-intensive reverse processes. However, pinpointing the exact layers for feature injection is complex, and may not generalize to other models. In addition, when composing style and content, InstantStyle (Wang et al., 2024a) relies on ControlNet (Zhang et al., 2023), which can limit the diversity of generated images to fixed layouts and deviate from the prompt.
(SSA) (Jeong et al., 2024) addresses these limitations by
Optimal Control:
Stochastic optimal control finds wide applications in diverse fields such molec- ular dynamics (Holdijk et al., 2024), economics (Fleming & Rishel, 2012), non-convex optimiza- tion (Chaudhari et al., 201 8), robotics (Theodorou et al., 201 1), and mean-field games (Carmona et al., 2018) Despite its extensive use, and recent works on its connections to diffusion based gen- erative models (Berner et al., 2024; Tzen & Raginsky, 2019; Chen et al., 2023), it has been less explored in training- free personalization. In this paper. we introduce a novel test-time optimization framework leveraging the main concepts from optimal control to achieve training- free personaliza- tion. A key aspect of optimal control is designing a controller to guide a stochastic process towards a desired terminal condition (Fleming & Rishel, 2012). This aligns with our goal of training-fres personalization, as we target a specific style or content at the end of the reverse diffusion process, which can be incorporated in the controller's terminal condition.
RB-Modulation overcomes several challenges encountered by SoTA methods (Hertz et al., 2023; Jeong et al., 2024; Wang et al., 2024a). Since RB-Modulation does not require DDIM inversion, retains fine-grained details unlike StyleAligned (Hertz et al., 2023). Using stochastic controller to refine the trajectory of single reverse process, it overcomes the limitation of coupled reverse processes (Hertz et al., 2023). By incorporating a style descriptor in our controller's 's terminal cost, eliminates the dependency on Adapters (Ye et al., 2023; Hu et al., 2021) or ControlNets (Zhang et al., 2023) by InstantStyle (Wang et al., 2024a).
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
Diffusion models consist of two stochastic processes: (a) noising process, modeled by Stochastic Differential Equation (SDE) known as forward-SDE: dX, = J(X,, 1) dt g g(X,, () dW,, Xo ~
Po. and (b) denoising process, modeled by the time-reversal of forward-SDE. under mild regularity conditions (Anderson, 1982), also known as reverse-SDE:
dX, = [f(x,, () - 9² (Xt, 1)V log t)] + t(,,,t)
X ~1, (I)
(W:)e20 is standard Brownian motion in a filtered probability space, (Q, (Ft) P), p(:,t) denotes the marginal density of p time and Vlog (r()) the corre- sponding score function. f(X,, t) and g t) are called drift and volatility, respectively. A popular choice of s(X,,() = -X, and g(x,, 1) = v2 corresponds to the well-known forward Ornstein- Uhlenbeck (OU) process.
Here, W
For T2I generation, the reverse-SDE (1) is simulated using a neural network (x,, t; 0) (Hyvärinen & Dayan, 2005; Vincent, 2011) to approximate Vx log p(x,,)) t). Importantly to accelerate the sampling process in practice (Song et al., 2021a; Karras et al., 2022: Zhang & Chen, 2022), the reverse-SDE (1) shares the same path measure with a probability flow ODE: dX, = [f(X,, 2 ()V (og:(X.: dl, where X1 ~ ~ (0, Ta)-
Personalized diffusion models either fully finetune 9 of s (x,, 1; 0) (Ruiz et al., 2023; Everaert et al., 2023), or train a parameter-efficien adapter AO for 8 (x,, t; + A0) on reference style images (Hu et al., 2021; Sohn et al., 2023; Shah et al., 2023). Our method does not finetune 0 or train A0. Instead, we derive a new drift field through a stochastic control that modulates the reverse-SDE (1).
METHOD
Personalization using optimal control: Normalize time t by the total number of diffusion steps T such that 0 < t < 1. Let us denote by u Rd [0. 1] + Rd controller from the admissible set of controls 24 C Rd. XI E Fd a state variable, cost, and h Rd + R the terminal cost of the reverse process (XF))=1- We show in g5 that training-free personalization can be formulated as a control problem where the drift of the standard reverse-SDE (1) is modified via RB-modulation:
R x Id x [0, 1] - R the transient
min | WEl 1,¹² (xx, w(x,, () di + (xx(()))), where
(2)
dxx" = ()) - (x,, t) log X t) + u(X;, 1)] di + ()d,, X ~ W [O)
Importantly, the terminal cost h-.), weighted by 7. captures the discrepancy in feature space be- tween the styles of the reference image and the generated image. The resulting controller u(:; t) modulates the drift over time satisfy this terminal cost. We derive the solution to this optimal con- trol problem through the Hamilton-Jacom Daccob--Bellman (HJB) equation (Fleming & Rishel, 2012); refer to Appendix A for details. Our proposed RB- 1B-Modulation Algorithm 1 has two key components: (a) stochastic optimal controller and (b) attention feature aggregation. Below, we discuss each in turn.
(a) Stochastic Optimal Controller (SOC): We show that the reverse dynamics in diffusion models can be framed as a stochastic optimal control problem with a quadratic terminal cost (theoretical analysis in 85). For personalization using reference style image X = Zo. we use a Contrastive Style Descriptor (CSD) (Somepalli et al.. 2024) to extract style features (x,)). Since the score functions 8 (x,, t; 0)=z V log (X,, f) are available from pre-trained diffusion models (Podell et al., 2023; Pernias et al., 2024), our goal is to add correction term u(:;t) to modulate the reverse- SDE and minimize the overall cost (2). We approximate X with its conditional expectation using Tweedie's formula (Efron, 2011; Rout et al., 2023b; 2024). Finally, we incorporate the style features into our controller's 's terminal cost as: h (Xg') (x,) - (E (x)xx))
Our theoretical results (65) suggest that the optimal controller can be obtained by solving the HJB equation and letting 7 + Dc. In practice, this translates to dropping the transient cost l (X", u(X,,, t), t) and solving (2) with only the terminal constraint, i.e.,
min min() (E 11
(3)
well
4

---

Published as a conference paper at ICLR 2025
Thus, solve (3) to find the optimal control u and use this controller in the reverse dynamics (2) to update the current state from X to X² A- (recall that time flows backwards in the reverse- SDE (1)). Our implementation of (3) is given in Algorithm 1, which follows from our theoretical insights.
Implementation challenge: For smaller models (Rombach et al., 2022), we can directly solve our control problem (3). However, for larger models (Podell et al., 2023; Pernias et al., 2024), the control objective (3) requires back propagation through the score network with tentatively billions of parameters. This significandly increases time and memory complexity (Rout al., 2023b; 2024).
We propose a test-time proximal gradient descent approach to address this challenge. The key in- gredient of our Algorithm is find the previous state X by modulating the current state X based on an optimal controller u². The optimal controller u' is obtained by minimizing the discrep- ancy in style between Xi [X X xz]. obtained using our controlled reverse-SDE (3), and the reference style image Zg- Motivated by this interpretation, an alternate Algorithm 2 avoids back propagation through s sx,,, t; f) by introducing a dummy variable Xo. which serves as proxy for X in the terminal cost. Instead of forcing Xa to be decided by the dynamics of the reverse- as in Algorithm 1, we allow it to be only approximately faithful to the dynamics. This is implemented by adding a proximal penalty, i.e. x = arg min xo || | (x,)) - v(xx) + A||x0 - E [X,] [ where the dyper-parameter À controls the faithfulness of the reverse dynamics. This penalty assumes that with a small step-size in (3), xj and [[X"" |X," = xz] will be close. Thus, Algorithm 2 enables personalization of large-scale foundation models, matching the speed of training-free methods and obtaining 5-20X speedup over training- -aaeed methods; see Table 4 in Appendix B.2 for details.
While prior works (Chung et al., 2023; Zhu al., 2023; He al., 2024) have used a proximal sampler in related settings, their underlying generative model is not personalized. We believe that this is an important reason why our method results in a significant speedup while satisfying the terminal constraints. Our paper takes the first step in personalizing the underlying generative model via a novel attention processor as discussed below.
(b) Attention Feature Aggregation (AFA): Let d denote the dimension of the latent variable Xr. fy the embedding dimension for query Q, and ny the output dimension of the hidden layer. Transformer-boe based diffusion models (Rombach et al., 2022; Podell et al., 2023; Pernias et al., 2024) consist of self-attention and cross-attention layers operating on latent embedding x, E Rdxas. Within the attention module Attention(22 Q, K, V), x; is projected into queries Q E Rddx,, keys K E Rdx",, and values V E Rd using linear projections. Through Q K, and V, attention layers capture global context and improve long-range dependencies within x,.
To incorporate a reference image (e.g., style or content) while retaining alignment with the prompt, introduce the Attention Feature Aggregation (AFA) module. Given prompt p. reference style image J, and a reference content image Ia. we first extract the embeddings using CLIP text encoder (Radford et al., 2021) and CSD image encoder (Somepalli al., 2024). These embeddings are projected into keys and values using linear projection. We denote by K₂ and Vú, the keys and values from p. K₂ and V, from L,, Ko and V. from Le (used only in content-styke composition). The query Q, derived from a linear projection of x,, remains consistent in the AFA module. To maintain consistency between text and style, we compose the keys and values of both text and style in our attention mechanism. The final output of the AFA module is given by
AFA Avg (Atent. Astgle. Atext+style) Atext Attention Q, [K; [pl.[V;VV Vpl),. Astyle = Attention(Q, [K; K.], Va]), style = Attention(Q, [K; Kp; K.), [V; V/s V.],,
where [K: Kp] E RR2dx indicates concatenation of K with K, along the number of tokens di- mension. For style-content composition, we process the content image Le the same way as the reference style image L,, and obtain another set of attention outputs:
AFA Avg (Atezt, Astyles Acontents Acontent++stylee.
Acontent = Attention(Q. [K; Ka), [V: Vo)), Acontent style = (Q, [K; K3; Kz), VV; V.l).
Importantly, the AFA module is computationally tractable it only requires the computation of multi-head attention, which is widely used in practice (Podell et al., 2023).
Disentangling content and style. In stylization (content described by text; style illustrated by reference style image), prior works (Hertz et al., 2023; Wang et al., 2024a) inject the entire refer- ence style image T. that does not disentangle content and style. However, our AFA module injects
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: RB-Modulation (Exact)
Algorithm 2: RB-Modulation (Proximal)
Input:
Diffusion sleps 7, reference prompi reference style
Input: DifTusion time steps T. reference prompt reference
image Ro; style descriptor (-())
styll image Rg, style descriptor (+)..
score network (..))
seore network s(-,..: 8)
Tunable parameter: Stepsize n. optimization steps
Tunable parameters: Stepsize v, optimization steps M,
Output: Personalized latent Xo
proximal strength
Initialize Xy < NN (0, Iz)
Output: Personalized latent X
for T
Initialize Xy <- .V (0, Ix)
Initialize controller a do
2 for t. = T to de
for m =
Compute posterior mean E[X X x. Initialize ope. variable xo for to M do Compute controller's cost ((x)) == (x0)
&. _ (1=02)
controlled state
@1 B,2 (x,, DD
X²" = var yau
(Ra, 1. p)
B |X xx]
var
h(X²)) (z0)
(
112 using Eq.
5
((x))
update controller
end
(x + A|Ixx E 6[x X ! x.
10
x
xx +
S optimally controlled state
Update optimization variable
Xa = X4 7/ ((x))
11
X var yoxt
x f p: 0) terminal state
8
end
12
DDIM X
one denoising update
X--1 DDIM xx)
denoising step
13
end
end
return X2"
return X
only the style features from I, using the style attention head of the Vision Transformer (VIT) in CSD (Somepalli al., 2024). The AFA module achieves content-style disentanglement by comput- ing separate attention maps for content from text and style from image. In this case, SOC does not handle content and focuses solely on style aspects by using the style attention head (T(.).
In content-style composition (content described by both text and a reference content image; style described by reference style image), the AFA module injects content (extracted from the reference content) and style features (from the reference style image) separately using their respective attention heads in the ViT (Somepalli et al., 2024). The SOC module controls content by minimizing the discrepancy between content features from the generated image and the reference content image, and style by minimizing the discrepancy between style features extracted from the generated and reference style image. This distinction from prior works enables our method to prevent leakage.
THEORETICAL JUSTIFICATIONS
Problem setup: We outline an approach to derive the optimal controller for a special case of our control problem (2). We substitute / <- 1-1 / to account for the time reversal in the reverse-SDE (1). Here, X ~ N (0. Ia) and X ~ Pdota- We consider the dynamic without the Brownian motion: dx,"' = eXX,' denotes the drift field. The optimal controller u² can be derived by solving the Hamilton- Bellman (HJB) equation (Fleming & Rishel, 2012; Basar et al., 2020), see Appendix A for details.
d), X Xo. where 0 S to t S tyy < 1 and v R x Rd x [tu, tx] Rd
Incorporating optimal control in diffusion: Following recent works (Kappen, 2008; Chen et al., 2023), consider dynamical system whose drift field minimizes a transient trajectory cost and terminal cost (weighted by 7) to ensure "closeness" to reference content æ (Appendix A. Propo- sition A.2 (Chen et al., 2023) outlines the optimal control in the limiting setting where ? co. Furthermore, suppose we replace .1 with its conditional expectation (discussed in Remark A.3), the resulting dynamic is the standard reverse- for the Orstein- Uhlenbeck (OU) diffusion process for a particular noise schedule. This connection between classic linear quadratic control and the standard reverse-SDE allows us to study other diffusion problems (e.g., personalization) through the lens of stochastic optimal control. For instance, we derive the optimal controller given reference style features y1 at the terminal time.
Orstein-Uhllnbeeck
Proposition 5.1. Suppose A Rexd be linear style extractor that operates on the terminal state X E IRd, Given reference style features yı, consider the control problem:
min ["1 11 1) dt + ||XX - -ill2 where dxp = =((X ttd X well Jtg
Then, in the limit when 7 oc, the optimal controller u² (1 A Ax2) following controlled dynamic: dxp= (A" A)=A) 1 AT Ax,) dl.
which yields the
1-t

---

Published as a conference paper at ICLR 2025
Reference style
Ours
InstantStyle
StyleAligned
StyleDrop
Ale
Sundoat
Breatee
FBR
Asew
AMH
Asvabtenc
Agaas
Figure 2: Qualitative results for stylization: A comparison with state-of-the-art methods (InstantStyle (Wang et al., 2024a), StyleAligned (Hertz et al., 2023), StyleDrop (Sohn et al., 2023)) highlights advantages in preventing information leakage from the reference style and adhering more closely to desired prompts.
Implication. The optimal controller depends on the reference style features y11 the terminal time, instead of the image content encoded in To simulate the controlled dynamic in practice, we use CSD (Somepalli et al., 2024) as a style feature extractor and replace y/1 with the style features extracted from the expected terminal state [[X as discussed in Appendix A.2.
Drift modulation through optimal controller: We then study control problem where the velocity field is linear combination of the state and the control variable. This problem is interesting to study because the reverse-SDD SDE dynamic of the standard OU process has drift field of the form: (X,, -X, 2V log pf Xt, l). For Gaussian prior X, ~ N (0, I), the law of the OU process satisfics V log [X,, t) -X,, and the corresponding drift field becomes e (X,, t) = X,, Our goal to modulate this drift field using controller u (X,", t). The result below provides the structure of the optimal control (again in the setting where the terminal objective known; see Appendix A1). Proposition 5.2. Suppose A E Rexd be linear style extructor that operates the terminal state X RRd. Lei Pt denote V VxV*(x,0) HJB equation (A. Given reference style features yı, consider the control problem:
||A yi112, where dxp"" [X," + u(X,, t)] dt, X zo. Then, the optimal controller becomes u" (t) Dr. where the instantaneous state X Xx and P satisfy the following coupled transitions:
mmim WELL 30
X1
xne
AT (Axi 11) elt + + (Axi -A)
Pr
7A" (Ax:
Summary. We build the connection between optimal control and reverse diffusion (see Ap- pendices A. for details). The general strategy is to derive the optimal controller with known terminal state, and then replace the terminal state the controller with its estimate using Tweedie's formula. For stylized models and Gaussian prior, the controllers have an explicit form. However in practice, the data distribution may not be Gaussian, and thus, we do not aim for a closed-form ex- pression to modulate the drift. This line of analysis, however, points to our method RB-Moun 1B-Modulation. As discussed in 84, we incorporate a style descriptor our controller's terminal cost and evaluate the resulting drift at each reverse time step either through back propagating through the score network (Algorithm 1), or an approximation based on proximal gradient updates (Algorithm 2).
6 EXPERIMENTS
Metrics: Evaluating stylized synthesis is challenging due to the subjective nature of style, mak- ing simple metrics inadequate. We follow two step approach: first using metrics from prior

---

Published as a conference paper at ICLR 2025
Table 1: User study: We report the % of human preference on ours vs. alternatives for overall
quality (OQ), style alignment (SA), and prompt alignment (PA), including ties where users couldn't t decide. Our method consistently outperforms alternatives, achieving higher scores in all metries.
Human
Ours vs. InstantStyle
Ours vs. StyleAligned
Ours vs. IP-Adapter
Preference (%)
0Q1 f
SA f
PA 1
0Q1 f
SA f
PA f
QQ1 f
SA f
PA
Alternative
39.8
38.5
39.5
24.4
27.8
29.4
8.1
20.1
8.3
Tie
9.3
6,4
7.3
8.8
7.1
5,8
6.9
4.8
4.5
RB-Modulation (ours)
51.0
55.1
53.3
66.9
65.1
64.9
85.0
75.1
87.2
works and then conducting human evaluation. To evaluate prompt-image alignment, we use CLIP-T score (Hertz et al., 2023; Sohn et al., 2023; Wang et al., 2024a) and ImageReward (Xu et al., 2024), which also consider human aesthetics, distortions, and object completeness. When style descrip- tion is provided, CLIP-T and ImageReward also capture style alignment. We assess style similarity using DINO (Caron et al., 2021) and content similarity using CLIP-I (Radford et al., 2021) as in prior work (Hertz et al., 2023; Ruiz et al., 2023; Sohn et al., 2023), and highlight their limitations in disentangling style and content performance in evaluation. Given the importance of human evalua- tion in T2I personalization (Hertz et al., 2023; Sohn et al., 2023; Ruiz et al., 2023; Shah et al., 2023; Jeong et al., 2024), we also conduct a user study though Amazon Mechanical Turk to measure both style and text alignment.
Datasets and baselines: We use style images from StyleAligned benchmark (Hertz et al., 2023) for stylization and content images from DreamBooth (Ruiz et al., 2023) for content-style composition. We base RB-Modulation on the recently released StableCascade (Pernias et al., 2024). We compare with three training-free ffree methods: InstantStyle (Wang et al., 2024a) -state-of-the-artt, IP-Adapter (Ye et al., 2023), and StyleAligned (Hertz et al., 2023). For completeness, we also compare with training-based methods StyleDrop (Sohn et al., 2023) and ZipLoRA (Shah al., 2023).
Implementation details: All experiments run on a single A NVIDIA GPU. We use the same syper-parameters for our method across tasks, and default settings for alternative methods as per their original papers. Details are provided in Appendix B.I.
6.1 IMAGE STYLIZATION
Qualitative analysis: This section describes image stylization experiments using a text prompt and a reference style image. Figure compares our method with SoTA training-free InstantStyle (Wang et al., 2024a) and StyleAligned (Hertz et al., 2023), and training-based StyleDrop (Sohn et al., 2023). Except for StyleDrop, which requires ~5 minutes of training per style, all methods, including ours, are training-free and complete inference in <1 minute. While all methods produce reasonable outputs, alternative methods encounter issues with information leakage. For instance, in the third row of Figure 2. StyleAligned and StyleDrop generate wine bottle and book resembling the smartphone in the reference style image. In the last row, StyleAligned leaks the house and the background of the reference image; InstantStyle exhibits color leakage from the house, resulting in similar-colored images. Our method accurately adheres to the prompt in the desired style. As illustrated in the second and the third row, our method generates only one glass of wine and a high- fidelity rubber duck, compared to basclines where extra items appear (wine bottles styled like the left smartphone) or incorrect styles (cartoon-style rubber duck).
User study: Given the subjective nature of this field, we conduct a user study on Amazon Mechan- ical Turk with 155 participants using 100 styles from the StyleAligned dataset (Hertz et al., 2023), collecting a total of 7,200 answers (8 responses for each question). Each user answers 3 questions comparing our method with an alternative method regarding (1) overall quality, (2) style alignment, and (3) prompt alignment (details in the Appendix B.8). Table 1 summarizes the percentage of human preferences for our method, the alternative method, or a tie. Our method consistently out- performs the alternatives, including the current SoTA method InstantStyle (Wang et al., 2024a). The preference rates over all three metrics highlight the effectiveness of our method RB-Modulation.
Quantitative analysis: Table 2 evaluates 300 prompts and 100 styles the StyleAligned dataset (Hertz et al., 2023) using three metrics, with and without style descriptions in the prompts. Our method outperforms others notably in the ImageReward metric, closely matching human aes- thetics assessment from the user study in Table 1. In addition, the CLIP-T score indicates our ef- fective alignment between generated images and text prompts. While IP-Adapter and StyleAligned
8

---

Published as a conference paper at ICLR 2025
Reference style
Content prompt
StableCascade
DirectConcat
AFA only
SOC only
AFA + SOC
"A cat"
in
"A piano"
Figure 3: Ablation study: We show the effectiveness of our different proposed components by sequentially adding them to vanila StableCascade (Pernias et al., 2024). DirectConcat involves concatenating reference image embeddings with prompt embeddings.
Table 2: Quantitative results for stylization: We compare alternative methods on three metrics: ImageReward (Xu et al., 2024) and CLIP-T (Radford et al., 2021) for prompt alignment, DINO (Caron et al., 2021) for style alignment. Note that DINO score does not capture information leakage, so higher scores are not necessarily better ($B.5).
IewgeReward f
CLIP-T score
DINO score
With style description?
No
Yes
No
Yes
No
Yes
IP-Adapter (Ye al., 2023)
-1.99
-1.51
0.21
0.26
0.89
0.89
StyleAligned (Hertz et al., 2023)
-0.68
0.01
0.26
0.31
0.80
0.85
InstantStyle (Wang et al., 2024a)
0.09
0.72
0.29
0.33
0.68
0.72
RB-Modulation (ours)
0.91
1.18
0.30
0.34
0.68
0.73
have higher DINO scores, their lower rating in ImageReward, CLIP-T and user preference expose information leakage from the reference style images. Nevertheless, our DINO score remains com- petitive with the leading method InstantStyle. Notably, all metrics show improvement with style deseriptions, particularly ImageReward, where leveraging style descriptions enhances prompt alignment. Our method achieves high ImageReward and CLIP-T score without style descrip- tions, suggesting robustness in prompt alignment without explicit style information the prompt.
Ablation Study: Figure 3 shows ablation study of the AFA and SOC modules. We include a baseline, "DirectConcat", which concatenates reference style embeddings with text embeddings in the cross-attention modules. DirectConcat mixes both embeddings, making it less effective in disentangling style from prompts (e.g., cat vs. lighthouse). While AFA or SOC alone mitigates this by modulating the reverse drift and attention modules (54), each has drawbacks. AFA alone fails to capture the cat's style accurately, and SOC alone misplaces elements, like "a lighthouse hat on the cat" and "a railroad trunk on piano". We observe consistent improvements with each module, with the best results when combined.
6.2 CONTENT-STYLE COMPOSITION
Since this paper primarily focuses on style-based personalization, we perform extensive experiments on stylization. To further demonstrate the versatility of our framework, we also explore content-style composition as an additional capability.
Qualitative analysis: Content-style composition aims preserve the essence of both content and style depicted in the reference images, while ensuring the resulting image aligns with a given text prompt. Figure compares our method against training-free InstantStyle (Wang et al., 2024a), IP- Adapter (Ye et al.. 2023), and training-based ZipLoRA (Shah et al., 2023). Notably, the training- free InstantStyle and IP-Adapter rely on ControlNet (Zhang et al., 2023), which often constrains their ability to accurately follow prompts for changing the pose of the generated content, such as illustrating "dancing" Figure 4(b), or "walking" (c). In contrast, our method avoids the need for ControlNet or adapters. and can effectively capture the distinctive attributes of both style and content images while adhering to the prompt to generate diverse images. In Figure 4(a), our method accurately captures elements like "table" and "river" that overlooked in InstantStyle and IP- Adapter. In addition, our method mitigates information leakage, as evidenced in Figure 4(b), where the trunk of the tree behind the sloth is erroneously captured by InstantStyle and IP-Atareer but not

---

Published as a conference paper at ICLR 2025
Ref. content
Reference styles
Reference styles
Reference styles
IP-Acapter
InsternStyle
ZIpLoRA
Ours
(a] dog dancing an table near the river*
(b) "A slath walking on the street"
(c) 'At walking"
Figure 4: Qualitative results for content-style composition: Our method shows better prompt alignment and greater diversity than training-free free methods IP-AAapter (Ye et al., 2023) and InstantStyle (Wang et al., 2024a), and have competitive performance with training-baac ZipLoRA (Shah al., 2023).
Table 3: Quantitative results for composition: In addition to stylization metrics,
CLIP-T score (Radford et al., 202 I) to evaluate content alignment with the reference image. Similar to DINO, CLIP-I could inflate test (Sohn al., 2023; Shah al., 2023) due content leakage, but does not correlate to user preference; higher scores do not indicate better human preference.
Reward
CLIP-T score
DINO score
CLIP-I score
IP-Adapter
-0.78
0.22
0.73
0.68
InstantStyle
-0.54
0.21
0.71
0.71
RB-Modulation (ours)
0.74
0.26
0.74
0.71
by ours. Compared to ZipLoRA (Shah al., 2023) that requires training of 12 LoRAs (Hu al., 2021) !) and additional merge layers for each composition, our method requires no training at all while yielding competitive or better results. For instance, our method effectively captures the 2D cartoon and 3D rendering styles as illustrated in Figures 4(a) and (b).
Quantitative analysis: Table shows quantitative evaluation using 50 styles from StyleAligned dataset (Hertz et al.,, 2023) and 5 contents from DreamBooth dataset (Ruiz et al., 2023). Unlike prior works (Hertz al., 2023; Sohn et al., 2023; Shah al., 2023; Ruiz al., 2023; Jeong et al., 2024) reporting either DINO and CLIP-I scores, we present both metrics and demonstrate comparable performance across them. Additionally, we obtain notably higher ImageReward score, which aligns closely with human aesthetics assessment as evidenced in $6.1 and (Xu et al., 2024), Consequently, we omitted a user study in this section. For more details, please refer to Appendix B.I.
CONCLUSION
We introduced Reference-Bace Based modulation (RB-Modulation), test-time optimization method for personalizing transformer- based diffusion models. RB- 1B-Muuulation builds on concepts from stochas- tic optimal control to modulate the drift field of reverse diffusion dynamies, incorporating desired attributes (e.g., style content) via a terminal cost. Our Attention Feature Aggregation (AFA) mod- ule decouples content and style in the cross-attention layers and enables precise control over both. In addition, we derived theoretical connections between linear quadratic control and the denoising diffusion process, which led to the creation of RB- 1B--Moulatiion. Empirically. our method outper- formed current state-of-th of-the-art methods in stylization and content-style composition. To our best knowledge, this the first training-free free personalization framework using stochastic optimal control, which marks the departure from external adapters ControlNets.
10

---

Stochastic Processes and their Published as a conference paper at ICLR 2025
8 BROADER IMPACT STATEMENT
Social impact: Image stylization and content-style composition based on diffusion models poten- tially have both positive and negative social impact. This technology provides easy-to-use tool to the general public for image generation which can help visualize their artistic ideas. On the other hand, our work on stylization and content-style composition poses a risk of generating arts that closely mimic or infringe upon existing copyrighted material, leading to legal and ethical issues. More broadly, our method inherits the risks from T2I models which are capable of generating fake contents that can be misused by malicious users.
We build on StableCascade (Pernias et al., 2024), which has a mechanism to filter offensive image generations. Our framework RB-M EB-Modulation inherits these safeguards. In addition, to mitigate misuse, we believe crucial to ensure the underlying model's safety, which may involve (i) watermarking Al-generated artworks and (ii) implementing NSFW filter to remove inappropriate contents.
Safeguards:
Reproducibility: The pseudocode and hyper- parameer details have been provided in the paper. The source code available on the project page: https: /tre:://rbbbmoee worrrbduull-..o lation gigggiitthtt thub. io/
ACKNOWLEDGMENTS
This research has been supported by NSF Grant 2019844, a Google research collaboration award, and the UT Austin Machine Learning Lab, Litu Rout has been supported by Ju-Nam and Pearl Chew Presidential Fellowship and George J. Heuer Graduate Fellowship from UT Austin.
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models. Applications, 12(3)331--22,, 1982.
Karl J. Astrom. Introduction to Stochastic Control Theory. Elsevier Science, 1971.
Tamer Basar, Sean Meyn, and William R Perkins. Lecture notes on control system theory and design. arXiv preprint arXiv:2007.01367, 2020.
Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective diffusion-based
generative modeling. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
/tttp:://pperevvete net/ forum? fooummidd=oijss330
René Carmona, François Delarue, al. Probabilistic theory of mean field games with applications 1-II. Springer, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pp.
2021.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. In Proceedings of the 40th International Conference on Machine Learning, pp. 4055-4075, 2023.
Pratik Chaudhari, Adam Oberman, Stanley Osher, Stefano Soatto, and Guillaume Carlier. Deep relaxation: partial differential equations for optimizing deep neural networks. Research in the Mathematical Sciences, 5:1-30 2018.
Tianrong Chen, Jiatao Gu. Laurent Dinh, Evangelos Theodorou, Joshua M Susskind, and Shuangfei Zhai. Generative modeling with phase stochastic bridge. In The Twelfth International Conference Learning Representations, 2023.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2025
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. arXiv preprint arXiv:2310.01110, 2023.
Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An altemattve to denoising diffusion for image restoration. Transactions on Machine Learning Research, 2023.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- lion, 106(496):1002-6614 2011.
Martin Nicolas Everaert, Marco Bocchio, Sami Arpa, Sabine Süsstrunk, and Radhakrishna Achanta. Diffusion in style. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2251-2261, 2023.
Wendell H Fleming and Raymond W Rishel. Deterministic and stochastic optimal control, vol- ume 1. Springer Science Business Media, 2012.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 2rXiv:2208.01611 2022.
Zinan Guo, Yanze Wu, Zhuowei Chen, Lang Chen, and Qian He. Pulid: Pure and lightning id customization via contrastive alignment. arXiv preprint arXiv:2404. 24X 2024.
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. The Twelfth International Conference on Learning Representa- tions, 2024. URL https: attps::/ooeenreviet nnt met/forummid-oo 1Bb.oooooo.
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. arXiv preprint arXiv:2312.02/33, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in 2020.
Neural Information Processing Systems,
Lars Holdijk, Yuanqi Du, Ferry Hooft, Priyank Jaini, Berend Ensing, and Max Welling. Stochastic optimal control for collective variable free sampling of molecular transition paths. Advances in Neural Information Processing Systems, 36, 2024.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Jiehui Huang, Xiao Dong, Wenhui Song, Hanhui Li, Jun Zhou, Yuhao Cheng, Shutao Liao, Long Chen, Yiqiang Yan, Shengeai Liao, et Consistentid: Portrait generation with multimodal fine- grained identity preserving. arXiv preprint arXiv:2404. 16771, 2024a.
Jiehui Huang, Xiao Dong, Wenhui Song, Hanhui Li, Jun Zhou, Yuhao Cheng, Shutao Liao, Long Chen, Yiqiang Yan, Shengeai Liao, et al. Consistentid: Portrait generation with multimodal fine- grained identity preserving. arXiv preprint arXiv:2404. 16771, 2024b.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Aapo Hyvärinen and Peter Dayan. Estimation of non non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Jacseok Jeong, Junho Kim, Yunjey Choi, Gayoung Lee, and Youngjung Uh. Visual style prompting with swapping self-attention. arXiv preprint arXiv:2402. 12974, 2024.
HJ Kappen. Stochastic optimal control theory. ICML, Helsinki, Radbound University, Nijmegen, Netherlands, 2008
12

---

Published as a conference paper at ICLR 2025
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 01126655-2655777 2022.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint arXiv:2210.02274, 2022.
Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Damiel Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam De- vlin. Imitating human behaviour with diffusson models. In The Eleventh International Confer- ence on Learning Representations, 2023. URL htttht:t///ponetette /ttp///ppeeeveeeeeeeeeeeeeeeeeee net/ forum? id= Pv1GPQZRrC8.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview wwt/forunte www.oooumnnngSddeseee......
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot eex-tt--image generation. In International Conference on Machine Learning. pp. 8821-883 PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEFF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Litu Rout, Advait Parulekar, Constantine Caramanis, and Sanjay Shakkottai. A theoretical jus- tification for image inpainting using denoising diffusion probabilistic models. arXiv preprint arXiv:2302.01217,, 2023a.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G Dimakis, and San- jay Shakkottai. Solving inverse problems provably via posterior sampling with latent diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023h. URL https: frtcss:/opeene http//eeeeeeeeeeee/eeeeeeeee..ep net forum? (itmmm/dddxxKdxfff-f
Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis. Sanjay Shakkottai, and Wen- Sheng Chu. Beyond first-order tweedie: Solving inverse problems using latent diffusion. In 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-
22510, 2023.
13

---

Published as a conference paper at ICLR 2025
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfrr Aberman. Hyperdreambooth: Hypernetorrss for fast personaliza- tion of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Viston and Puttern Recognition, pp. 6527-6536, 2024.
Dan Ruta, Gemma Canet Tarrés, Andrew Gilbert, Eli Shechtman, Nicholas Kolkin, and John Col- lomosse. Disf-nst: Diffusion interleaving for deformable neural style transfer. arXiv preprint arXiv:2307.04157, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11488) 2022.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. ZipLoRA: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311.13600 2023.
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation any style. In 37th Conference Neural Information Processing Systems (NeurIPS). Neural Informa- tion Processing Systems Foundation, 2023.
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geip- ing, Abhinav Shrivastava, and Tom Goldstein. Measuring style similarity in diffusion models. arXiv preprini arXiv:2404.01229 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffiusion implicit models. In Interna- tional Conference on Learning Representations, 2021a. URL htttpss:///peeeeeeettttt
forum? Sdgettiittttirrrr CHLP.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b. URL https: httpps://openreview net/
Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, et al. Realfill: Reference-driven generation for authentic image completion. arXiv preprint arXiv:2309.16666 2023.
Gemma Canet Tarrés, Dan Ruta, Tu Bui, and John Collomosse. Parasol: Parametric style control for diffusion image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2432-2442, 2024.
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to- image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Evangelos Theodorou, Freek Stulp, Jonas Buchli, and Stefan Schaal. An iterative path integral stochastic optimal control approach for learning robotic tasks. IFAC Proceedings Volumes, 44(1): 11594-11601, 2011.
Belinda Tzen and Maxim Raginsky. Theoretical guarantees for sampling and inference in generative models with latent diffusions. In Conference on Learning Theory, pp. 3084-3114, PMLR, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 1367):1661-1674, 2011.
Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arX7v:2404.007733 2024a.
14

---

Published as a conference paper at ICLR 2025
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024b.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-
preserving generation in seconds. arXiv preprini arXiv:2401.07519, 2024c.
Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 1900-1910, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint (xXiv:2308.0672) 2023.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guides conditional diffusion model. In Proceedings of the IEEF/CVF International Con- ference Computer Vision, pp. 23174-23184, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image the IEEF/CVF International Conference on Computer Vision,
diffusion models. In Proceedings the pp. 3836-3847, 2023.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2022.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1219-1229 2023.
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL THEORETICAL RESULTS
In this section, we restate the propositions more precisely and provide their technical proofs. First, we recall standard terminologies from optimal control literature (Fleming & Rishel, 2012). For 0 S to tt tv I 1, the cost function associated with the controller u(-) is defined by the integral:
-0) 1.a 1 e(xi, u, 1) di + (XWx); X = X0,
V(u; xo, 10) -
where f(-..) denotes a scalar valued function of the state X²" controller u(-), and instantaneous time t. The value function V (xx,,t) defined as the minimum value of V Xg. over the set of admissible controllers U, i.e.,
min /.²² u,
V (x(xx, to) -min (u; to) =min
u, dt (Xixx) Xto Xo, (5)
which satisfies a Partial Differential Equation (PDE) given below in Theorem A.1.
Theorem A.1 (HJB Equation, (Fleming & Rishel, 2012: Basar al., 2020)). If has continuous partial derivatives, then it must satisfy the following PDE, also known as Hamilton-Jacobi Bellmaa
(HJB) equation:
@v*
(x.t) min H (x. VxV* (x, t) = (x, t, + (V.V* (x, v (x, u.
Öt
wel
Also, the Hamiltonian H (x, V V (x, t). u. f), optimal controller u² (t) and the state trajectory (1) must satisfy
min H (x* V (xx, (), u, = H (x" (1), V (x" (1),),,,""(),)
well
A.1 INTERPRETING -EVVRSSE-DE AS SOLUTION OPTIMAL CONTROL
For clarity, we restate the problem setup here and describe the main ideas from 84 in more details. Problem setup: We discuss a standard approach to derive the optimal controller in a special case of our control problem (2). We substitute +- 1 to account for the time reversal in the reverse-
SDE (1). In this setup, XØ ~ N (0, and X ~ Pdata- We consider the following dynamic without the Brownian motion:
dxx vXX,", t)dt, X Xo.
(6)
where 0 S la < 1 S IN S and v Rd x Rdd x [No, - Rd denotes the drift field. The optimal controller w² be derived by solving the Mamilton-Jacobi-Bellmee Rishel, 2012; Basar et al., 2020), see Appendix for details. By certainty equivalence (when the drift and diffusion coefficients are linear time-varying (Astrom, 1971), which occurs when Pdato is Gaussian; see also discussion in Section A.3), the same u' applies to a more general case with the Brownian motion (Chen al., 2023), where
(HJB) equation (Fleming &
d.x v(X,,, u. t)t + dW,, X² = Xo-
Therefore, we analyze the reverse dynamic in the absence of the Brownian motion, and employ the same controller in more general cases with the Brownian motion.
Below, we consider a dynamical system whose drift field is chosen minimize transient trajec- tory cost and a terminal cost (weighted by 7) that enforces "closeness" to reference content Z1. Proposition A.2 provides the structure of the optimal control in the limiting setting where 7 - co. Furthermore, suppose we replace .11 with its conditional expectation (discussed in Remark A.3), the resulting dynamic, interestingly, is the standard reverse-SDE for the Orstein-Uhiee Uhlenbeck (OU) dif- fusion process. This connection between optimal control (more precisely, classic Linear Quadratic Control) and the standard reverse- SDE provides us a path to study other diffusion problems (e.g- personalization (Ruiz et al., 2023; Hertz al., 2023; Sohn et al., 2023; Wang et al., 2024a), image editing or inversion (Mokady et al., 2023; Delbracio & Milanfar, 2023; Rout et al., 2023b; 2024; 2023a)) through the lens of stochastic optimal control.
16

---

Published as a conference paper at ICLR 2025
Proposition A.2 (Linear optimal control with quadratic cost (Chen al., 2023)). Consider the control problem:
min {'1
well Jeo (x( t)l dt + ? || x1ll2.
where dx,"" u(xx t) dt,
X 20
Then, in the limit when + 0o, the optimal controller is given by u² = I-- =1-Xi + d for the stochastic case. 1-t
-1-X" which yields
dxp." = -1-Xi for the deterministic case and dXy
The optimal controller for the problem presented in Proposition A.2 be derived using estab- lished techniques from control theory (Fleming & Rishel, 2012; Basar et al., 2020; Kappen, 2008); the specific form of the above result follows from (Chen al.. 2023) (but without their momen- tum term). The key steps in this derivation include: (1) computing the Hamiltonian, (2) applying the minimum principle theorem to derive set of differential equations, and (3) taking the limit as - DC. These three steps are fundamental in deriving a closed-fore solution. The final step is critical for satisfying hard terminal constraint and is essential for the practical implementation of Algorithm/ and Algorithm 2, as detailed in $4.
For generative modeling, the controlled dynamics described Proposition A.2 cannot be directly applied. This limitation arises because the optimal control u² depends on the terminal state making non-causal or reliant on future information. Inspired by recent advancements flow-based generative models (Lipman et al., 2022; Liu et al., 2022), we make the optimal controller causal by replacing the terminal state with its conditional expectation given the current state, i.e., í.e. 21 <- E[XY X xy]. This modification results in controlled dynamic that can be simulated to produce a generative model incorporating principles from optimal control, as elaborated in Remark A.3.
Remark A.3 (Connections between diffusion-based generative modeling and stochastic optimal control). Following conditional diffusion models and optimal transport paths (Lipman et al., 2022; Liu et al., 2022), where x! tX- (1 t)c, the state variable X is equal distribution to x = (1 - -) + te, <~ N (0, Ia) after time reversal. Now, we use Tweedie's 's formula (Efron, 2011) to compute the posterior mean:
Xu
12 Vlog (X.,
(8)
1-t
1-t
Substituting the posterior mean in the controlled reverse dynamic of Proposition A.2, we arrive
dxp"" = (E - X. (1- dt ddW
X²" + t)- (1 t)2
V Vlogp(X, t) dt + aw,
We observe that the above equation is structurally the same as reverse-SDE associated with a forward Orstein- Uhlenbeck (OU) diffusion process. This relation between diffusion-basee generative models and optimal control is further explored in the Appendices below.
Indeed, diffusion models (Ho et al., 2020; Song et al., 2021b; Rombach et al., 2022; Podell et al., 2023; Pernias al., 2024) provide effective approximation to the terminal state of denoising process. This approximation has been used for variety generative modeling tasks. Also, the terminal state can be approximated using Tweedie's formula (Efron, 2011) with learned score function (Ho et al., 2020)'. By utilizing these pre-trained diffusion models, we can employ the connection to optimal control discussed above to develop practically implementable generative models that incorporates terminal objectives such style and personalization. Consequently, the subsequent sections are dedicated deriving the optimal controller assuming a known terminal state; we will approximate this in practice using Tweedie's 's formula as above.
A Ieernativyy,, when the reverse process is descrihed by prohability flow ODE, trained neural network directly predict the terminal state (Song et al., 2021a).
17

---

Published as a conference paper at ICLR 2025
A.2 INCORPORATING PERSONALIZED STYLE CONSTRAINTS THROUGH TERMINAL COST
In this section, we derive the optimal controller when we have access to the reference style features yi the terminal time (instead of the content of the image encoded through 3 )
Proposition A.4. Suppose A E R be linear style extractor that operates the terminal state X E R. Given reference style features yı, consider the control problem:
min [[] |lu(X;, O]1 di ||AXT yı 11
(9)
well 2 where dx;"" = u(x² 1) dl,
X = F0:
(10)
Then, in the limit when 7 oc, the optimal controller u² (A* "1 A (v) (x--) the following controlled dynamic:
which yields
1-t
AT (1) AX;)) xxp AAT
dl.
(1I)
Proof. We derive the closed-form solution of the optimal controller given a fixed terminal state condition. This is similar to (Chen et al., 2023), where the reverse process is accelerated using momentum (see also (Kappen, 2008; Basar et al., 2020) for further details on this approach). The distinction, however, lies in the treatment of the terminal constraint. For completeness, we provide full details of the proof below.
To derive the closed-form solution?, recall from equation (5) that f(x,, U1 t) 1 ||u and the terminal cost (x())) 2 |Ax 2 Let Pt represent V (x, in Theorem A.1. Then, the Hamiltonian of the control problem (9) given by
((x, Pt, ue t) = E(xi, ue t) + p u
2 Iu, 1 p.P u,
1
Since the minimizer of the Hamiltonian is 3² = -p,, the value function becomes
min (u, Ps, ur f) = Hu, Pa, ui,t) = - |p.11
(12)
Now, minimum leincipl theorem (Basar al., 2020) to obtain the following set of differen- tial equations:
dx;
VpHl (Xe: Pr, ui, 1) -Pri
(13)
d/
dp:
dt VxH (xe, Pt. ui,t)
(14)
X1, Fo:
(15)
PiN Vxh (XIN) tN) Y (AxeN y1)
(16)
Integrating both sides of (13), have
[' dx,= =
J' =pd
(17)
where the last equality due to (14), which states that p. is constant independent of time This implies X1 Xto p(1 to). From (16), we know for iN that
pi A (Axi y1)
= (A A (3o - p(1 - -4)) - AT y1)
= 7AT Axo 7A" (1 - to) - 7A7
(18)
With slight abuse of notation, we x, denote X and u, denote u/ (X", f) in the deterministic case.
18

---

Published as a conference paper at ICLR 2025
Rearranging (18) and solving for P1, get
P1 = 7 (T + 7 A(1 A(1 to))
(1" - -1"
1 + A A(1 (
=
(A" Axo - -A- A²y1) =
(19)
Passing (19) through the limit
+ oc, we get
lim (AT -1 (ATANo Ano AT y1)
(20)
1- 10
Therefore, the optimal control becomes u, =p AA dynamical system given by
- (AA (A"AAx Axi - A v'vv)
and the resulting
1-2
dx, (AT -1 AT (y1 Ax₂)
dt,
1
for the deterministic process and
(A -1 A (y1 Ax+) dt dW.,
dx,
t
for the stochastic process with the Brownian motion. This completes the statement of the proof.
The optimal controller depends the reference style features 1/1 the terminal time (instead of the image content 21 as Appendix A.1). The reverse dynamic can be simulated in practice by using CSD (Somepalli et al., 2024) style feature extractor and replacing y1 with the extracted style features from the expected terminal state [[X"" | as discussed in Remark A.3. This makes the controller drift causal and non-anticipating future information
Implications:
A.3 INCORPORATING STYLE THROUGH MODULATION AND A TERMINAL COST
In this section, we study a control problem where the velocity field is linear combination of the state and the control variable. This problem is interesting study because of the following reason. The reverse-SDe SDE dynamic of the standard OU process has a drift field of the form:
v (Xe,() () _x, 2V log Dogp(Xx..) l).
For a Gaussian prior Xo ~ N (0. the law of the OU process satisfies V log (X,, t) - Xx,, and the corresponding drift field becomes (X,,4) X,, Our goal is to modulate this drift field using a controller u (X7, l). The result below provides the structure of the optimal control (again in the setting where the terminal objective is known: see Appendix Al).
Proposition A.S. Suppose A E Rexd be a linear style extractor that operates on the terminal state X Rd. Let Pt denote VxV* (x,t) h in HJB equation (A.1). Given reference style features yı, consider the control problem:
min (' [lu(x;,,) t)1 dt 7 ||AX -v1l1 well
(21)
2
where dx," [X] (X('', 8)] dt, X = 3o.
(22)
Then, the optimal controller becomes u (t) - where the instantaneous state X X2 and pa satisfy the following:
X1 pi
Tye' - 3AAT (Ax1 1/1 e l+ + 7.A (Ax: -11)e'- 7AT (Axi - -y))e
Proof. The proof of Proposition A.5 is similar Proposition A.4. One key distinction is the set
of differential equations obtained using minimum principle theorem (Basar et al., 2020). We begin with the Hamiltonian:
(xx,, Pt, ue t) E(x,, ue t) + P (u, + xx)
| 1| Pe + p xt.
2
19

---

Published as a conference paper at ICLR 2025
which gives the minimizer of the Hamiltonian ui Pt and its value function becomes
V* = mina, (u, Pt. ug, t) = H (u, Pt: : = -- 1Pll? p x₁. By the minimum principle
theorem (Basar et al., 2020),
dx;
X dt
V vpH (xe: Pas u(,t) -p; + X
(23)
Pr = dp
-xHt (xt, Pr, ui,0) -Pr
(24)
dl
Xto = JDS
(25)
PNN = Vxh ( = = (AxIn -1).
(26)
This leads to coupled system of differential equations with boundary conditions as given below:
x
1
x
Pt
-1 P
Xto
Toi
pi > (Axi 91)
This can be solved numerically using ODE solvers, (Fleming & Rishel, 2012; Basar et al., 2020)
1 x and M = 0 - We seek a solution of the form q()) = qeat.
for details. Denote qi- =
Pt
(t)) is solution of the above problem, then must satisfy the following eigen value problem:
qe A = Mqe
(27)
Writing the characteristic polynomial of (27), get det (M AT) 0, which gives the eigen values À = (1, 11. Substituting these eigen values, we have
0
-1 91
2 0
91
O,
-0
0
2
92
which gives two fundamental solutions. By combining these two, we obtain the final solution
xx Pr
J e' +
1 2
where w and & can be found using the boundary conditions. Since = 3o and
yA" (Ax; - y1), we get w = xo - 3AAT (Axi - v1) e and E = 3A" (Axi - y1) Substitut-
ing the values w and ç, we arrive at
X1 P
AT (Axi 11) e+ + 3A" (AxI et
7AT (Ax1 v/)
This completes the proof of the proposition.
Summary: Though Appendices A-1-A.3, have seen the connection between optimal control and diffusion based generation with personalized terminal constraint. The general strategy has been to derive the optimal controller with known terminal state, and then replace the terminal state in the controller with its estimate using Tweedic's formula. While the controllers so far have an explicit form, in practice, the data distribution not Gaussian, and thus, we do not have a closed-form expression for the drift of the controller.
This line of analysis, however, points our method RB -oduulula discussed in $4, we incor- porate a contrastive style descriptor in our controller's terminal cost and numerically evaluate the drift the controller each reverse time step either through back propagation through the score network, or approximation based on proximal gradient updates.
B ADDITIONAL EXPERIMENTS
In this section, we provide implementation details and additional experimental evaluation which have been omitted from the main draft due limited space.
20

---

Published as a conference paper at ICLR 2025
B.1 IMPLEMENTATION DETAILS
Baselines: We demonstrate the applicability of our method RB-Modulation with StableCas- cade (Pernias et al., 2024) (released before April 2024). To best knowledge, RB-Modulation is the first framework that introduces new capabilities to StableCascade by incorporating SOC and AFA modules. Since there are no existing training- frreingg-eeeeeeeee personalization baselines designed for Sta- bleCascade, we seek alternatives built on other comparable state-of-the-ee models such as SDXL (Podell et al., 2023) and Muse (Chang et al., 2023)*
Among alternate training- free baselines, InstantStyle (Wang et al., 2024a) does not directly apply to StableCascade because requires feature injection into specific layers of an IP-Adapter, which not available for StableCascade. Similarly, StyleAligned (Hertz al., 2023) relies on DDIM inver- sion, which is currently applicable only to single-stage diffusion models. In contrast, StableCascade utilizes two-stage diffusion process, making the application of standard DDIM inversion (Song et al., 202 la) infeasible. We run the official source code for InstantStyle" and StyleAligned", In the absence of style description, ing InstantStyle (Wang et al., 2024a), we also compare with -P-Adaeeer. We include the quantitative comparison in Table 2, and only compare qualitatively with stronger baselines in Figure 2.
use "image in style" for DDIM inversion in StyleAligned. Follow-
For completeness, we also compare with raining-based baselines: StyleDrop (Sohn et al., 2023) and ZipLoRA (Shah al., 2023). Since the official codebase for StyleDrop' and ZipLoRA not publicly available, we use the third-party implementation and follow the training details in the corresponding papers. It takes minutes for training StyleDrop for 1000 steps and 20 minutes for training each LoRA for ZipLoRA. We train each LoRA with only one reference image for both content and styles make fair comparison with other methods. Similarly, we train StyleDrop with only one reference image. When a style description is not provided, we follow the original paper (Sohn et al., 2023) and use "in a [v*] style" instead.
Tunable parameters. Our method introduces only two hyper-parameters, stepsize 1, and optimiza- tion steps M in Algorithm We use DDIM sampling with v/ 0.1 and M 3 for all the experiments. Figure 5 illustrates overall pipeline of RB-Modulation.
Denoising Score Network
Xo
X--1
Fixed
Cross Attention
AFA Module
Added
Dropped
butterlly in melting golden 3d rendering style
Text Encoder
Image Encoder
Fixed Modules
Added Modules
Figure 5: Overall pipeline of RB-Modulation. AFA module replaces the cross-attention processor in the denoising UNet, disentangling the content and style of the reference image using CSD [43].
Content-style composition. The prompt-guided content-style composition task introduces a new layer of complexity beyond stylization. This task necessitates the disentanglement of the text prompt, reference style image, and reference content image through additional conditioning (Shah et al., 2023; Wang et al., 2024c; Huang et al., 2024b; Guo al., 2024). Such complexity poses significant challenges for DDIM inversion (Song et al., 202 la) and attention caching mechanisms (Hertz al., 2023) due the inherent dependencies multiple reverse paths.
'Note that StableCascade and SDXL have comparable performance in prompt alignment whereas Stable- Cascade is efficient due highly compressed semantic latent space (Pernias al., 2024).
'nttps: h//gttttta thub. com/ htt.annntttyle//sstaattttt 'tttps: httpth/...ommogggge///tlle----ee
'https: http://ttot....ooaooooooooaaaaaa
7https: / thub. http://ttw......oeeeaacaaaaaa/ap
21

---

Published as a conference paper at ICLR 2025
Our AFA module effectively addresses these challenges. It manipulates transformer layers to easily incorporate these additional conditions. The content information is integrated a manner similar to pre-trained ViT-L/14 model to extract content features
the style information. Specifically, in the SOC framework and update the latent embeddings concurrently via the AFA module, using an additional set of keys and values illustrated in Figure
Contenn- Curpoation
K
K
V
K
K
V
V.
K.
K
V,
K.
V.
K.
V.
K.
V.
Multi-Head Attention
Figure 6: Attention Feature Aggregation (AFA): Within the cross-attention layers, the keys and values from the previous layers (K,V), ) text embedding pp). reference style image (K,.V.) and reference content image (Ke.Ve) are concatenated and processed separately disentangle the information, which is followed by an averaging layer for the output. K Ve and only used for content-style composition.
Furthermore, to better preserve the identity of the foreground content, we extract the desired content using LangSAM based on the content prompt. This step is optional but offers more user control when multiple subjects are present in the reference image.
B.2 IMPLEMENTATION USING LARGE-SCALE SCALE DIFFUSION MODELS
The exact implementation our control problem (3) is given in Algorithm 1, which follows from theoretical insights. In practice, controller encounters challenge when the generative model contains billions of parameters as StableCascade (Pernias et al., 2024) due to back propagation through the score network, discussed in $4. Our strategy to overcome this practical challenge involves a proximal gradient update, given in Line 7-8 of Algorithm 2. To accelerate the sampling process, we run few steps (M = 3) of gradient descent after initializing Xo E [X XX' xt). resulting in only two hyperparameters to tune: stepsize and the number of optimization steps M. Further, since the CSD model expects clean image extract style features, we apply the previewer model available in StableCascade on the terminal state before extracting style features. After obtaining the final personalized latent using our Algorithm and Algorithm 2, we follow the decoding process as per the inference pipeline of the adopted generative model. In Table 4, we show the computational overhead of our method in comparison with competing methods.
Table 4: RB-Modulation matches the speed of training-free methods and offers 5-20X speedup over training-baai based methods like StyleDrop [11] and ZipLoRA [10]. For instance, StyleDrop and ZipLoRA require 300 seconds (s) and 1200s, respectively. for training specific components, in addition to their standard inference times of 30s and 40s. Mo--uuuulation does not use DDIM inversion additional parameters in the UNet, thus further reducing the computational overhead.
Method
Runtime (s) Training-Free
DDIM Inv.
Params in UNet
IP-Adapter [21]
21
Yes
Yes
Adapters
StyleAligned [12]
39
Yes
Yes
No
InstantStyle [13]
22
Yes
Yes
Adapters, ControlNets
StyleDrop [1]]
300+30
No
No
Adapters
ZoR [10]
1200+40
No
No
2 LoRAs, Merge layer
RB 18-Muoulationn (ours)
44
Yes
No
No
hhtpps:// thub. combbccom//iii www.//oooo//aeeeeee medel ros/1ang aatp:///angggggaatatattiiiiiii
22

---

Published as a conference paper at ICLR 2025
Reference style
StyleAligned
InstantStyle
StyleAligned
InstaniStyle
(+) infographic style"
(b) "Asola in- sng.
Figure 7: Impact of style descriptions in the prompt: (a) When style descriptions are provided, all methods yield better results. (b) Without style descriptions (e.g.. hard for users to describe text), alternative methods could struggle to capture the intended style in the reference image. Our method offers consistent stylization even without explicit style descriptions.
B.3 IMPACT OF HYPERPARAMETERS ON CONTROLLING STYLE AND CONTENT FEATURES
As detailed $4 and the ablation study $6.1, SOC helps disentangle the style and the prompt information by updating the drift field in the standard reverse-SDE. We study the impact of the two hyperparameters present in Algorithm and Algorithm that enables this disentanglement, as shown in Figure 8. We found better disentanglement when the step size イ 0.1 and the number of optimization steps M = 3. However, increasing the step size further results style image in- formation leaking into the output (top row). Additionally, adding more optimization steps increases computational overhead without yielding much performance gain (bottom row).
Reference Style
dog wearing glasses"
0.001
0.01 step
0.1
0.2
running robot"
2
3
4
Optimization steps
Figure 8: Qualitative results of different tunable hyperparameters: Improved style-prompt disentanglement are shown when increasing to our best configurations optimization step size 0.1 and optimization steps M = 3.
B.4 STYLE DESCRIPTION
PROMPTS FOR BETTER ASSIMILATION UNIQUE STYLES
In addition the quantitative analysis in $6.1, Figure 7 demonstrates that our method generates consistent stylized results with and without the style description. In contrast, the alternatives fail to accurately follow the prompt when the style description is absent. Although all results show noticeable improvement when the style description is provided, it is often challenging for users to describe styles many real-world scenarios. We believe our early results by RB-Modulation will pave the way for interesting future research along this direction.
We present additional qualitative results on stylization with (Figure 11) and without (Figure 12) style descriptions using StyleAligned dataset (Hertz et al., 2023). Our results consistently align with the reference style and the prompt, while other methods encounter several issues: (1) difficulty in following prompt guidance, (2) information leakage from the style reference image, and (3) failure
23

---

Published as a conference paper at ICLR 2025
Reference Style
"mountain"
"pillow"
"building"
"bottle"
"turtle"
Figure 9: A gallery of additional qualitative results stylization using RB-Modulation.
achieve reasonable prompt/style alignment in the absence of style descriptions. Figure presents a gallery of text-driven stylization results using 1B-Modulation.
B.5 EVALUATION CHALLENGES IN MEASURING STYLE AND CONTENT LEAKAGE
In $6, we discussed the limitations of metrics used in previous works (Sohn et al., 2023; Hertz al., 2023: Shah et al., 2023), such DINO (Caron et al., 2021) and CLIP-I score (Radford al., 2021). To quantify these limitations, we use results from our ablation study shown in Figure 3. As illustrated in Figure 10, DINO and CLIP-I scores are not well-suited for measuring style similarity in the presence of content leakage. This is because images with high semantic correlations to the reference style image consistently receive higher scores. For instance, in the top row, although the last two columns visually align more closely with the isometric illustration styles of the reference image, the DirectConcat output featuring lighthouse receives higher scores. The margin is particulariy pronounced for CLIP-I score.
similar observation can be made in the bottom row, where images containing train-related objects receive higher scores regardless of their stylistic similarity. Conversely, images with less content leakage (as seen the column) are assigned lower scores. This indicates that DINO and CLIP- I scores prioritize semantic content over stylistic fidelity, thus failing to accurately measure style similarity in scenarios where content leakage prevails.
On the other hand, our final method (last column), which combines AFA and SOC, demonstrates high scores for both prompt alignment metrics: ImageReward (Xu al., 2024) and CLIP-T (Radford et 2021). This method also shows higher user preference, evidenced in Table 1. In contrast, the DirectConcat results suffer from information leakage and poor alignment with the prompt, resulting in significantly lower even negative reward scores.
In the ablation study, our primary focus is on the disentanglement of prompts and reference styles. The conventional metrics fail to accurately reflect true performance due to information leakage. Consequently, we emphasize qualitative demonstrations and place greater importance on user study results, shown in Table similar previous approaches (Hertz et al., 2023; Sohn al., 2023).
24

---

Published as a conference paper at ICLR 2025
Reference style
StableCascade
DirectConcat
AFA only
SOC only
AFA SOC
ImwgeReward
0.99
-1.63
0.17
-1.20
0.40
CLIP-T
0.27
0.23
0.27
0.22
0.27
DINO score
0.48
0.74
0.66
0.75
0.72
CLIP-I
0.50
0.88
0.70
0.73
0.72
Iewae
1.49
0.06
1.39
1.11
1.43
CLIP-T
0.28
0.23
0.29
0.28
0.29
DINO score
0.55
0.80
0.68
0.70
0.65
CLIP-I
0.57
0.82
0.66
0.61
0.61
Figure 10: Comparison of different evaluation metries:
The StableCascade output provided
for reference because doesn't 't use the reference style image. The highest score for each metric marked bold with underscore. We compare four metrics: ImageReward and CLIP-T score for prompt alignment, DINO and CLIP-I score for style alignment. The prompt for the top row is cat" and for the bottom row is "A piano".
B.6 MORE QUALITATIVE RESULTS ON STYLIZATION AND CONTENT-STYIE COMPOSITION
We also showcase results on consistent style generation using user defined prompts Figure 13. Our results with different prompts consistently align with the styles while introducing various sce- narios following the prompts. The other methods face challenges like information leakage (e.g. hiking boots and the monocular) and monotonous scenes (e.g. InstantStyle). Note that the origi- nal Style paper (Sohn et al., 2023) has mentioned its difficulty when training with one image without description. We keep the results for completeness even though they are less satisfying. In Figure 15, we provide additional comparison with training-based and training-free personalization approaches. Figure 14 shows stylization given hand drawn reference style images: plastic crayon', pencil sketch 10, and commercial paint" In Figure 16, show qualitative results obtained by integrating the AFA and SOC modules in SDXL (Podell et al., 2023) pipeline, justifying the plug- and-play nature of RB-Modulation.
Compatibility with ControlNet. Our method readily adapts to layout guidance via Control- Net (Zhang et al., 2023), shown in Figure 17. Since ControlNet enhances the denoising network, the proposed method effectively minimizes the terminal cost associated with the expected terminal state, ensuring that SOC remains practical and effective. Furthermore, the AFA module integrates seamlessly by replacing the default attention processor in the denoising network, maintaining its functionality even when ControlNet employed.
Controllability of AFA Module. Figure 18 demonstrates the precise control provided by the AFA module. The pair (Kp, V2) computed using the given prompt (e.g., "a cat") without using text description of the reference style image, and (K,, V,) using the style attention head of the CSD feature extractor applied the reference style image. By gradually increasing the strength of the
25

---

Published as a conference paper at ICLR 2025
style image embedding, method progressively incorporates features from the reference style image, enabling fine-grasd ane-grained control over stylization.
Figure 19 demonstrates the ability of method RB-Modulation to generate novel and unseen styles by continuously interpolating between CSD style embedding of two reference style images.
In Figure 21, demonstrate more qualitative results for content-style composition Figure 22 shows the impact of content image in content-style composition. Figure 23 highlights the robustness of RB OB-Modulation in capturing content-specific features independently of color.
B.7 ADDITIONAL RELATED WORK
In this section, we discuss missing related works from the main paper. DiffusionnDisetaaggle ment (Wu et al., 2023) relies on VGG 16 for perceptual loss and ViT/B-32 for directional CLIIP loss, which prone to content leakage (Wang al., 2024a). contrast, our method injects fea- tures exclusively from the style attention head of the fine-tuned CSD-CLIP model, ensuring better sontent-style disentanglement in the AFA module. Additionally, our approach introduces an optimal controller framework to minimize terminal cost, offering richer design space and superior trollability compared to (Wu et al., 2023). Lastly, our method reduces sampling bias by optimizing the controller in Algorithm unlike (Wu et al., 2023), which can provably fail sample from the correct posterior.
In FreeDoM (Yu et al., 2023), the conditional guidance term Vx, log p( xx) is approximated by the gradient of an energy function, Ex,E(-xx:) Our Algorithm 1 differs by replacing Vx, log logp(-(x,, with controller u. optimized minimize this approximation error. Algorithm in FreeDom intro- duces a time-travel resampling strategy to mitigate poor guidance problem in their Algorithm by iteratively noising and denoising the intermediate latents. While effective, this process is computa- tionally expensive. In contrast, our approach (Algorithm 2) is grounded in optimal control, where we optimize the expected terminal state to satisfy constraints, such as aligning the style of the gener- ated image with the input. Thus, our Algorithm 2 avoids the need for gradient computation through the denoising score network, which is particularly expensive for large-scale models like SDXL or StableCascade. Additionally, entangle content and style, whereas FreeDoMM uses the standard attention processor, known to suffer from content leakage (Hertz al., 2023; Wang et al., 2024a).
propose a novel attention processor, namely AFA module to dis-
PARASOL (Tarrés et al., 2024) and Diff-NST (Ruta et al., 2023) are training- based methods, while approach is entirely training-free free. PARASOL requires supervised data via cross- cross-modall search (Section 3.1 in (Tarrés et al., 2024)) to train both the denoising U-Net and a projector network. Diff- NST (Ruta et al., 2023) trains the attention processor by targeting the "V" values within the denoising U-Net architecture. In contrast, our method uses two training-free modules: the AFA module re- places the default attention processor in the denoising U-Net to disentangle content and style, and the SOC module minimizes terminal cost to enhance stylization and content-style composition.
B.8 HUMAN EVALUATION TO DISCERN HIGHLY SUBJECTIVE NATURE OF STYLE
We conduct study with 155 participants via Amazon Mechanical Turk using 100 styles from the StyleAligned dataset (Hertz et al., 2023). The study requires no personally identifiable infor- mation of the participants. There is no risk incurred and no vuinerable population. The standard guidelines have been followed while conducting the user study.
We first provide participants with instructions familiarize them with the relevant terminologies. For each style, we randomly sample three outputs using three different prompts. Participants see two rows of model outputs in random order (3 images per row) and answer 3 questions, as illustrated in Figure 20.
1. In which row below, the images align better with the reference style image?
2. In which row below, the images align better with the reference text prompt above each image?
3. In which row below, the images overall align better with the reference style image AND the text prompt above each image AND with high quality?
26

---

Published as a conference paper at ICLR 2025
For each question, participants choose one of three options. We collect 8 responses for each question, with each question comparing our method against one of the alternatives. In total, we gathered 7,200 responses.
B.9 FAILURE CASES OF TRAINING-FREE STYLIZATION USING RB-MODULATION
In Figure 24, we illustrate stylization of different letters using single reference style image. Al- though our method captures the intended style and generates prompted letters, we notice that there is an inherent tendency to generate upper-case letters (Figure 24 (a)), even though it is prompted to generate lower-case letters. Upon further investigation, we observed that this issue stems from the underlying generative model StableCascade, as shown in Figure 24 (b). This highlights crucial limitation of our method. As training-free frre method, RB-Modulation shares a concern with other training-fre free methods (Wang et al., 2024a; Hertz et al., 2023; Jeong al., 2024) that the performance influenced by the original generative prior.
B.10 LIMITATIONS
In this paper, proposed framework and demonstrated its efficacy by incorporating style de- scriptor (Somepalli et al., 2024) in a pre-trained diffusion model (Pernias et al., 2024). The inherent limitations of the style descriptor or diffusion model might propagate into our framework. We be- lieve these limitations can be addressed by appropriate descriptor generative prior.
27

---

Published as a conference paper at ICLR 2025
Reference style
Ours
InstantStyle
StyleAligned
StypeDrop
*An airplane in watercolor painting style"
*A bowl of cornflakes in 3d rendering style*
"An elephant wooden sculpture style"
2019
A
"The letter abstract rainbow-colored flowing smoke wave design" "
"A vintage camera in retro hipster style"
milkshake in 1950s dinner style"
"A train in cafe logo style"
Figure 11: Additional qualitative results for stylization with style description: While the alternative methods face challenges like following the prompts (e.g., multiple airplanes instead of an airplane) and information leakage (e.g., the clouds the cornflake bowl and the guitar in the milkshake image), our method demonstrates strong performance on both prompt and style alignment. Style description is in blue.
28

---

Published as a conference paper at ICLR 2025
Reference style
Ours
InstantStyle
StyleAligned
StyleDrop
"A cat"
"A skyscraper"
*A leopardd
SLL
drum"
"A ladybug"
*A fireman"
winter evening by the fire"
Figure 12: Additional qualitative results for stylization without style description: StyleAligned and StyleDrop show severe performance drop after removing the style descriptions (e.g., see fireman and cat images). InstantStyle results show more information leakage (e.g.. the pink ladybug and leopard), whereas no obvious performance drop observed in our results.
29

---

Published as a conference paper at ICLR 2025
Reference style
Reference style
Ours
InstantStyle
StyleAligned
StyleDrop
Ours
InstantStylo
StyleAligned
StyleDrop
man reading book the park"
FiI 2oT
"A dog running in the park"
"A woman reading in the park"
"A soaring dragon'
Figure 13: Additional qualitative results for consistent stylization for user defined prompts: With no style description, our results demonstrate more diversity while following the styles and prompts. InstantStyle results show monotonous scenes and StyleAligned results suffer from severe information leakage. We report StyleDrop results for completeness and it known to perform worse with style description and single training image (Sohn al., 2023).
30

---

Published as a conference paper at ICLR 2025
"plastic crayon"
"pencil sketch
"comm. paint"
Reference Style "house on mountain"
"racing car"
"futuristic robot"
"tiger"
"lion"
Qualitative results for hand-drawn reference style images. The proposed method is agnostic to real or generated reference images. Given hand drawn reference style images (e.g., "paint" from commercial service provider) and desired text prompts (e.g., "a tiger*+stlle description), RB- 1B-Modulation captures the reference style in the generated content image. Please see $B.6 for the reference style image credits.
Figure 14:
"3d rendering"
Treining-bess
Treining-free
Reference Style
DreamBooth
DreamBooth-LoRA
StyleDrop
IP-Adapter
InstantStyle
Ours
Figure 15: Qualitative comparison with classical personalization methods. The proposed method significantly outperforms other training-fre free methods while remaining comparable to better than classical training-baan based personalization approaches. Prompt:"a baby penguin in 3d rendering style."
31

---

Published as a conference paper at ICLR 2025
Reference Style
guitar
piano
Reference Style
skyscraper
lighthouse
Reference Style
A dwarf
A dragon
Reference Style
A racing car
A sports bike
Figure 16: Qualitative results using SDXL (Podell et al., 2023) as base model. This verifies the plugannd-layy nature of RB-Modulation for training-free ffree personalization.
Prompt: 'A dog" Reference Style
(b) Prompt: 'A Reference Style
ControiNet Imput
Ref. Style
"line drawing
"vintage poster"
HALAND
"melting golden"
Figure 17: Qualitative results demonstrating compatibility with ControlNet (Zhang et al., 2023). Given the Canny edge map of reference content and an image of reference style, the proposed method effectively controls the pose of the generated samples while accurately capturing the desired style.
32

---

Published as a conference paper at ICLR 2025
Interpolaaionvsttliiaatiioo strength
Reference style
0.0
0.2
0.6
0.8
cabin"
"a cat"
car"
Figure 18: Qualitative results showing controllability of our method for stylization. By progressively increasing the strength of the style image embedding derived from the CSD style descriptor, our method gradually integrates features from the reference style image, providing fine-grained control over stylization.
"retro
giant ship"
"psychedelic"
"mosalc"
"a lighthouse"
"yyerrpunk"
'neon graffit
"a tiger*
"glowing"
Interp. Strength
Figure 19: Qualitative results showing interpolation of two different reference style images. The interpolation strength parameter provides additional control for blending features from multiple reference styles (e.g., "a lighthouse in mosaic art style" - "a lighthouse in cyberpunk art style"). This highlights RB-Modulation' capability to generate novel and previously unseen styles.
33

---

Published as a conference paper at ICLR 2025
Instructions:
Oivon 2 rowa machine genorated tmages, srower 3 quextions for style similority. prompt refevence. and overull ranking.
Rewkow of the definiien:
Styte aimirrttc Lusurra
masee cupture the stre reternice maks ma eareer cour scneme pourser, tuture draihwon, and prest consoar the
Pronpt relavancee: inctude new clenely mach image
actate
prunge the pacrel jecriiateee
srage
Select option
option
Quesioon
images drgn
style image??
Select an option
which row below, the images slign better with texd prompt above each image??
Bursnn Caniut GetrrorTooo
Reference Image
Aeople
Image
Adg
Question
above each image AND with high quality?
Reference Image
Reference Image
Image
lergle
Figure 20: User study interface: Three randomly sampled outputs are shown for each method given a style reference image, forming two rows of images. The users are asked to answer three questions on (1) style alignment (2) prompt alignment and (3) overall alignment and quality.
34

---

Published as a conference paper at ICLR 2025
Ref. content
Reference styles
2019
IP-Adapter
InstantStyle
Ours
Ref. content
Reference styles
IP-Adapter
InstantStyle
Ours
Figure 21: Additional qualitative results for content-style composition: Our results show better prompt and style alignment while preserving reference content without leaking contents from the reference style images (e.g. background of the first column and fruits in the last column,). Unlike compared baselines, our method is not restricted to fixed pose reference content image, illustrating sample diversity.
35

---

Published as a conference paper at ICLR 2025
ref. content
ref. style
Ours
w/o ref. content
ref. content
ref. style
Ours
w/o ref. content
Qualitative results content-style composition to illustrate the impact of content image. Excluding the content reference image (i.e., removing Ke and Ve from the AFA module) results in loss of content details, such the dog breed and car type, highlighted in the red box.
Figure 22:
Ref. content
Ref. style
Figure 23: Qualitative comparisons for content-style composition by graying the reference content image. Notably, the content (e.g., dog) is effectively transferred even after the grayscale transformation, demonstrating the robustness of our method in capturing and transferring content-specific features independently of color.
style
REMODULAT1O N
T
"b"
m
"d"
'u"
T
'8'
"L
"0"
'n'
RBMODUEATION
Prompt: "A lower-case letter ()'
Figure 24: Failure cases for stylization: The top shows the results of our method,
MB-MModulation, while the bottom row displays the results the backbone, StableCascade. Notably, the stylized images do not adhere to the prompt, "lower-case letter". This highlights the limitations imposed by the pre-trained generative priors on the capabilities training- free personalization models (top row).
36


Reviews:
[{"review_id": "30DreJhW7p", "reviewer": "Reviewer_ufZj", "summary": "This paper borrows some concepts from optimal control and applies them to the diffusion models, which are training-free. The method is SOTA for the problems attempted on image stylization and composition.", "questions": "* Sorry to quote from your limitation. However, I am curious to know how feature descriptors can help this work.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper presents theoretical underpinnings from optimal control. It uses those ideas to solve the training-free stylization problems similar to Deep Image Prior, which is somehow not cited in the paper.\n\n+ The work has several merits, especially in the problem of prompt-based image stylization in the training-free framework, though a pre-trained diffusion model is used for generation. \n\n+ The title RB-Modulation is pretty weird and does not suit the paper's excellent contribution. This must be revised to put the paper in the correct research context. \n\n+ The idea and theoretical contribution are quite good, and this work could lead to more interest in similar works.", "weaknesses": "- Title needs revision\n\n- Missing references such as Deep Image Prior need to be cited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vx4Jo2L4oR", "reviewer": "Reviewer_NtHN", "summary": "This paper introduces reference-based modulation (RB-Modulation) for training-free personalization of diffusion models. The modulation builds on concepts from stochastic optimal control to modulate the drift field of reverse diffusion dynamics, incorporating desired attributes (e.g., style or content) via a terminal cost. Besides, the author also proposes Attention Feature Aggregation (AFA) module to decouple content and style in the cross-attention layers. The qualitative and quantitative results verify its effectiveness.", "questions": "Please refer to the weaknesses part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The idea is novel. The author provides the first training-free personalization framework using stochastic optimal control. \n2. The author provides theoretical justifications connecting optimal control and reverse diffusion dynamics. \n3. The qualitative and quantitative results are very promising for not only stylization, but also content-style composition. The author also conducts user study to further verify its superiority.", "weaknesses": "1. As the title said, the method aims for personalization. Therefore, I recommend the author add some comparison with classical personalization method, not just style transfer methods.\n2. The section on ablation study in the paper is too brief. Considering that the AFA and SOC models are central to the article, I recommend that the authors include numerical comparisons and more qualitative comparisons.\n3. For the content-style composition experiment, I recommend the author to provide more complicated ref content (dog, sloth, cat are simple cases, or you can change the color of the ref to see whether the proposed method has achieved better content-style composition) to verify its effectiveness.\n4. For situations that require ControlNet to control the layout, can the method generalize well to these scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vBg6qize7t", "reviewer": "Reviewer_iCFz", "summary": "The paper presents am encoder-based (i.e. fine-tuning free) method for style personalization/ customization in text2image diffusion models.  The approach (RB-Modulation) is to consider the reverse dynamics of diffusion as a stochastic control problem with a terminal cost objective that aligns extracted style features with the exemplar style features using a pre-existing  style descriptor [Somepalli, CVPR 2024].   A novelty of the approach is the ‘AFA module’ proposed which separates style and content using cross-attention to mitigate content leakage from the reference image.  The authors claim but do not comprehensively show this allows controlled integration of style and content elements.", "questions": "Please see the above questions in the weakness section.  Overall this appears a technically sound paper but that does not fully contextualize its contribution in the literature or fully evidence its practicality.  The authors can consider addressing this in the rebuttal.\n\nPOST-REBUTTAL:  Additional works raised in this review have been discussed within Section 2 of the main paper to contextualize the contribution.  Additional experiments have been added to the supplemental material to show the visual results of continuous variation of style strength and of style interpolation.  I have raised my initial score accordingly to reflect a recommendation to accept the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The formulation of the feed-forward instance personalization problem using a ‘optimal controller’ is novel over the majority of prior works in this category that focus up adaptation.  However the practicalities of the problem seem to demand iterative reverse/feed-forward steps which negates a lot of the benefit of theoretical arguments and underlying assumptions presented.", "weaknesses": "Deeper comparisons could have been made to the NST literature which bear some relevance both in task and in approach particularly the recent diffusion based approaches.  Specifically the paper bears some similarity with prior works that partly reverse the diffusion process and then run it ‘forward’ again with style conditioning either from CLIP or from style descriptors.  For example ‘Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models’ [Wu et CVPR 2023) and PARASOL [Canet-Tarres et al., CVPR 2024] which uses ALADIN style descriptors for conditioning as does DIFF-NST [Ruta et al., ECCVW 2024].  Although the literature survey is quite broad in the areas of NST and style conditioned diffusion, there is limited focus on prior descriptor-based stylization work including these works and how the proposed approach fundamentally differs especially considering the use of cross-attention in AFA versus these works.  \n\nThe paper claims extensive experiments covering stylization and content-style composition as one of their three contributions.  It would have been helpful to see experiments exhibiting more nuanced control over style i.e. fine-grained variations of similar style, to exercise the controllability offered via the style descriptor conditioning.  Conditioning on text is sufficient to give coarse grain control over style (e.g. neon may be specified versus a descriptor for neon, as shown in the examples).  The fine-grained control offered by a continuous style descriptors in a feed-forward framework seems the main benefit but is not discussed or explored.  Similarly a matrix-like experiment showing descriptor interpolation and varying weights on the disentangled content/style would have been helpful to show the practical use/controllability of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KZNzfBJrJ9", "reviewer": "Reviewer_YTQn", "summary": "The authors propose RB-Modulation, a method that allows training free stylization and content-style composition with diffusion models. The authors reframe the personalization task as an optimal control problem, and propose an additional Attention Feature Aggregation (AFA) module that performs content/style disentanglement. The authors include extensive qualitative/quantitative results that demonstrate the effectiveness of their proposed method.", "questions": "1. Does RB-Modulation also work for recent flow based models? Does the standard reverse SDE for the OU process hold for flow based models, or will the optimal controller have to be redefined?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. AFAIK, personalization with diffusion models are inherently tricky in the sense that training based methods take a long time or training free methods lack in fidelity or fail in content/style disentanglement. The proposed RB-Modulation takes on a training free method that achieves both content/style disentaglement in a training free manner.\n\n2. Evaluations are comprehensive and the proposed method surpasses previous SOTA methods. Qualitative results are especially impressive.\n\n3. The paper is well written and easy to follow.", "weaknesses": "1. The choice of the baseline model (StableCascade) seems like a design choice. Authors might want to provide additional results on more widely adapted baseline models (e.g. SDXL), since this can show whether the inherent high fidelity results stems from the potency of StableCascade or their proposed method.\n2. Most of the qualitative results seem to be on generated reference style/content images. The proposed method seems to be agnostic to whether the reference image is generated or not. Results with non-generated reference images would be appreciated.\n3. Authors use Consistent Style Descriptor (CSD) to extract style features. Are there alternatives for this module? What was the main reasoning behind this choice?\n4. For content style composition, it seems like the content is also processed in the same manner as style. Is this optimal in terms of content identity preservation? In algorithm 1 and 2, it seems like the image latent is only updated by the loss w.r.t the style descriptor loss. Would this not result in information loss about the identity of the content?\n5. In figure 1, the resulting images do not seem to hold the content identity of the reference content image, but instead seems to heavily rely on the given text description. Using examples from figure 2, additional content-style composition results on including/excluding $K_c, V_c$ would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)