Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations
Authors: Litu Rout
Yujia Chen
Nataniel Ruiz
Constantine Caramanis
Sanjay Shakkottai
Wen-Sheng Chu
Venue: ICLR 2025 Poster (2025)
Keywords: Inverse Problems
Generative Modeling
Diffusion Models
Rectified Flows
Posterior Sampling
Optimal Control
Primary area: generative models
TL;DR: We present an efficient inversion method for rectified flow models, including Flux, that requires no additional parameter training, latent variable optimization, prompt tuning, or complex attention processors.

Abstract: Generative models transform random noise into images, while their inversion aims to reconstruct structured noise for recovery and editing.
This paper addresses two key tasks: (i) *inversion* and (ii) *editing* of real images using stochastic equivalents of rectified flow models (e.g., Flux).
While Diffusion Models (DMs) dominate the field of generative modeling for images, their inversion suffers from faithfulness and editability challenges due to nonlinear drift and diffusion.
Existing DM inversion methods require costly training of additional parameters or test-time optimization of latent variables.
Rectified Flows (RFs) offer a promising alternative to DMs, yet their inversion remains underexplored. 
We propose RF inversion using dynamic optimal control derived via a linear quadratic regulator, and prove that the resulting vector field is equivalent to a rectified stochastic differential equation. 
We further extend our framework to design a stochastic sampler for Flux.
Our method achieves state-of-the-art performance in zero-shot inversion and editing, surpassing prior works in stroke-to-image synthesis and semantic image editing, with large-scale human evaluations confirming user preference.
See our project page https://rf-inversion.github.io/ for code and demo.

Published as a conference paper at ICLR 2025
SEMANTIC IMAGE INVERSION AND EDITING USING RECTIFIED STOCHASTIC DIFFERENTIAL EQUATIONS
Litu Rout Yujia Chen'
Nataniel Ruiz!
Constantine Caramanis"
Sanjay Shakkottai
Wen-Sheng Chu
Google 2 UT Austin
Flitu.rout, constant sanrjayssaakotttaalltte.e.... edu edu [lit [liturout, yutiachen, natanielruiz, wschu) wgcoh) (ow.glgoooooooooooooooooooo com
ABSTRACT
Generative models transform random noise into images, while their inversion aims reconstruct structured noise for recovery and editing. This paper addresses two key tasks: (i) inversion and (ii) editing of real images using stochastic equivalents of rectified flow models (e.g, Flux). While Diffusion Models (DMs) dominate the field of generative modeling for images, their inversion suffers from faithfulness and editability challenges due nonlinear drift and diffusion. Existing DM in- version methods require costly training of additional parameters or test-time opti- mization of latent variables. Rectified Flows (RFs) offer a promising alternative DMs, yet their inversion remains underexplored. We propose RF inversion using dynamic optimal control derived via linear quadratic regulator, and prove that the resulting vector field is equivalent to rectified stochastic differential equation. We further extend our framework to design a stochastic sampler for Flux. Our method achieves of-te-of-thh-at performance in zero-shot inversion and editing, surpassing prior works in stroke-to-image synthesis and semantic image editing, with large-scale human evaluations confirming user preference. See project page https::///////mmm
github. io/ for code and demo.
INTRODUCTION
Vision generative models typically transform noise into images. Inverting such models, given a ref- erence image, involves finding the structured noise that regenerate the original image. Efficient inversion must satisfy two crucial properties. First, the structured noise should produce an image that is faithful to the reference image. Second, the resulting image should be easily editable using new prompts, allowing fine modifications over the image.
Diffusion Models (DMs) have become the mainstream approach for generative modeling of im- ages (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020), excelling at sampling from high-dimensional distributions (Ramesh et al., 2021; Saharia al., 2022: Ramesh al., 2022; Rombach et al.. 2022; Podell et al.. 2023; Pernias et al., 2024). The sampling process follows a Stochastic Differential Equation known as reverse SDE (Anderson, 1982; Efron, 2011: Song al., 2021b). Notably, these models can invert a given image. Recent advances in DM inversion have shown significant impact on conditional sampling, such as atroke-to-image synthesis (Meng al., 2022), image editing (Hertz et al., 2022; Mokady et al., 2023; Couairon et al., 2023; Rout et al., 2023a;b; 2024; Delbracio Milanfar, 2023) and stylization (Hertz et al., 2023; Rout et al., 2025).
Despite its widespread usage, DM inversion faces critical challenges in faithfulness and editability. First, the stochastic nature of the process requires fine discretization of the reverse SDE (Ho et al., 2020: Song et al., 2021b), which increases expensive Neural Function Evaluations (NFEs). Coarse discretization, on the other hand, leads to less faithful outputs (Meng et al., 2022), even with deter- ministic methods like DDIM (Song et al., 2021 Second, nonlinearities in the reverse trajectory introduce unwanted drift, reducing the accuracy of reconstruction (Karras et al., 2024). While ex- isting methods enhance faithfulness by optimizing latent variables (Rout et al., 2024) prompt embeddings (Mokady et al., 2023; Miyake et al., 2023), they tend to be less efficient, harder to edit,
"This work done during internship at Google.

---

Published as a conference paper at ICLR 2025
Typioal Data
Typical
"I kafremme
Typical Noise
Typical Noise
Atypical Noise
Atypical Noise
Atypical Date
Atypical Data
yrufeture
Rectified Flows
Optimal Control
Forward (ours)
Reverse (ours)
Figure Our method consists of three key steps: (i) Inversion We map a (corrupted) image to a structured noise using our controlled forward ODE (8); (ii) Conditioning We condition the reverse process on the desired text (e.g.. "A bedroom") "); and (iii) Generation We then transform the structured noise into the desired image using our controlled reverse ODE (15). (Left) Inversion with rectified flows results in an atypical noise sample (+), while optimal control produces a typical noise
sample (o). Our method interpolates between these two processes to generate a typical structured noise (Right) We follow a similar strategy when converting noise back into an image.
and rely on complex attention processors to align with a given prompt (Hertz et al., 2022; Rout et al., 2024). These added complexities make such methods less suitable for real-works deployment.
For inversion and editing, we introduce a zero-shot conditional sampling algorithm using Rectified Flows (RFs) (Liu et al., 2022; Albergo & Vanden 2024), a powerful alternative to DMs. Unlike DMs, where sampling is governed by a reverse SDE, RFs Ordinary Differential Equation known reverse ODE, offering advantages in both ef- ficient training and fast sampling. We construct controlled forward ODE, initialized from a given image, generate the initial conditions for the reverse ODE. The reverse ODE is then guided by an optimal controller, obtained through solving Linear Quadratic Regulator (LQR) problem. We prove that the resulting vector fields have stochastic interpretation with an appropriate drift and diffusion. We evaluate RF inversion on stroke-to-imape generation and image editing tasks, with additional qualitative results on cartoonization, object insertion, image generation, and content-style composition. Our method significantly improves photorealism in stroke-to-image generation, sur- passing a state-of-the-ans art (SoTA) method (Mokady et al., 2023) by 89%, while maintaining faithful- ness to the input stroke. In addition, show that RF inversion outperforms DM inversion (Meng et al., 2022) in faithfulness by 4.7% and in realism by 13.8% on LSUN- beuuoom dataset (Wang al., 2017). Figure! shows graphical illustration of our method RF-Inversion.
Vanden-Eiinden Eijnden, 2023; Lipman et al., 2022; Esser al.,
Our theoretical and practical contributions can be summarized as:
We present an efficient inversion method for RF models, including Flux, that requires no addi- tional training, latent optimization, prompt tuning, or complex attention processors.
We develop a new vector field for RF inversion, interpolating between two competing objectives: consistency with a possibly corrupted input image, and consistency with the "true" distribution of clean images ($3.3). We prove that this vector field is equivalent to rectified SDE that interpolates between the stochastic equivalents of these competing objectives ($3.4).
We demonstrate the faithfulness and editability of RF inversion across three benchmarks: (i) L.SUN-Bedroom, (ii) LSUN-Chureh, and (iii) SFHQ, on two tasks: stroke-to-image synthesis and image editing. In addition, we provide extensive qualitative results and conduct large-scale human evaluations to assess user preference metrics ($5).
Finally, we note that RI-Inversion has the interpretation of counterfaciual sampler, specifically as an instantiation noise abduction (Pearl et al., 2016); please see Appendix for discussion.
RELATED WORKS
DM Inversion. Diffusion models have become the mainstream approach for generative modeling, making DM inversion an exciting area of research (Meng et al., 2022; Couairon al., 2023; Song et al., 2021b; Hertz et al., 2023; Mokady et al., 2023; Rout et al., 2024). Among training-fres free methods, SDEdit (Meng et al., 2022) adds noise to an image and uses the noisy latent as structured noise. For semantic image editing based a given prompt, it simulates the standard reverse SDE starting from this structured noise. SDEdit requires no additional parameter training, latent variable optimization, or complex attention mechanisms. However, it is less faithful to the original image because adding noise in one step is equivalent to linear interpolation between the image and noise, while the standard reverse SDE follows nonlinear path (Liu et al., 2022; Karras al., 2022).
2

---

Published as a conference paper at ICLR 2025
An alternate method, DDIM inversion (Song et al., 2021a;b), recursively adds predicted noise each forward step and returns the final state as the structured noise (illustrated by Y process in Figure (a)). However, DDIM inversion often deviates significantly from the original image due to nonlinearities in the drift and diffusion coefficients, as well as inexact score estimates (Mokady al., 2023). To reduce this deviation, recent approaches optimize prompt embeddings (Mokady et al., 2023) or latent variables (Rout et al., 2024), but they have high time complexity. Negative prompt inversion (Miyake al., 2023) speeds up the inversion process but sacrifices faithfulness. Methods like CycleDiffusion (Wu & De la Torre, 2023) and Direction Inversion (Ju et al., 2023) use inverted latents as references during editing, but they are either computationally expensive or not applicable to rectified flow models like Flux (Black Forest Labs, 2024) or SD3 (Esser et al., 2024).
DM Editing. Efficient inversion crucial for real image editing. Once a structured noise obtained by inverting the image, new prompt is fed into the T21 generative model. Inefficient inversion often fails to preserve the original content and therefore requires complex editing algorithms. These edit- ing algorithms can be broadly classified into (i) attention control, such as cooppttt-ppoony (Hertz et al., 2022), plug-and-play (PnP) (Tumanyan et al., 2023), (ii) optimization- fusionCLIP (Kim et al., 2022). DiffuselT (Kwon & Ye, 2023), STSL (Rout et al., 2024), and (iii) latent masking to edit user- provided regions of an image (Nichol et al., 2022; Huberman- Suberman-Spiegelglee al., 2024) or automatically extracted from the generative model (Couairon al., 2023). We focus on efficient inversion, avoiding the need for complex editing algorithms.
methods like Dif-
Challenges in RF Inversion. Previous inversion editing approaches have been tailored towards diffusion models and do not directly apply to SoTA rectified flow models like Flux (Black Forest Labs, 2024). This limitation arises because the network architecture of Flux is MM-DiT (Peebles & Xie, 2023), which is fundamentally different from the traditional UNet used in DMs (Ho et al., 2020: Song et al., 2021a;b). In MM-DiT, text and image information are entangled within the archi- tecture itself, whereas in UNet, text conditioning is handled via cross-attention layers. Additionally, Flux primarily uses T5 text encoder, which lacks an aligned latent space for images, unlike CL.IP encoders. Therefore, extending these prior methods to modern T21 generative models requires thorough investigation. We take the first step by inverting and editing a given image using Flux.
RF Inversion and Editing. DMs (Ho et al., 2020; Song al., 202 la; Rombach et al., 2022) tradi- tionally outperform RFs (Lipman et al., 2022; Liu et al., 2022; Albergo & Vanden-Lijnden, 2023) in high-resolution image generation. However, recent advances have shown that RF models like Flux can surpass SoTA DMs in text-to-image (T2I) generation tasks (Esser et al., 2024). Despite this, their inversion and editing capabilities remain underexplored. In this paper, we introduce cient RF inversion method that avoids the need for training additional parameters (Hu et al., 2021; Ruiz et al., 2023), optimizing latent variables (Rout al., 2024), prompt tuning (Mokady et al., 2023), or using complex attention processors (Hertz et al., 2022). While our focus is on inversion and editing, we also show that our framework can be easily extended to generative modeling.
effi-
Filtering, Control and SDEs. There is rich literature on the connections between nonlinear filter- ing, optimal control and SDEs (Fleming Rishel, 1975; Øksendal, 2003: Tzen & Raginsky, 2019; Zhang & Chen, 2022). These connections are grounded in the Fokker-Planck equation (Øksendal, 2003), which RF methods (Lipman et al., 2022; Liu et al., 2022: Albergo & Vanden-Fin Dijnne,,,nden, 2023; Albergo et al., 2023) heavily exploit in sampling. Our study focuses on rectified flows for conditional sampling, and shows that the resulting drift field also has an optimal control interpretation.
METHOD
3.1 PRELIMINARIES
In generative modeling, the goal is to sample from a target distribution po given a finite number of samples from that distribution. Rectified flows (Lipman et al., 2022; Liu et al., 2022) represent a class of generative models that construct a source distribution qu and a time varying vector field
e (x,) to sample po using an ODE:
dx,- = Dz (X,)dt, Xo ~ go. E 10.
(1)
Starting from X, = Xg. the ODE (1) is integrated from t: 0 to yield a sample x₃ distributed according to Po (e.g., the distribution over images). A common choice of qu is standard Gaussian N (0, I) and 2n (X,) = _u,X, t;p,), where u is a neural network parameterized by s. The neural network is trained using the conditional flow matching objective as discussed below.
3

---

Published as a conference paper at ICLR 2025
Training Rectified Flows. To train neural network to as the vector field for the ODE (1), we couple samples from po with samples from do which we call pi to simplify the notation -via a linear path: Y, tt + (1 t. The resulting marginal distribution of Y, becomes:
pr. (y,) = By (yxa))))) = (dy)
(2)
Given an initial state Yo Yo and terminal state Y1 = Y1. the linear path induces an ODE: dy, = ue (Yily,) ) dt with the conditional vector field u, (Y = Y1 Yu- The marginal vector field is derived from the conditional vector field using the following relation (Lipman et al., 2022):
(1) Pr (v)))
(xdlx)) pe(yalyi)
u (v:) = By mp
Px())) PI (v1)dy1
(3)
P)().)
We can then use a neural network "yr. 4; (), parameterized by 4., to approximate the marginal vector field (y)) through the flow matching objective defined as:
(P) (o-, [l)) u(Y, t; + p)]3
(4)
For tractability, we can instead consider a different objective, called conditional flow matching:
(CF = (o-
|| [1/(())))) - tY
(5)
LCFM and LFM have the identical gradients (Lipman al., 2022, Theorem 2), and hence equiv-
alent. However, LCFM (fp) is computationally tractable, unlike au (F)) and therefore preferred during training. Finally, the required vector field in (1) computed as ot (Xe) = uu(,, 1 t; (p). In this way, rectified flows sample a data distribution by an ODE with learned vector field.
3.2 CONNECTION BETWERN RECTIFIED FLOWS AND LINEAR QUADRATIC REGULATOR
The unconditional rectified flows (RFs) (c.g., Flux) from Section $3.1 above, enable image gener- ation by simulating the vector field (-)-) initialized with sample of random noise. Subsequently. by simulating the reversed vector field -V1-e(-) starting from the image, we get back the sample of noise that started with. We formalize this statement below.
Proposition 3.1. Given an image yo and the vector field ve(-) of the generative ODE (1), suppose the structured noise y1 is obtained by simulating an ODE:
dY, = =Ye(Y))dl
Yo = Yo: I [0,
(6)
(-) (-) and Xu y1. then the ODE (1) recovers the original image, i.e. X1 yo-
Implication. Rectified flows enable exact inversion of given image when the vector field of the generative ODE (1) is precisely known. Employing ODE (6) for the structured noise and ODE (1) to transform that noise back into an image, RF inversion accurately recovers the given image.
Suppose instead that we start with a corrupted image and simulate the reversed vector field --)). Then we obtain a noise sample. There are two salient aspects of this noise sample. First, it is consistent with the original image: when processed through v,(-) it results in the same corrupted image. Second, if the image sample is "atypical" (e.g., corrupted, or, say, a stroke painting as in $5), then the sample of noise is also likely to be atypical. In other words, the noise sample is only consistent to the (possibly corrupted) image sample.
Our goal is to modify the pipeline above that when we start with corrupted image, get back a clean image (see stroke-to-inee to-image synthesis in Figure 3), but for this, we need to process by ve(-) noise sample that is closer being "typical""
Thus, as a first step, we derive an optimal controller that takes a minimum energy path to convert any image Yo (whether corrupted not) typical noise sample Y1 Pı (Fleming & Rishel, 1975). Specifically, we consider optimal control in a d-dimensional vector space IRd;
V(c) = S
X 1 - rill. dz, (Zc, t) dt. Zu yo. Y1 2
2 (zc 1)
(7)
where À is the weight assigned to the terminal cost and V(c) denotes the total cost of the control
Rd [0, 1] + dd. The minimization of V(c) over the admissible set of controls, denoted by C, is known as the Linear Quadratic Regulator (LQR) problem. The solution of the LQR problem (7)
given in Proposition 3.2, which minimizes the quadratic transport cost of the dynamical system.
4

---

Published as a conference paper at ICLR 2025
Proposition 3.2. For Zo Yo and Y1 Y1. the optimal controller of the LQR problem (7), denoted by c* (,,t) is equal to the conditional vector field un (-ly1) of the rectified linear path Y, = tti + (1 t) when Yo Yo. Le., c* (z,,t) 24 (z.ly1) (y1 z1 -1-1) t).
3.3 INVERTING RECTIFIED FLOWS WITH DYNAMIC CONTROL
So far, we have vector fields. The first, from the RFs, transforms image Y, typical for distri- bution Po to typical sample of random Gaussian noise Y1 ~ P1- As discussed above, if the image sample is atypical, then the sample of noise is also likely to be atypical. We also have second vector field resulting from the optimal control formulation that transforms any image (whether cor- rupted or not) to a noise sample that is typical-by -o-ll------gg from the distribution p1- Therefore, this sample, when passed through the rectified flow ODE (1) results in a "typical" image from the "true" distribution Po- This image is clean, i.e., typical for po. but is not related to the image Yo. Our controlled ODE, defined below, interpolates between these two differing objectives consistency with the given (possibly corrupted) image, and consistency with the distribution of images po - with tunable parameter y:
dy, = (Yi) + (uo(Yel))) (x.)) -x.))) dt,
Yo= yo.
(8)
is computed based on the insights from Proposition 3.2, and uy (Y;) = -v1-e(Y2 (Y) established in Proposition 3.1. Here, we call y [0, 1] the controller guidance. Thus, ODE (8) generalizes (6) to editing applications, while keeping its inversion accuracy comparable.
where uu (Yll)))
When = 1, the drift field of the ODE (8) becomes optimal controller of LQR problem (7), ensuring that the structured noise Y; = y1 adheres to the distribution p1. Consequently, initializing the generative ODE (1) with y1 results in samples with high likelihood under the data distribution Po-
Conversely, when y = O, the system follows the ODE (6) described in Proposition 3.1, resulting a structured noise Y1 that is not guaranteed follow the noise distribution p1- However, initializing the generative ODE (1) with this noise precisely recovers the reference image yo-
Beyond this vector field interpolation intuition, we show in the next section 33.4 that the controlled ODE (8) has an SDE interpretation. As is well known (Ho et al., 2020; Song et al., 2021a; Meng al., 2022; Song et al., 2021b), SDEs are robust to initial conditions, in proportion the variance of the additive noise. Specifically, errors propagate over time in an ODE initialized with an incorrect or corrupted sample. However, SDEs (Markov processes) under appropriate conditions converge to samples from a carefully constructed invariant distribution with reduced sensitivity to the initial con- dition, resulting in a form of robustness to initialization. As we see, the parameter 7 (the controller guidance) appears in the noise term to the SDE, thus the SDE analysis in the next section again provides intuition on the trade-off between consistency to the (corrupted) image and consistency to the terminal invariant distribution, and helps design a stochastic sampler for Flux (Appendix D.7).
Remark 3.3. We note that our analysis extends to the case where Y is time-varying, though we omit these results for simplicity of notation. This is useful in practice, especially when yo a corrupted image, because for large " the stochastic evolution (22) moves toward sample from the invariant measure N (0,I), I). This noise encodes clean images. Starting from this noise, the corresponding reverse process operates in pure diffusion mode, resulting in clean image. As the process approaches the terminal state, 7 is gradually reduced to ensure that yo is encoded through ue(+) into the final structured noise sample.
3.4 CONTROLLED RECTIFIED FLOWS AS STOCHASTIC DIFFERENTIAL EQUATIONS
An SDE (Ho et al., 2020) is known to have an equivalent ODE formulation (Song et al., 2021a) under certain regularity conditions (Anderson, 1982; Song et al., 2021b). In this section, we derive the opposite: an SDE formulation for our controlled ODE (8) from $3.3. Let W, be d-dimensional Brownian motion in a filtered probability space (Ω, F, (Ft). P))
Theorem 3.4. Fix any T E (0, 1). For any t E T), the controlled ODE (8) is explicitly given by:
1
(Y: -y1) (1 7)1
1 Vlogre(Y)) dt, Yo po-
dY,
5

---

Published as a conference paper at ICLR 2025
Its density evolution is identical the density evolution of the following SDE for Yo pu:
t T- 1 (1 dy: (-T-1(x-- -y1) (X)11)
2((1 - 7)t dW.
Vlog Vlog (ogg):(YY) dt +
1
(10)
Finally, denoting Pt, (-) the marginal pdf of Yt, the density evolution is explicitly given by:
Opr. (Y)) V.
(1 - xy.) + (1 (1-7)
- Vog Pt. (Yi)
(11)
Properties of SDE (10). Elaborating on the intuition discussed at the end of $3.3, when the con- troller guidance y = 0, then V (Y,) = V log Pt. (Y,), and SDE (10) becomes the stochastic equivalent of the standard RFs; see Lemma A.2. The resulting SDE is given by
/ V dW. Yo ~ po:
dv; = 1-t Yd + V
(12)
which improves faithfulness to the image Yo- When y = the SDE (10) solves the LQR problem (7) and drives towards the terminal state Y1 =y. This improves the generation quality, because the sample Yi is from the correct noise distribution P1 as previously discussed in $3.3. Therefore, a suitable choice of yy retains faithfulness while simultaneously applying the desired edits.
Finally, we assume T = 1- 6 for sufficiently small (such that 0 < 6 < 1) to avoid irregularities at the boundary (Øksendal, 2003). This is typically considered in practice for numerical stability (even for diffusion models). Thus, in practice, the final sample y1-8 is returned as yı.
Comparison with DMs. Analogous to the SDE (12), the stochastic noising process of DMs is typically modeled by the -hllebee (OU) process, governed by the following SDE:
dy, -Y,dt + V2WW,
(13)
The corresponding ODE formulation is given by:
dY,- = [-Y, log Pt (Yi)) dt.
(14)
Instead, our approach is based on rectified flows (1), which leads to a different ODE and conse- quently translates into different SDE. As additional result, we formalize the ODE derivation in Lemma A.1. In Lemma A.2, we show that the marginal distribution of this ODE is equal to that of an SDE with appropriate drift and diffusion terms. In Proposition A.3, we show that the stationary distribution of this new SDE (12) converges to the standard Gaussian (O,, 7) in the limit as t - 1.
The standard OU process (13) interpolates between the data distribution at time t and a standard Gaussian as t - 2o. The SDE (12), however, interpolates between the data distribution at time t = 0 and a standard Gaussian at L = In other words, effectively "accelerates" time as it progresses to achieve the terminal Gaussian distribution. This is accomplished by modifying the coefficients of drift and diffusion as in (12) to depend explicitly on time f. Thus, a sample path of (12) appears like noisy line, unlike that of the OU process (see Appendix D.3 for numerical simulations).
3.5 CONTROLLED REVERSE FLOW USING RECTIFIED ODEs AND SDES
In this section, we develop an ODE and an SDE similar to our discussions above, but for the reverse direction (i.c., from noise to images).
Reverse process using ODE. Starting from the structured noise yı obtained by integrating the controlled ODE (8), we construct another controlled ODE (15) for the reverse process (i.e., noise to image). In this process, the optimal controller uses the reference image yo for guidance:
dx,- (Xx) + (v.(X)))o) (x.))) dt,
Xu y1,
E [0, 1].
(15)
where 77 E [0, 1] is the controller guidance parameter as before that controls faithfulness and ed- itability of the given image yo- Similar to the analysis in Proposition 3.2, 21 (X,(y)))) is obtained by solving the modified LQR problem (16):
((c) /.'
1 1
lc (Z, t) dt 11 volla. dZ, c(Ze,t) Zo yı.
(16)

---

Published as a conference paper at ICLR 2025
Solving (16), get c(Z,,t) 2 Our controller steers the samples toward the given image 1-t
Yo- Thus, the controlled reverse ODE (15) effectively reduces the reconstruction enoor incurred in the standard reverse ODE (1) of RF models (e.g. Flux).
Reverse process using SDE. Finally, in Theorem 3.5, provide the stochastic equivalent of our controlled reverse ODE (15) for generation. Recall that we initialize with the terminal structured noise by running the controlled forward ODE (8), along with reference image Yo. As discussed above, we terminate the inversion process at a time T = 5 for numerical stability, resulting in vector Y1-5- Our reverse SDE thus starts initialization, and terminates at time T < 1.
corresponding time 6 with this vector y1-5
Theorem 3.5. Fix any T E (6, 1). and for any E [8. T" the density evolution of the controlled ODE (15) initialized at Xo -1-8 is identical to the density evolution of the following SDE:
dx,-
Xt + vlyo (1 - ()(1 1)
1 (Vlogpi-((X))
log gt.n (t.n (X:)
(17)
1(1 - 1)
2(1 1)(1 11 dWc.
Furthermore, denoting (e(-) as the marginal paf of Xx, its density evolution given by:
Byt.m (X.)
V. [-
1(1 1(1-t) 10-10-11--
1-1--1 "1 X. + 7
1 1- yo + (1 1) (1 -nn V) ge (Y)).
(18)
Properties of SDE (17). When the controller parameter 7) 0, we obtain stochastic sampler (22) for the pre-trained Flux, as given in Lemma A.4. This case of our SDE (17) corresponds to the stochastic variant of standard RFs (Liu et al., 2022; Lipman et al., 2022; Albergo & Vanden- Eijnden, 2023). Our key contribution lies in conditioning on X = Yo for inverting rectified flows. Importantly, our explicit construction does not require additional training or test-time optimization, enabling for the first time efficient sampler for zero-shot inversion and editing using Flux. When yo I-
7 = 1, the score term and Brownian motion vanish from the SDE (17). The resulting drift becomes
X the optimal controller for the LQR problem (16), exactly recovering the given image yo-
Remark 3.6. Similar to Remark 3.3, our analysis extends to the case when 1j is time-varying. This useful in editing, as it allows the flow to initially move toward the given image yo by choosing large n,. As the flow approaches yo on the image manifold, n is gradually reduced, ensuring that the text- suide edits are enforced through the unconditional vector field vi(-) provided by Flux.
Finally, we note that RI-Inversion has an interpretation counterfactual sampler; please see Appendix B for more discussion on this interpretation (Pearl et al., 2016).
ALGORITHM: INVERSION AND EDITING VIA CONTROLLED ODES
In this section, we define the problem setup and outline the procedure using controlled ODEs (8) and (15). We employ Algorithm for inversion and Algorithm 2 for editing: see Appendix D. Problem Setup. The user provides text "prompt" edit reference content, which could be a corrupt or a clean image. For the corrupt image guide, we use the dataset from SDEdit (Meng al., 2022), which contains color strokes to convey high-level details. In this setting, the reference guide yo typically not a realistic image un- der the data distribution po- The objective is to transform this guide into a more realistic image under po while maintaining faithfulness to the original guide.
For the clean image guide, the user provides a real image yn along with an accompanying text "prompt" to specify the desired edits. The task is to apply text-guided edits to yo while preserving its con- tent. Examples include face editing, where the text might instruct change in age or gender.
Original
inversion
Figure 2: Inverting flows by
controlled ODEs (8) and (15).
Procedure. Our algorithm has two key steps: inversion and editing. We discuss each step below.
7

---

Published as a conference paper at ICLR 2025
HIR
Input
SDEdit
DDIM Inversion
NTI
NTI+P2P
Ours
Figure 3: Stroke2Image generation. Our method generates photo-realistic images of bedroom or church given stroke paints, showing robustness to initial corruptions.
Inversion. The first step involves computing the structured noise Yı by employing our controlled ODE (8), initialized at the reference content Yo Yo. To compute the unconditional vector field, we use the preitriined Flux model u (+, 13 4p), which requires three inputs: the state Y, the time t, and the prompt embedding (pproppt). During the inversion process, we use null prompt in the Flux model, i.e., (yr) u(y,, E, $ ((""); ); :p). For the conditional vector field, we apply the analytical solution derived in Proposition 3.2. The inversion process yields latent variable that is then used to initialize our controlled ODE (15), i.e., Xo = yI. this phase, we again use the null prompt to up): Figure for the final output.
compute the vector field (xx, (xx,, 1 ((""");
Editing. The second step involves text-guided editing of the reference content yo. This process is governed by our controlled ODE (15), where the vector field is computed using the desired text prompt within Flux: U/ (x())) = -u(x,, t, 4 (prompt); (p). The controller guidance in (15) balances faithfulness and editability: higher 7 improves faithfulness but limits editability, while lower allows significant edits at the cost of reduced faithfulness. Consequently, the controller guidance 1) provides a smooth interpolation between faithfulness and editability, a crucial feature in semantic image editing. Motivated by Remark 3.3 and 3.6, we consider a time-varying controller guidance T),, such that for fixed 17 [0, 1] and [0, 1]. 77 vt S and 0 otherwise.
EXPERIMENTAL EVALUATION
We show that RF inversion outperforms DM inversion across three benchmarks: LSUN-church, LSUNNbedroom (Wang et al., 2017), and SFHQ (Beniaguev, 2022) two tasks: Stroke2Image generation and image editing. Stroke2Image generation shows the robustness of our algorithm to initial corruption. In semantic image editing, we emphasize the ability to edit clean images without additional training, optimization, or complex attention processors. We use Flux (Black Forest Labs, 2024) as base foundation model and show compatibility with SD3.5 (Esser et al., 2024).
Baselines. As this paper focuses on inverting flows, we compare with SoTA inversion approaches, such NTI (Mokady et al., 2023), DDIM Inversion (Song al., 2021 a), and SDEdit (Meng al., 2022). We use the official NTT implementation for both NTI and DDIM inversion, and Diffusers library for SDEdit. Hyper-paramene compare with NTI for both direct prompt change and with prompt-to-promy editing. All methods are training-free however, NTI (Mokady et al., 2023) solves an optimization problem at each denoising step during inversion and uses P2P (Hertz et al., 2022) attention processor during editing. We follow the evaluation protocol from SDEdit (Meng et al., 2022). More qualitative results and comparison are in Appendix $D.
for all these baselines are tuned for optimal performance. We
Hertz et al. (2022)
Smroee2lmage generation. As discussed in $4, goal is to generate photo-realistic image from a stroke paint (a corrupted image) and the text prompt "photo-realistic picture a bedroom".. this case, the high level details in the stroke painting guide the reverse process toward clean image.
In Figure 3, we compare RF inversion (ours) with DM inversions. DM inversions propagate the corruption from the stroke painting into the structured noise, which leads to outputs resembling the input stroke painting. NTI optimizes null embeddings to align the reverse process with the DDIM forward trajectory. Although adding P2P the NTT pipeline helps localized editing as in Figure 4,
8

---

Published as a conference paper at ICLR 2025
Table I: Quantitative results for Stroke2Image generation. L2 and Kernel Inception Distance (KID) capture faithfulness and realism, respectively. Optimization-iie based methods are colored gray.
User Pref. shows the percentage of users that prefer our method over each alternative in pairwise comparisons (and ties). E.g.: 62.11% (+ 8% ties) prefer ours over SDEdit-Flux for LSUN Bedroom.
LSUN Bedroom
LSUN Church
Method
L2
KID
User Pref. (%) f
L2
KID L
User Pref. (%) 1
SDEdit-SDI.5
86.72
0.029
59.67 (5.33)
90.72
0.089
65.33 (4.11)
SDEdit-SDXL
96.82
0.133
98.19
0.112
SDEdit-SD3
93.40
0.037
98.14
0.096
SDEdit-Flux
94.89
0.032
62.11 (8.00)
92.47
0.081
66.22 (5.22)
DDIM Inv-SD1.5
87.95
0.113
82.56 (1.67)
97.36
0.107
85.44 (2.78)
NTI-SD1.5
82.77
0.095
80.89 (4.33)
87.88
0.098
77.11 (4.89)
NTI+P2P-SDI.5
46.46
0.234
98.11 (1.78)
34.48
0.168
99.22 (0.56)
Ours
82.65
0.025
80.36
0.059
Original SDEdit Figure 4: Image editing for adding face accessories. Prompt: "face of a man/woman wearing glasses". The proposed method better preserves the identity while applying the desired edits.
DDIM Inversion
NTH+PP
Ours
for corrupted images, it drives the reverse process even closer to the corruption. In contrast, our controlled ODE (8) yields a structured noise that is consistent with the corrupted image and also the invariant terminal distribution, discussed in $3.3, resulting in more realistic images.
In Table I, we show that our method outperforms prior works faithfulness and realism. On the test split of LSUN bedroom dataset, approach is 4.7% more faithful and 13.79% more realistic than the best optimization free method SDEdit-SD1.5. Ours is 73% more realistic than the optimization- based method NTI, but comparable in L2. As discussed, NTI+P2P gets closer toward the corrupt image, which gives a very low L2 error, but the resulting image becomes unrealistic. Our approach 89% realistic than NTI+P2P. We observe similar gains on LSUN church dataset.
User study. We conduct user study using Amazon Mechanical Turk to evaluate the overall perfor- mance of the our method. With 3 responses for each question, we collected in total 9,000 compar- isons from 126 participants. As given in Table 1, our method outperforms all the other baselines by least 59.67 % in terms of overall satisfaction. More details are provided in Appendix SD.6.
Semantic Image Editing. Given clean image and text "prompt", the objective is to mod- ify the image according to the given text while preserving the contents of the image (identity for face images). In rectified linear paths, editing from a noisy latent becomes straightforward, fur- ther enhancing the efficiency of our approach. Compared with SoTA approaches (Figure 4), our method requires additional optimization et al., 2023)+P2((Hertz et al., 2022). Thus, it is more efficient than a current SoTA approach, and importantly, more faithful to the original image while applying the desired edits.
complex attention processors in NTI (Mokady
In Table 2, we show that our method outperforms the optimization ffree methods by at least 29% in face reconstruction, 6.6% in DINO patch-wise similarity, and 26.4% in CLIP- CLIP-Image similarity while being comparable in prompt alignment metric CLIP-T. Importantly, our approach offers 54. 1 1% gain in runtime while staying comparable to NTI+P2P.

---

Published as a conference paper at ICLR 2025
(a)
Original
"laugh"
"angry"
(b) "A woman"
"old"
"older"
"pepperoni" Figure 5: Semantic Image Editing: (a) stylized expression, (b) age, (c) gender, (d) object insert.
(c)
man > woman
(d) Original
"mushroom"
Given image and text prompt, our algorithm performs semantic image editing in the wild.
Table 2: Quantitative results for face editing SFHQ for "wearing glasses".
Method
Face Rec.
DINO f
CLIP-I CLIP-T
Runtime(s) +
SDEdit-SDD 5D1.5
0.626
0.885
0.300
0.712
8
SDEdit-Flux
0.632
0.892
0.292
0.710
24
DDIM Inv.
0.709
0.884
0.311
0.669
15
NTI
0.707
0.876
0.304
0.666
78
NTI+P2P
0.443
0.953
0.293
0.845
85
Ours
0.442
0.951
0.300
0.900
39
In Figure 5, we showcase four complex editing tasks: (a) prompt- stylization with the prompt: "face of boy in disney 3d cartoon style", where facial expressions, such as "laugh" or ""angy"" are used for editing; (b) ability to control the age of a person; (c) interpolating between two concepts: "A man" < "A woman"; (d) sequentially inserting pepperoni and mushroom to an image of pizza. We provide more examples of editing in the wild in Appendix $D.
Comparison using the same backbone: Flux. In Figure 6, we compare our method with SDEdit and DDIM inversion both adapted to Flux. NTI optimizes null embeddings to align with forward latents before applying text- guided edits, an approach well-suited for DMs that use both null and text embeddings. How- ever, this strategy cannot be applied to Flux, as does explicitly use null embeddings. Con- sequently, we only reimplement SDEdit and DDIM inversion with the Flux backbone and compare them to our method. Since all meth- ods leverage the same generative model, the im- provements clearly stem from controlled ODEs, grounded in solid theoretical foundation ($3).
Original
SDEdi (Flux)
Figure 6: Comparison using Flux backbone.
6 CONCLUSION
We present the first approach for inversion and editing with state-of-art rectified flow models such as Flux. Our method interpolates between two vector fields: (i) the unconditional RF field that transforms "clean" image to "typical" noise, and (ii) conditional vector field derived from optimal control that transforms any image (clean or not) to "typical" noise. Our new field thus navigates between these two competing objectives of consistency with the given (possibly corrupted) image, and consistency with the distribution of clean images. Theoretically, we show that this is equivalent to rectified SDE formulation, sharing this intuition of interpolation. Practically, show that our method results of-er- zero-shot performance, without the need of additional training, optimization of latent variables, prompt tuning, complex attention processors.
10

---

Published as a conference paper at ICLR 2025
BROADER IMPACT STATEMENT
Semantic image inversion and editing have both positive and negative social impacts.
On the positive side, this technology enables (i) the generation of photo-realistic images from high level descriptions, such as stroke paintings, and (ii) the modification of clean images by changing various attributes like the age, gender, or adding glasses ($5).
On the negative side, it can be misused by malicious users to manipulate photographs of individuals with inappropriate or offensive edits. Additionally, it carries the inherent risks associated with the underlying generative model.
To mitigate the negative social impacts, we enable safety features such as NSFW filters in the under- lying generative model. Furthermore, we believe watermarking images generated by this technology can reduce misuse in inversion and editing applications.
Reproducibility. The pseudocode and hyper-parameter details have been provided in the paper. Refer to our project page: https: http:///ffffisseeeeeeeeee inversiiv.snnnn github. io/ for source code and demo.
ACKNOWLEDGMENTS
This research has been supported by NSF Grant 2019844, a Google research collaboration award, and the UT Austin Machine Learning Lab. We thank Dr. Karthikeyan Shanmugam (Google Deep- Mind) for insightful discussions on the connections to causal inference and counterfactuals. We also thank the reviewers and area chair for their suggestions during the discussion phase.
REFERENCES
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint Se0.::23303.07799, 2023.
Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023.
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
Tamer Basar, Sean Meyn, and William R Perkins. Lecture notes on control system theory and design. arXiv preprint arXiv:2007.01367, 2020.
David Beniaguev. Synthetie faces high quality (sfhq) dataset, 2022. URL https:///Sthhub wtt./eeffisheeeeeeee SeteeenesSSH-ddatasseee
Black Forest Labs. Black forest labs, 2024. URL htt.:ttss/bbbackooretttbbbbbbb Ac-
cessed: September 1, 2024.
Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski. Kristian Kersting, and Apolinário Passos. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8861-8870, 2024.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In ICLR 2023 (Eleventh International Con- ference Learning Representations),, 2023..
Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. Transactions on Machine Learning Research, 2023.

---

Published as a conference paper at ICLR 2025
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- tion, (114966):66221114,, 2011.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müllrr, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Wendell H Fleming and Raymond W Rishel. Deterministic and stochastic optimal control, vol- ume 1. Springer Science Business Media, 1975.
Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Eloo and Daniel Cohen-Or. Renoise: Real image inversion through iterative noising. 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompp image editing with cross-attention control. The Eleventh International Con- ference on Learning Representations, 2022.
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. arXiv preprint arXiv:2312.02/33, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Inbar Huberman- Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22469-12478, June 2024.
Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang. Lei Zhang, and Qiang Xu. Humansd: A native skeleton dicletoo-guided diffusion model for human image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15998, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
Tero Karras, Miia Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24174-24184, 2024.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2426-2435, 2022.
Gihyun Kwon and Jong Chul Ye. Diffusion- based image translation using disentangled style and content representation. In The Eleventh International Conference on Learning Representations, 2023. URL https::/// wpppoeeeeeeeeeeeeeeeeeeeeeeeeeee net/ fo Forummideeee Naw.mmiuiaaaaaeeeeeaaammm EWXU.
Yaron Lipman, Ricky TQ Chen, Heli Ben Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2022.
Chentin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative- prompt inversion: Fast image inversion for editing with text- -uided diffusion models. arXiv preprint rrXi:23305.1687, 2023.
12

---

Published as a conference paper at ICLR 2025
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp.
2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp- 16784-16804, PMLR, 2022.
Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang. Effective real image editing with accelerated iterative diffusion inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15912-15921, 2023.
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer.
John Wiley & Sons, 2016.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2023.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview n forum? (oorumidd=gUSBd5eeG.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann. Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEF/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Litu Rout, Advait Parulekar, Constantine Caramanis, and Sanjay Shakkottai. A theoretical jus- tification for image inpainting using denoising diffusion probabilistic models. arXiv preprint arXiv:2302.01227.. 2023a.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G Dimakis, and San- jay Shakkottai. Solving inverse problems provably via posterior sampling with latent diffusion models. In seirth-eeventh Conference on Neural Information Processing Systems, 2023b. URL https: http///eeeeeeeeeeeeeeeeeeeeeeeee .nt/ forum?
? i 1wd.kkddXKXF........
Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen- Sheng Chu. Beyond first-order tweedie: Solving inverse problems using latent diffusion. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Shena Chu. RB-modulation: Training-fre free stylization using reference-based modu- lation. The Thirteenth International Conference on Learning Representations, 2025. URL
https: http::///peeeeeeeeeeemme net
13

---

Published as a conference paper at ICLR 2025
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuming text-to- eext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreumbooth: Hypemetworks for fast personaliza- tion of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6527-6536, 2024.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to tonge-tt-image diffusion models, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Pro- ceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. mwr.press vv7 wohl-dicksteing html.
Lille, France, 07-09 Jul 2015. PMLR. URL
btttps://pzcceediie
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising difftusion implicit models. Interna- lional Conference on Learning Representations, 2021a. URL https: http//ppneeeeeeeeeeeeeeee forum? sesiidttlggae Ser-sttggarCCHH..
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar. Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b. URL https:/ pettpss//operrevieee /pppnrvvieweeeeeeeeeeeeeee net/
Corum?id=Pxtl 2i..ddPPIII22RRHS....
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-ing to-e translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1921-1933 2023.
Belinda Tzen and Maxim Raginsky. Theoretical guarantees for sampling and inference in generative models with latent diffusions. In Conference on Learning Theory, pp. 3084-3114, PMLR, 2019.
Limin Wang, Sheng Guo, Weilin Huang, Yuanjun Xiong, and Yu Qiao. Knowledge guided disam- biguation for large-scale scene classification with multi-resolution enns. IEEE Transactions on Image Processing, 26(4):2055-2068 2017.
Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero- shot image editing and guidance. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7378-7387, 2023.
Qinsheng Zhang and Yongxin Chen. Path integral sampler: stochastic control approach for sampling. In International Conference on Learning Representations, 2022. URL https: //openeeveew.co wwwp/neuut//neeeeeeeee u000
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL THEORETICAL RESULTS
In this section, we present the theoretical results omitted from the main draft due to space constraints. We formalize the ODE derivation of the standard rectified flows in Lemma A.1.
Lemma A.1. Given coupling (Yo, Y1) ~ X Pt. consider the noising process Y, = =Y1 + (1- Y. Then, the rectified flow ODE formulation with the optimal vector field given by
dy,- 1
dt, Yo po-
(19)
i-t
Furthermore, denoting pe(-) as the marginal paf yt, its density evolution is given by:
(yr(Yi) Øt
v. -v.
Yo [1_
t
Vlogn((Y)) pa pa(Yi))
(20)
In Lemma A.2, we show that the marginal distribution of the rectified flow (6) is equal to that of an SDE with appropriate drift and diffusion terms.
Lemma A.2. Fix any T E (0, 1), and for any t E [o, T), the density evolution (20) of the rectified flow model (19) identical to the density evolution of the following SDE:
dy,
1
Y,d
V Vi-,dw,
21 dW,,
Yo ~ Po
(21)
In Proposition A.3, we show that the stationary distribution of the SDE (21) converges to the stan- dard Gaussian (oO, I) in the limit as t - 1.
Proposition A.3. Fix any T (0, 1), and for any t E [0, the density evolution for the rectified flow ODE (6) is same as that of the SDE (12). Furthermore, denoting Pc(-) as the marginal pdf of Y, its stationary distribution Pe Px(Y.) x exp
[Ix,11") 2t ), which converges to N (0, as t + 1.
We note that Lemma A.l and Lemma A.2 follow from the duality between the heat equation and the continuity equation (Øksendal, 2003), where it is classically known that one can interpret a diffusive term as a vector field that is affine in the score function, and vice-versa. This connection has been carefully used to study large family of stochastic interpolants (that generalize rectified flows) in (Albergo of ODE-SDE pairs. In the lemmas above, we have provided explicit coefficients that have been directly derived, instead of using the stochastic interpolant formulation. Our key contribution lies in constructing controlled ODEs (8) and (15), along with their equivalent SDEs (10) and (17) in Theorem 3.4 and Theorem 3.5, respectively. This aids faithfulness and editability as discussed in $4.
Vanden-Fijnden
2023; Albergo et al., 2023), and which can lead a family
In Lemma A.4, we derive rectified SDE that transforms noise into images by reversing the stochas- tic equivalent of rectified flows (12).
Lemma A.4. Fix any small 8 E (0, 1), and any E (ō, 11. the process X, governed by the SDE:
dx,- = [ [
2(1-t)
2(1-t) 2(1 - t) dW,
log ( dt +
Xo ~ Pp,
(22)
1
is the time-reversal of the SDE (12).
Implication. The reverse SDE (22) provides a stochastic sampler for SoTA rectified flow mod- els like Flux. Unlike diffusion- generative models that explicitly model the score function V P-)) in (22), rectified flows model a vector field, as discussed in $3.1. However, given a neural network u(y,,t; t; up) approximating the vector field u Lemma A.1 offers an explicit formula for computing the score function:
V Vlogp:(Y)) - 1 1- u uYr,, 4:p))
-
(23)
This score function is used to compute the drift and diffusion coefficients of the SDE (22), resulting in practically implementable stochastic sampler for Flux. This extends the applicability of Flux to downstream tasks where 3DE-based samplers have demonstrated practical benefits, as seen in diffusion models (Ho et al., 2020; Song al., 2021b; Rombach et al., 2022; Podell et al., 2023).
15

---

Published as a conference paper at ICLR 2025
Ref. Image
w/o controller + null-text (*""
w/ controller null-text (**")
w/ controller + "silver sculpture"
w/ controller "tiger"
Figure 7: Noise abduction and counterfactual sampling using our controlled rectified flows. The first image is the original reference image of a cat. The second image is generated by mapping the original cat image to a structured noise and then transforming this noise into image. In this case, we do not apply the controller and use null-text prompt during both inversion and generation. As expected, without the controller and text conditioning, the generated image appears unrelated to the original cat. Interestingly, when the controller is applied (and with null-text prompt) according to (15), the generated image closely reconstructs the original cat, illustrating that the structured noise preserves the salient features of the reference image, drawing parallel to noise abduction discussed in (Pearl et al., 2016). The rightmost two images illustrate counterfactual sampling, Using the text prompts "A silver cat sculpture" and "A tiger", we generate images that correspond to the 'do' action in (Pearl al., 2016). This demonstrates how our controlled rectified flow ODEs enable sampling of counterfactual images- imagining the original cat as silver sculpture or tiger.
B RF-INVERSION AS COUNTERFACTUAL SAMPLER
RF-Inversion pipeline consists of three key steps: (i) Inversion Mapping image to a structured noise with null-text in RF vector field: (ii) Conditioning - Modifying the text input of the RF vector field to desired prompt; and (iii) Generation Transforming the structured noise back to the desired image. This triplet can be interpreted as a prototype implementation of a counterfactual sampler. In particular, Pearl al. (2016) (Section 4.2.4) propose three-step process for counterfactual sam- pling in a Structural Causal Model (SCM): (i) Noise Abduction - Identifying the underlying noise sample that leads to the observed event: (ii) Action Applying a "do" intervention on the SCM to encode new constraints; and (iii) Prediction (Sampling from the new SCM) Using the abducted noise sample to generate outcomes in the updated SCM, ensuring sample-path coupling between the original and the modified SCMs.
As a concrete example, consider an image of a cat where we wish generate silver sculpture of the same cat: see Figure 7. In the framework of Pearl et al. (2016), we start with an SCM as world model. Given the observed event (the sample is the reference cat image). we first abduct to the underlying noise that led to this sample. This corresponds to the inversion step in our method, where we map the image to a structured noise using our controlled forward ODE (8). Next, we apply "do" action by modifying the text conditioning of the RF model to "silver cat sculpture". In a flow-based model, text conditioning functions as an intervention because the score network is deterministic. This ensures meaningful propagation of the desired prompt through the generation process. Finally, we generate the new image using the controlled reverse ODE (15), producing a counterfactual version of the original cat now imagined as silver sculpture. Thus, RF Inversion has an interpretation as a counterfactual sampler working with a powerful world model (here, the combination of the Flux generative model (Black Forest Labs, 2024) and the optimal control flow)'. This interpretation, we believe, has several applications beyond image editing and stylization, and is a fruitful direction for future work.
'Witoout our controller, the base rectified flow model does reconstruct the original image under null- text input. Our controller grounded on the original image, and hence enables the pipeline of (noise ab- duction, action, prediction). Other training- free approaches such as SDEdit (Meng et al., 2022) and DDIM Inversion (Song et al., 2021a) do not result in noise abduction because they do not provide an invertible and editable sample path coupling between the original image and noise, with both null-text and text-edit prompt. As result, they do have this interpretation counterfactual sampler.
16

---

Published as a conference paper at ICLR 2025
C TECHNICAL PROOFS
This section contains technical proofs of the theoretical results presented in this paper.
C.I PROOF OF PROPOSITION 3.2
The standard approach to solving LQR problem is the minimum principle theorem that can be found in control literature (Fleming & Rishel, 1975; Basar et al., 2020). We follow this approach and provide the full proof below for completeness.
Proof.
The Hamiltonian of the LQR problem (7) is given by
H (Z,, Pa, Co2 1)
|cell + pp c..
(24)
For ci Pp the Hamiltonian attains its minumum value: Hzz, Pt. ct. - |Ip:|l Using minimum principle theorem (Fleming & Rishel, 1975; Basar et al., 2020), we get
dp:
2x, H (Ze; Pe, Pc, ci () 0;
(25)
dc
dz:
Vp, H (Ze, Pr, ct, = -Pt;
(26)
dt
yo:
(27)
A 2 1 -ill
P1
Vn
A (z) - y1))
(28)
From (25), we know Pd is constant p. Using this constant in (26) and integrating from
have 2, p1 t). Substituting 21 in (27),
P X(z. p(1 y1) A(zi -y1) A(1 ()P,
which simplifies to
== (1+ + - ()) A(z. - y1) ( (1- -
(1)-
Taking the limit X + xc, we get J.yL I-t and the optimal controller of = Y--
Since
I-1
(z(zely)) y1 1 Vo. the proof follows by substituting yo
C.2 PROOF OF PROPOSITION 3.1
Proof. Initializing the generative ODE (1) with the structured noise y1, get
dX;
dt (X,),, Xo yı, e [0, 1].
(29)
Substituting u.(-) 21 (.)) in ODE (6),
dY: uYY)) = - -v -)))(()), Yo = =o. Vt E [0, dt
Replacing + t),
dY = 02 (Y))))), Vt E [0, 1].
(30)
dt
Since (29) and (30) hold Vt [0, and X, = yı, then X, Vi- that implies X; Yo Yu-
C.3 PROOF THEOREM 3.4
Proof. From Proposition 3.2, we have We (YrlV, = Y In Lemma A.1, we show that 1-
1
A logg((Y))
u. (Vi)
V,
1
17

---

Published as a conference paper at ICLR 2025
Now, the controlled ODE (8) becomes:
dy:-
Ue (Yi) + (ue(Yi/)i) - uu (m())))))))((()))) ddl, Yo ~ po.
Yi = y1
1-1 Y v (og +7
V1 11-1 Yú
(1 7)
1-1 dt
1-1
Yi t
1-t (1 7) (Xi)
+ 1 1 t Yi dt
| (1-)
(Y - (\1) 1-
dt.
Using continuity equation (Øksendal, 2003), the density evolution of the controlled ODE (8) then becomes:
Pp. (Yi) -v. v. [r--, (1
PL, (X)
(31)
ŌL
Applying Fokker-Planck equation (Øksendal, 2003) the SDE (10), have
Opt.y (Y₂) V.
[--T- (Y, -8)) - - 7) (YE) - V Vlogpe (Y)) Pt. (X,)
0t
(1 PP)
which can be rearranged to equal (31) completing the proof.
C.4 PROOF OF LEMMA A.1
Proof. Given (Yo, Y1) Po Pr, the conditional flow matching loss (5) can be reparameterized as:
LCFM(*) (1 Ni) xpo [[CY) Yo) (Y,,,1;)) = + (1-
where the optimal solution is given by the minimum mean squared estimator:
ue (y:) = E(y. xpo [Yi Yo = yr].
(32)
Since Y, = +Y1 + (1, we use Tweedie's 's formula (Efron, 2011) 1) to compute
12
E Y x 1
V log.)(y.).
(33)
Using the above relation, we obtain the following:
E YY
yıl
E [Yg (1 t)YolYi = yr]
(ye (1 t)E YolYe = yel)
t2
y: (1 t)) - yt +
v log pogp)(()))
- V lo
(34)
Combining (33) and (34) using lincarity of expectation, we get
uv.)) [Y|YY, y.] E [[ yrl t
(35)
V log - Yt
v log
(36)
1-t
t
yt 1-t The density evolution of Y, now immediately follows from the continuity equation (Øksendal, 2003) applied to (19).
V (y.).
(37)
18

---

Published as a conference paper at ICLR 2025
C.5
PROOF OF LEMMA A.2
Proof. The Fokker-Planck equation of the SDE (12) is given by
(Y.) + - 1 - Y (X()
1
t I
Vpi(K))
(38)
Rearranging (38) by multiplying and dividing p. (Yi) in the right hand side, get
(Y,
0L (
+ t v Vlogm(())) pe (Xx)
(39)
To conclude, observe that that the density evolution above is identical to (20).
C.6 PROOF PROPOSITION A.3
Proof. The optimal vector field of the rectified flow ODE (6) is given by Lemma A.1. The proof then immediately follows from the Likker-Planck equations in Lemma A.1 and Lemma A.2.
From Lemma A.2, the density evolution of the SDE (12) is given by
OY. V.
Ye [1
t - - Vlogp((X)) p((Y))
0t
The stationary (or steady state) distribution satisfies the following:
@p.(Y.) = --
1 Y [
1-t Viogpa((i) pa(Yi))
t
0t
Using the boundary conditions (Øksendal, 2003), we get
Yi + 1-t Vlogp:(Y)) = 0,
1 t
which immediately implies (YG)
C.7 PROOF THEOREM 3.5
Proof. Using Fokker-Planck equation (Øksendal, 2003), Lemma A.4 implies dge (X,) = V. [xx(
(+xx
v log qe (X,) ]
Ot
t
Therefore, the optimal vector field ex (Xx,) of the controlled ODE (15) is given by
1 X, + 1 t
(X,)
V Plog)i-((X.))
(40)
The LQR problem (16) identical the LQQ problem (7) with changes in the initial and terminal states. Similar to Proposition 3.2, we compute the closed-form solution for the conditional vector field the ODE (15) as:
vx vx(X((X)) = X1 X₁ -
(41)
Combining (40) and (41), we have
dX,-
[or (Xe) + 7/ (1) - (x())
1 1
/ X1 -xxx) xx)
1 1-t ) dt
(1 11) X,
v VlogpI-((XX)
+
11 m)(1 - t) - nt
(1 (1- 1)(1 t)
X, +
X1
v log ( dt
t(1 t)
1-t
1- 1
" Xi +
(1
- () 1)
log (logpi (xx) dr.
X,
f(1-))
t
19

---

Published as a conference paper at ICLR 2025
The resulting continuity equation (Oksendal, 2003) becomes:
(Xi)
V. [-
1 1 11-1--11-
"1
1-1 X1 + (1- 1) V log (o8))- gen(X))
11
01
1(1-6) 1)
- 1-t X1 1-1
X1 + (1- 1) - t)
v.
(V (VoogpI-(XX)) + V log (xnn(())))
ge (Xe)(ee)
1(1 -1 1)
(1 (1--)(1- t) V log (.)()))))))))))))))))())) ()n((e))
Using time-reversal property from Propsition 3.2, the above expression simplifies to
Ōqt,n((x) + V.
[(m1-)) X1
17 7) t)
t) 1-t X1 +
+
V log (oog + V logpi-tn((i))) gxn
0t
t
7)( 1) ve (X.)
1
which yields the following SDE:
dx. = 1-t Xt + 1-t t
7
X1
+
(1 1)( -t
t)
( log P1 (Xx,) + V log P1 Pn (Xx)) dt
f(1 tt)
t
2(1 - 1) 1) + t dW.
and thus, completes the proof.
C.8 PROOF OF LEMMA A.4
Proof. It suffices to show that the Fokker-Planck equations of the SDE (22) and (12) are the same after time-reversal. Let (-(-) denote the marginal pdf of X, such that go(-) = p-((). The Fokker- Planck equations of the SDE (22) becomes
(X+) Öt + V. [a(xx) (xx,+ 2(1 2
V log p1- (x) - -v. v.[((") ((=-1) v(((X))
t
which can be rearranged to give
(Xe)
V (X.)
1
t X 2(1-t) - t)
v log P1 (xx) (xx))
+
1 1
vi((X,)
Ot
t
1
Xx+ + 2(1 - 1) v Vlopii (Xi) 1 v log de (Xi)
V.
(X,)
Since Y, is the time-reversal process of X, discussed in Proposition (3.1),
(x.(X.) V. [-a((x)) (xx+- 1- t "Vlog (x)) ]
Øt
Substituting + 1-th t
0q ( V (X1-)) (
t 1-t X1-1 + 1-t
Vlog Plognn (X
Øt
which implies the density evolution of (12):
(Yi
v. (p((Y))
1
t
v log pc (Y2)
Ye +
Øt
1
1-t
This completes the proof of the statement.
20

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTS
This section substantiates contributions further by providing additional experimental details.
Baselines. We use the official NTI codebase" for the implementations of NTI (Mokady et al., 2023), P2P (Hertz et al., 2022), and DDIM (Song et al., 2021a) inversion. We use the official Diffusers implementation' for SDEdit and Flux". We modify the pipelines for SDEdit and DDIM inversion to adapt the Flux backbone. In Figure 6, for SDEdit, 0.7 (Meng et al., 2022), and for DDIM inversion, we use a midway starting point which is the same as ours
the optimally tuned strength parameter
6/28 0.78 for glasses and
3/28 0.89 for stroke2image).
For completeness, we include qualitative comparison with leading training-bann approach In- structPix2Pix (Brooks et al., 2023)* and a higher-order differential equation based LEDIT+ (Brack et al., 2024) (BD). Table 3 summarizes the requirements of the compared baselines.
Table 3: Requirements of compared baselines. Our method outperforms prior works while requiring no additional training, optimization of prompt embedding, attention manipulation scheme.
Method
Training
Optimization
Attention Manipulation
SDEdit (Meng et al., 2022)
X
X
DDIM (Song al., 2021a)
X
X
X
NTI (Mokady al., 2023)
X
NTI+P2P (Hertz et al.. 2022)
X
LEDIT++ (Brack et al., 2024)
X
X
InstructPix2Pix (Brooks al., 2023)
X
X
Ours
X
X
X
Metrics. Following SDEdit (Meng et al., 2022), we measure faithfulness using 1.2 loss between the stroke input and the output image, and assess realism using Kernel Inception Distance (KID) between real and generated images. Stroke inputs are generated from RGB images using the algo- rithm provided in SDEdit. Given the subjective nature of image editing, we conduct large-scale user study to calculate the user preference metric.
For face editing, evaluate identity preservation, prompt alignment, and overall image quality using a face recognition metric (Ruiz et al., 2024), CLIP-T scores (Radford et al., 2021), and using CLIP-I scores (Radford et al., 2021), respectively. For the face recognition score, calculate the L. distance between the face embedding of the original image and the edited image, obtained from Inception ResNet trained CASIA-Webfas dataset. Similar SDEdit (Meng et al., 2022). conduct extensive experiments on Stroke2Image generation, and showcase additional capabilities qualitatively on a wide variety of semantic image editing tasks.
Algorithm. As shown in Figure 8, given reference style or reference content denoted by we first use Algorithm obtain a structured noise Y; as discussed in $4. Then, we use Algorithm to transform the structured noise Yi back into an image based on new text prompt.
D.1
CONFIGURATIONS
In Table 4, we provide the param-parsmeters for the empirical results reported in $5. We use a fix y 0.5 in our controlled forward ODE (8) and time-ngrying guidance parameter n in our controlled reverse ODE (15), as motivated in Remark 3.3 and Remark 3.6. Thus, our algorithm introduces one additional hyper-parameter тı into the Flux pipeline. For each experiment, fixed time-varying schedule of 1/2 described by starting time (s), stopping time т, and strength (7)-
: htttta///.hhaaaoccoo///cc//aoo/s
21

---

Published as a conference paper at ICLR 2025
Algorithm 1: Controlled Forward ODE (8)
Input: Discretization steps N, reference image yo. prompt embedding network Φ, Flux model a (p), Flux noise scheduler a : [0, 1] - IR
Tunable parameter: Controller guidance 7
Output: Structured noise Y1
Initialize Yo = Yo
Fix noise sample y1
for i N do
Current time step: t,
Next time step: 4+1 N
1+1
Unconditional vector field: us, (Y,,) = u(Y,, ((m))::))
Proposition 3.1
Conditional vector field: ur, (Yr, (yi) y1-Y
Proposition 3.2
Controlled vector field: dr, (Vr.) = 11t (V:,) + 7 (Vrly)) - -1, (Ya))
&ODE (8)
Next state: Ye.. = Yr, + d1, (Yr,) (o(t++1) o(tt))
end
11 return
Algorithm 2: Controlled Reverse ODE (15)
Input:
Discretization steps N, reference text "prompt", reference image yo. prompt embedding network 4, Flux model (()))))))) Flux noise scheduler a: :0, 1] R,
structured noise Y1
Tunable parameter: Controller guidance 77
Output: Edited image X1
Initialize X, = Y1
for i 0 to N - do
Current time step: 4
Next time step: 4.+1 N
++1
Unconditional vector field: ut (X,.) -u(Xe, t;, ((prompt):p)
Proposition 3.1
Conditional vector field: (X,, (yo) = Vo-Xt,
b Proposition 3.2
Controlled vector field: 6 (X,,) Ut, (Xe,) (2), (Xe,lyo) 2e, (X,,)
>ODE (15)
Next state: X = Xt, + 8, (X,,) (a(l++1) - -o(1))
end return X1
We use the default config for Flux model: 3.5 for classifier-free guidance and 28 for the total number of inference steps.
Table 4: Hyper-parameter configuration of our method for inversion and editing tasks.
Task
Starting Time (s)
Controller Guidance (7))
Stopping Time (7)
Strength (7)
Stroke2Image
0.9
Object insert
1.0
Gender editing
8
1.0
Age editing
1.0
Adding glasses
25
0.7
Stylization
0.9
Inversion only
8
25
1
22

---

Published as a conference paper at ICLR 2025
(a)
Ref. style
girl'
"a panda"
(b)
Ref. style
"face of boy"
"a dwarrt
Ref. content
"sleeping cat"
"silver sculpture"
"origami cat"
'lion"
"tiger"
(d)
Ref. content
"smiling cartoon'
"angry cartoon"
"gin'
"old man
"young bly*glasses"
Figure 8: Rectified flows for image inversion and editing. Our approach efficiently inverts refer- ence style images in (a) and (b) without requiring text descriptions of the images and applies desired edits based on new prompts (e.g. "a girl" "a dwarf"). For reference content image (e.g. cat in (c) or a face in (d)), it performs semantic image editing (e.g. "sleeping cat") and stylization (e.g. "a photo of cat in origmai style"), without leaking unwanted content from the reference image. Input images have orange borders.
D.2 ABLATION STUDY
In this section, we conduct ablation study for our controller guidance parameter We consider two different time-varying schedules for M)s and show that our controller strength allows for smooth interpolation between unconditional and conditional generation.
In Figure we show the effect of starting time in controlling the faithfulness of inversion; starting time E [0, 1] is defined the time at which our controlled reverse ODE (15) is initialized. The initial state X, = yi-s obtained by integrating the controlled forward ODE (8) from 8.
0
2
6
8
10
12
15
20
25
27
Original
Figure 9: Effect starting time. Prompt: "A young man". The number below each figure denotes the starting time scaled by 28 (the total number of denoising steps) for better interpretation. In the absence of controller guidance (7) = 0), increasing the starting time (s) our controlled ODE (15) improves faithfulness to the original image.
23

---

Published as a conference paper at ICLR 2025
In Figure 10, study the effect of stopping time. We find that increasing controller guidance ne by increasing the stopping time т guides the reverse flow towards the original image. However, observe a phase transition around controlled reverse ODE (15) dominated by the conditional vector field (x)o)) for 2 т. Therefore, the reverse flow solves the LQR problem (16) and drives toward the terminal state (i.e., the original image).
0.14 4/28, indicating that the resulting drift in our
2
6
10
12
15
20
25
27
Original
Figure 10: Effect of controller guidance. Prompt: "A young man". For fixed starting time s 0, consider a time-varying controller guidance schedule π) 17 Vt < and otherwise. The number below each figure denotes the stopping time scaled by 28 (the total number of denoising steps) better interpretation. Increasing increases the controller guidance (71) that improves faithfulness to the original image.
In Figure 11, visualize the effect of controller guidance for another time-varying schedule. We make a similar observation as Figure 10: increasing Tt improves faithfulness. However, we notice smooth transition from the unconditional the conditional vector field, evidence from the smooth interpolation between "A young man" at the top left (n = 0) and the original image at the bottom right.
0
0.1
0.3
0.4
0.5
0.6
0.7
Original
Figure 11: Effect of controller guidance for another varying schedule. Prompt: "A young man". For a fixed starting time 0 and stopping time - = 8/28, consider a time-varying controller guidance schedule 7). 17 Vt < т and 0 otherwise. Increasing 7 increases the controller guidance (1p) that improves faithfulness to the original image.
D.3 NUMERICAL SIMULATION
In this section, we design synthetic experiments to compare reconstruction accuracy of DM and RF inversion. Given Yn Po. where the data distribution Pn = N(,,, 7) and the source distribution
24

---

Published as a conference paper at ICLR 2025
Qu == =V, I), we numerically simulate the ODEs and SDEs associated with DM and RF inversion; see our discussion in 83.
For 10, fix y 0.5 in the controlled forward ODE (8), and 0.5 in the controlled reverse ODE (15). These ODEs are simulated using the Euler discretization scheme with 100 steps. Addi- tionally, simulate the uncontrolled rectified flow ODEs (6) - (1) special case of our trolled ODEs (8) - (15) by setting = = O, and the deterministic diffusion model DDIM (Song al., a)) in the same experimental setup.
The inversion accuracy is reported in Table 5. Observe that RF inversion has less L.2 and L1 error compared DDIM inversion (14). The minimum reversing the standard rectified flows), which supports our discussion in $3.3.
is obtained by setting 7 = 1) (i.e.,
Furthermore, simulate the stochastic samplers corresponding these ODEs Table 5, high- lighted in orange. Similar to the deterministic samplers, we observe that stochastic equivalents of rectified flows more accurately recover the original sample compared diffusion models. Our con- troller in RF Inversion (10) + (17) effectively reduces the reconstruction error in the uncontrolled Inversion (12) + (22), which are special cases when 7 = Y/ = O. Thus, we demonstrate that (controlled) rectified stochastic processes are better at inverting a given sample from the target dis- tribution, outperforiing the typical OU process used in diffusion models (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021a;b).
Table 5: DM and RF inversion accuracy. Stochastic samplers are highlighted in orange.
Method
L2 Error
LI Error
DDIM Inversion (14)
6.024
19.038
DDPM Inversion (13)
6.007
15.758
RF Inversion (7 =1 -))=
0.092
0.20
RF Inversion (7 ===0)(()--()
3.564
8.795
RF Inversion (7 -0.5:)=()(0-(555)))
4.777
11.628
RF Inversion (? = =0.1 = (1)))
1.219
3.074
RF Inversion ( -0.5.1=
0.628
1.643
RF Inversion ( ! 0..5) (10)
0.269
0.694
RF Inversion (7 = f) = 1.0) (10)
0.003
0.010
In Figure 12, compare sample paths of diffusion models and recitified flows using 10 IID samples drawn from Pu- In Figure 13, we visualize paths for those samples using our controlled ODEs and SDEs with 7 7 0.5.
10
10
10
10
5
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(a) DDPM (13) Fwd.
(b) DDIM (14) Fwd.
(c) SDE (12) Fwd.
(d) RF (6) Fwd.
10
10
5
(e) DDPM (13) Rev.
(f) DDIM (14) Rev.
SDE (12) Rev.
(h) RF (6) Rev.
Sample paths of DMs and RFs. Top row corresponds to the forward process {Y²), and bottom row, reverse process {X,). In each plot, time is along the horizontal axis and the process, along the vertical axis. The sample paths of RFs are straighter than that of DMs, allowing coarse discretization and faithful reconstruction.
Figure 12:
25

---

Published as a conference paper at ICLR 2025
10
10
10
10
5
5
1.00
0.00
0.25
1.00
0.00
(a) ODE (8) Fwd.
(b) ODE (15) Rev.
(c) SDE (10) Fwd.
(d) SDE (17) Rev.
Figure 13: Sample paths of our controlled ODEs and SDEs. (a,c) The optimal controller
ur (Y, steers Y, towards the terminal state Yı pi during inversion. (b,d) Similarly, (X)
guides X₃ towards the reference image Yu ~ Po. significantly reducing the reconstruction error.
D.4 ADDITIONAL. RESULTS ON STROKE2IMAGE GENERATION
In Figure 14 and Figure 15, we show additional qualitative results on Stroke2lmage generation. Our method generates more realistic images compared to leading training- upproaches in se- mantic image editing including optimization-bused NTI (Mokady et al., 2023) and attention-base NTI+P2P (Hertz et al., 2022). Furthermore, it gives a competitive advantage over the training-bant approach InstructPix2Pix (Brooks et al., 2023).
Input
SDEd#
DDIM Inversion
NTI
NTH+2P
LEDIT+
InstruciPiv2Pix
Ours
Stroke2Image generation. Additional qualitative results on LSUN-Bedroom dataset comparing our method with SoTA training-fiee and training- based editing approaches.
Figure 14:
In Figure 16, demonstrate the robustness of approach corruption at initialization. All the methods transform the stroke input (corrupt image) to structured noise, which is again transformed back to a similar looking stroke input, highlighting the faithfulness of these methods. However, unlike our approach, the resulting images in other methods are not editable given a new prompt.
D.5 ADDITIONAL. RESULTS ON SEMANTIC IMAGE EDITING
Figure 17 illustrates a smooth interpolation between "A man" - "A woman" (top row) and "A woman" + "A man" (bottom row). The facial expression and the hair style are gradually morphed from one person to the other.
In Figure 18, show the ability to regulate the extent of age editing. Given image of young woman and the prompt "An old woman". we gradually reduce the controller strength nh to make the person look older. Similarly,
reduce the strength to make old man look younger.
Figure 19 shows the insertion of multiple objects by text prompts, such as "pepperoni", "mush- room". and "black leaves" to image of pizza. Interestingly, pepperoni is not deleted while
26

---

Published as a conference paper at ICLR 2025
NTI-2PP
InstruciPu2Pix
Figure 15: Stroke2Image generation. Additional qualitative results on LSUN-Church dataset com- paring our method with SoTA training-free and training-basee based editing approaches.
Stroke Input
Inversion
Ediling
SDEdit
DDIM
NTI
NTH+P2P
Ours
Figure 16: Robustness. For inversion, all methods perform well recovering the stroke input when given a null prompt. However, when new prompt like "a photo-realistic picture of a bedroom" is provided, only our method successfully generates realistic images. The other methods continue to suffer from the initial corruption, failing to make the output more realistic.
inserting mushroom, and mushroom is not deleted while inserting black leaves. The product is finally presented in lego style.
Figure 20 captures a variety of facial expressions that stylize reference image. Given the original image and text prompt: e.g. "Face of girl in disney 3d cartoon style", we first invert the image to generate the stylized version of the original image. Then, we add the prompt for the expression (e.g., "surprised") "Face of girl in disney 3d cartoon style, surprised". By changing the expression, we are able to preserve the identity of the stylized girl and generate prsmdt-based facial expressions.
the end of the prompt and run editing algorithm (15) with this new prompt:
27

---

Published as a conference paper at ICLR 2025
Figure 17: Gender editing, Our method smoothly interpolates between "A man" + "A woman".
"A woman"
"old"
"older"
"A man"
"young"
"younger"
Figure 18: Age editing, Our method regulates the extent of age editing.
Original
"pepperoni"
"mushroom"
"grenn leaves"
"in lego style"
Figure 19: Object insert. Text-guided insertion multiple objects sequentially.
D.6 HUMAN EVALUATION
We conduct user study on the test splits of both LSUN Bedroom and LSUN Church dataset using Amazon Mechanical Turk, with 126 participants in total. As shown in Figure 21, each question was accompanied by an explanation of the task, the question, and the evaluation criteria. Participants were shown pair of stroke-to-image outputs from different models, in random order, along with the input stroke image. They were asked to select one of three options based on their preference using the following two criteria:
1. Realism: which of these two images look more like real, photorealistic image?
2. Faithfulness: which of these two images match more closely to the input stroke image?
We collect 3 responses question. With 300 images in the test dataset and 10 pairwise compar- isons, we gathered 9,000 responses for this evaluation. The example Figure for the LSUN Church dataset; for LSUN Bedroom dataset, we simply replace the word "church" to "bedroom" in the instructions.
D.7 GENERATIVE MODELING USING RECTIFIED STOCHASTIC DIFFERENTIAL EQUATIONS
In Figure 22, we compare images generated by the ODE (1) and SDE (22) variant of Flux across different discretization steps. Figure 23 illustrates text-to-image generation using the stochastic
28

---

Published as a conference paper at ICLR 2025
*Face of young girl disney 3d cartoon style"
Original
"surprised"
"scared"
"frowning"
"yawning"
"laughing"
"grinning"
"sad"
"smiling"
"winking"
"gasping"
"angry"
"Face boy disney cartoon style"
Original
"'augh"
"smirking"
"frown"
"angry"
"yelling"
Stylization of reference image given prompt-based faciill expressions. Given the reference content (e.g., image of woman) and the desired prompt (c.g., "a young girl in disney 3d cartoon style"), our method makes the person younger and follows the disney 3d cartoon style. This demonstrates the ability of our method to generate various stylized expressions.
Figure 20:
Instructions:
This lewar leaei' you way priqus seedii Thr deli wellede
Seaber: Peeterts
Meawon
Select an option
Brewnadieggr Graeraddage
Figure 21: Interface for human evaluation. Each participant asked select their preferred image based on two criteria: realism and faithfulness.
sampler for Flux?, highlighting the practical significance of our theoretical findings 53 and Ap- pendix A.
D.8 DISCUSSION ON ACCELERATING CONTROLLED RECTIFIED FLOWS
"https://g/ hww.b.oo/bblcc.....c forest- labs/ flux
29

---

Published as a conference paper at ICLR 2025
FluxODE
FluxSDE
100
300
500
700
1000
Prompt: "This dreamlike digital art captures vibrant, kaleidoscopic bird in lush rainforest".
FluxODE
ICR
FluxSDE
50
100
300
500
700
1000
Prompt: futuristic depicted entirely out fractals".
Figure 22: T21 generation using rectified ODE (top) and SDE (bottom) for different number of discretization steps marked along the X-axis. The stochastic equivalent sampler FluxSDE generates samples visually comparable to FluxODE at different levels of discretization.
In this paper, we used 28 inference steps, the default setting for Flux. It is well established that reconstruction error increases as the number of inference steps decreases due coarse discretiza- tion. Prior works on diffusion models have developed methods to mitigate this error with fewer steps (Garibi et al., 2024; Pan al., 2023). In particular, the accelerated iteration procedure from AIDI (Pan et al., 2023) and the renoising iteration step in ReNoise (Garibi et al., 2024) could en- hance the inversion and editing capabilities of our method with fewer steps. Integration of such sampling techniques in rectified flows represents promising direction for future work.
D.9 RF-INVERSION A PLUG-AND-PLAY SOLUTION FOR RECTIFIED FLOWS
Figure 24 presents stylization results achieved using RF-Inversion with Stable Diffusion 3.5 (Esser al.. 2024). The generated images accurately reflect the artistic styles the reference style im- ages while faithfully adhering to the desired prompts. The top row includes reference styles from StyleAligned (Hertz et al., 2023) benchmark. The bottom row contains hand-drawn styles: plastic crayon and pencil sketch".
D.10 RF-INVERSION WITH DIFFERENT INITIALIZATION
30

---

Published as a conference paper at ICLR 2025
Flux
FluxSDE (Ours)
Flux
FluxSDE (Ours)
Prompt: "portrait, looking to one side of frame, lucid dream-like 3d model of an owl, video game character. forest, wonderland, photorealism, cinematic artistic style."
Prompt: "a robot with reflective helmet, iron armor. photorealistic, in shades of red and golden brown, dark gloomy environment, epic scene."
Prompt: 'a space cinematic scifi art, spacecrafts flying the background"
Prompt: 3d model a magical creature, long nose, green color, movie asset, ultra detailed."
Lix ll
Frmm for o hample qarc
Veppast
Lantros
Pest Colcali
Prssl dlanes
Prompt: "a dragon soaring through the sky, battle ground, people fighting on the ground."
Prompt: "a genius scientist, in his 60s stands, writing on the black board, white hair. white beard, round spectacles."
T2I generation using rectified flow SDE (22). Our stochastic sampler is visually com- parable to the standard deterministic sampler provided by Flux.
Figure 23:
Figure 25 shows the effect of yı on reference image-based stylization and structure preservation. The proposed method reliably generates accurately stylized image and preserves the structure of the input image. While the choice of yı does have an impact, our controller ensures that the semantics of the reference input are still preserved for different random choices of y
Regarding stroke-to-ingg tomiiage quality, the manually annotated clean stroke images used in SDEdit are not publicly available. Therefore, we simulate stroke images following Section D.2 (Figures 30 and 31) in SDEdit, as this strategy was also used in their large-scale evaluation. The simulated stroke images noisy which increases variability in the layout of the generated images. We also added one example in Figure 25 (d) using an annotated bedroom stroke input (by taking a screenshot of stroke paint from Figure of SDEdit), showing that the layout of the generated images are better aligned with a cleaner stroke input.
D.11 COMPATIBILITY OF RF-INVERSION WITH FLUX-LORA
Figure 26 illustrates content-style composition using Flux-LoRA (Hu et al., 2021). In this experi-
ment, we apply LoRA fine-tuning only for content while using our method for stylization. We train two LoRA models with the sks token using images from the DreamBooth (Ruiz et al., 2023)
31

---

Published as a conference paper at ICLR 2025
dataset : (1) "a sks dog" (6 images) and (2) "a sks cat" (5 images). For each LoRA,
rank 16 and fine-tune for 1000 iterations, which takes around 25 minutes. We first use (y (y:) in Algorithm to obtain structured noise corresponding to the reference style image (e.g., "line drawing"), Subse- quently, we employ Algorithm with LoRA weights added to vy using the desired prompt (e.g., "a sks dog in line drawing style"). Observe that the unique identifiers of the dog (e.g., the beard) are effectively captured in the generated sample.
(a) Ref. style
boat"
car"
(b) Ref. Style
"A mad scientist"
"A lion boy"
(c) Ref. style
"A house hill"
"A racing car"
Ref. Style
elephant"
"A fuluristic robol"
Figure 24: Compatibility of our method with another RF model, Stable Diffusion 3.5. Ref- erence styles in (a) "melting golden 3d rendering" and (b) "vintage travel poster" from StyleAligned benchmark (Hertz et al., 2023), and (c) "plastic crayon drawing art" and (d) "pen- cil sketch" hand-drawn styles provided by users. Please see $D.10 for the reference style credits.
D.12 ADAPTABILITY OF RF-INVERSION TO GENERAL INVERSE PROBLEMS
Figure 27 shows that our method Inverrssoo easily extends to a broad class of inverse prob- lems without using additional training, latent variable optimization, or complex attention proces- sors. For restoration task, we consider super-resolution by 8X (left) and motion blur by kernel (61 x (right).
1.13 3 STUDY OF CONTROLLER GUIDANCE IMAGE EDITING
Figure 28 demonstrates the influence of the controller guidance parameters (forward flow) and (reverse flow) on the image transformation process. The interplay between y and rj crucial for balancing the forward and reverse flows, ensuring that the resulting output follows the structure of the given input. Increasing y improves realism but the structure changes. For instance, the church door is facing towards right. For a fixed (say 0.5), increasing и aligns the structure with the reference input (yn). Interestingly, at a midway point (y = 0.5, 17 = 0.5), the image looks blurry because this is superposition of two conflicting images (e.g., in one image, the door is facing to the right, and in another, to the front). Finally, our controller guidance (n) rectifies this process to become more faithful to the reference input.
D.14 ADDITIONAL. EDITING RESULTS USING 8 STEP DISTILLED MODEL
Figure 29 shows additional qualitative results on face editing and stroke2image generation using method integrated with Flux- Mux-TTrbb--LRAA These results demonstrate the compatibility of our method with a distilled base model capable of sampling in as few as steps. In this experiment, we employ the distilled Flux- Torn-Turbo-LRR, model for computing u--)) and v.(.) in our Algorithm 1 and Algorithm 2, respectively.
10 Phttps://gtt http::///gghhhbbbcooooooooe//o/o goo. "https: hhttpss//hugiinggfaccee. all mama creat ww/pw/w...//.////// 1 Turbo-
1oogloooooo wtt.g/oggoooooooooeeooooooooo dreambeoth
32

---

Published as a conference paper at ICLR 2025
Input
yi
yf
yi
(a) Reference content
*face wearing glasses"
(b) Simulated church
"a church"
Simulated bedroom
bedroom
(d) Annotated bedroom
"a bedroom"
(e) Reference style
cute panda"
Figure 25: Robustness to initialization (y1). Given an input image (e.g., style image of "pen- cil sketch" in (e)) as Yo and different typical samples {yî 15 from pi, our method effectively captures the semantics of the reference input while adhering to the desired prompt (e.g.. "a cute panda"). The annotated bedroom input in (d) is screenshot of the bedroom from SDEdit (Meng al., 2022). Compared to simulated stroke inputs (b.c), the annotated stroke input (d) helps with better alignment of the generated layout.
D.15 COMPARISON WITH DDIM INVERSION FOR DIFFERENT START STEP
Figure 30 shows the effect of different starting time for DDIM Inversion. The generated samples become less realistic but more faithful as the starting time increases.
D.16 LIMITATION
The lack of comparison with expensive diffusion-baan based editing solutions may be viewed limita- tion. However, these implementations are either not available for Flux not directly applicable due to Flux's i distinct multi-modal architecture. The key contribution this paper lies theoretical foundations, validated using standard benchmarks and relevant baselines.
33

---

Published as a conference paper at ICLR 2025
"line drawing"
"cyberpunk"
"minimal origami"
"vintage poster"
"sticker"
Ref. Style
Ref. Content
RISLAND
'a sks dog"
"a sks cat
Figure 26: Compatibility of our method with LoRA for content-style composition. The gener-
ated image preserves the identity of the reference content while adhering to the desired style.
(a)
Input
"jumping dog"
Input
"cat yawning"
Non-rigid
(b)
Input
"a rabbit"
Input
"a turtle"
Restoration
(c)'blue car, flowers" "blue car, yellow flowers"
Input
"a red car"
Colar change
Figure 27: Image editing (a) non-rigid task, (b) image restoration and (c) local color change. The proposed method generalizes to wide-variety of inverse problems without training, test-time optimization or atos-attention manipulation.
34

---

Published as a conference paper at ICLR 2025
Input (Vo)
0.1
0.3
0.5
0.7
0.9
y
0.3
0.5
0.7
0.9
Figure 28: Impact of the forward and reverse controller guidance in image editing. Increasing helps transform an "atypical" sample to "typical" sample, making the generated samples more realistic. Subsequently, increasing the controller guidance for reverse flows (7) improves faithfulness to the reference input, as discussed in $3.
35

---

Published as a conference paper at ICLR 2025
young girl disney cartoon"
young girl anime style"
"a girl wearing glasses"
Input
"an old woman"
Input
Input
"A church
Input
"A church
Input
"A bedroom'
Figure 29: Face editing and Stroke2lmage generation in 8 steps. We noise and denoise using distilled model that is capable of sampling from the data distribution steps.
Input
DDIM Inv (0/50)
DDIM Inv (10/50)
DDIM Inv (20/50)
DDIM Inv (30/50)
Ours
Figure 30: Qualitative comparison with DDIM Inversion with different starting time. Our
method outperforms DDIM Inversion with a mid-way starting point.
36


Reviews:
[{"review_id": "5rLeD0Ff9s", "reviewer": "Reviewer_XYr8", "summary": "This paper propose RF inversion using dynamic optimal control derived via a linear quadratic regulator. We prove that the resulting vector field is equivalent to a rectified stochastic differential equation.", "questions": "1. *In Equation (8), since the value of \\( y_1 \\) is unknown, how is the conditional term in Equation (8) calculated when \\( y_1 \\) is needed?*\n\n2. *Similarly, in Equation (15), the calculation relies on \\( y_0 \\). This implies that the target \\( y_0 \\) is fully known, which raises fairness concerns. Reconstructing a value that is already available seems unfair and could be viewed as “cheating,” making a 100% reconstruction accuracy almost inevitable. Many training-free methods do not require access to \\( y_0 \\) during sampling. The authors should explain why they have chosen to compare reconstructions in this potentially biased manner.*", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper propose RF inversion using dynamic optimal control derived via a linear quadratic regulator. We prove that the resulting vector field is equivalent to a rectified stochastic differential equation.", "weaknesses": "1. The paper lacks a clear pipeline diagram, algorithm, or pseudocode, making it challenging to understand the methodology. The reviewer suggests adding an algorithm or pipeline diagram to improve clarity and readability.\n\n2. The paper does not include a comprehensive comparison on established benchmarks for text-guided image editing. The sole focus on face editing in SFHQ for the “wearing glasses” task is insufficient to demonstrate the method's effectiveness. The authors are encouraged to refer to recent surveys or the latest papers in the field and to identify one or two well-recognized benchmarks for text-controlled image editing for a more complete evaluation. It would be beneficial to include comparisons with state-of-the-art image editing methods from 2024, rather than relying on older methods.\n\n3. The proposed method involves a large number of hyperparameters, which is a potential drawback. The authors should perform a complete sensitivity analysis on all hyperparameters, as most training-free methods are known to be highly sensitive to hyperparameter choices, with different settings leading to vastly different editing results. The reviewer suggests specifying the hyperparameter values used for each case to provide better transparency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lzrXLFzGXJ", "reviewer": "Reviewer_Mj6G", "summary": "This paper tackles the problem of image inversion and editing with Rectified Flow models such as Flux. In particular, the authors are interested in the case where the input image is corrupted, and where faithful inversion and reconstruction is not necessarily ideal. To this end, the authors introduce an optimal control-inspired guidance scale, which interpolates between the unconditional flow trajectory and one given by an optimal controller. This effectively controls the tradeoff between faithfulness and editability during inversion. Additionally, the paper derives the stochastic differential equation associated with the ODE of flow matching, providing additional insights. They demonstrate the effectiveness of their method on two tasks: stroke-to-image synthesis and image editing.", "questions": "Regarding the inversion process (Section 3.2 & 3.3):\n\n- I was confused by the implication of Proposition 3.1. Is it correct to understand that, out of the box, RF models are better than DM models for inversion? i.e., one can get almost perfect reconstruction in RF models by simply solving the ODE backward and then forward? This seems to be the case according to Table 5. If this is really the case, the problem to solve is actually more about editability than faithfulness. I would suggest the authors make it more explicit in the main paper by contrasting/comparing with DM models, where DDIM inversion is known to be sensitive to discretization errors.\n- Proposition 3.2 states that the solution to the LQR problem is simply a linear path between the input image $\\mathbf{y}_0$ and a randomly sampled (fixed) noise $\\mathbf{y}_1$. This seems exactly like what SDEdit proposes. I was wondering if the optimal control perspective is different in any way? What additional insight or value does introducing the LQR problem bring compared to the straightforward linear interpolation explanation of SDEdit?\n- The main result of the first part is shown in Eq. 8, where the authors introduce the *controller guidance* $\\gamma$. Conceptually, is Eq. 8 essentially a linear interpolation between SDEdit (linear path solution to the LQR problem) and DDIM inversion (following the ODE backwards)? If so, wouldn’t it be more efficient to solve the ODE backward (normal inversion) and interpolate the result with a Gaussian noise at time $t=1$?\n\nRegarding the SDE formulation (Section 3.4):\n\n- Eq. 10 is the SDE formulation of Eq. 8. However, the stochastic part only relates to the unconditional ODE (from RF model), not the linear path conditioned on $\\mathbf{y}_1$. Have the authors considered also making the linear path ODE stochastic? Intuitively, I assume that it would remove the need to fix the noise sample, as the SDE would converge to the entire Gaussian distribution.\n- Around l.235, the authors mention that SDEs are more robust to initial conditions, which motivates the SDE formulation in Section 3.4. However, looking at Algorithms 1 & 2 in the appendix C, it seems that this SDE formulation is never used in practice (they only appear in C.3 for the numerical simulations). Why?\n\nRegarding the Algorithm & Results (Section 4 & 5):\n\n- What values of $\\gamma$ and $\\eta$ are used in Figure 2.?\n- In Fig. 5, what models are used for SDEdit, DDIM inversion, NTI?\n- The authors point out in the appendix that they use a fix value $\\gamma=0.5$ through out the experiments. I wonder how does $\\gamma$ affect the final result for a given $\\eta$? It would be also interesting to have a two-axis grid comparison with varying $\\gamma$ and $\\eta$ to better understand the interplay between these two guidance parameters.\n\nMiscellaneous:\n\n- The caption in Fig. 10 says “The number below each figure denotes the starting time scaled by 28”, which is not true, since the number is the controller guidance value in this Figure.\n- Typo at l.192: “we need to process**s**”.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem that the authors are solving is significant, as the literature is progressively moving towards RF models.\n- The proposed solution is sound and the results show the effectiveness of the method. In particular, the key contribution of conditioning the reverse process on the input image $\\mathbf{y}_0$ is new and interesting.\n- The derivation of the SDE associated with the RF ODE (Eq. 10) provides theoretical  value regarding RF models.\n- Overall, the writing quality of the paper is very good. The explanations are clear, the equations rigorous.", "weaknesses": "While the structure of the paper is clear, some parts seem to lack purpose. For example, the introduction of the LQR problem and the optimal controller in the method seems superficial, as the resulting path is simply a linear path between the input image $\\mathbf{y}_0$ and a sampled noise $\\mathbf{y}_1$ (which is similar to SDEdit). Similarly, while the derived SDE in Eq. 10 has theoretical value, it is not used in practice as suggested by the Algorithms 1 & 2 in appendix C. Finally, the joint effect of the controller guidance parameters $\\gamma$ and $\\eta$ is not clearly exposed in the paper (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PR9v4xSrE", "reviewer": "Reviewer_g1Xz", "summary": "The paper proposes an \"inversion\" method for flow-based models, like Flux. The method allows to edit either natural images (from the distribution on which the model was trained) or unnatural images (like paint strokes). The inversion process balances between a forward process that inverts the reverse flow ODE (like in DDIM inversion) and a process that drives the inversion towards some fixed, random noise vector, so that the inversion lands in a region of typical realizations of the prior Gaussian distribution. Editing is done by starting from the noise obtained during the inversion and performing a reverse process that balances between the standard vector field and a vector field that pushes towards the input image, all while injecting the target text prompt.\nExperiments are presented on several datasets and editing tasks, including quantitative evaluations and a user study, comparing to several baseline methods.\nAn additional contribution of the paper is the introduction of an SDE variant for the proposed Controlled Rectified Flow process, which as a special case provides an SDE variant for the standard flow sampling process. This allows sampling using a stochastic process, which is less sensitive to the initial noise, rather than a deterministic process, which is a deterministic function of the initial noise.", "questions": "I would be happy the hear the authors' thoughts on the weaknesses mention above.\nIn particular:\n- What is the effect of y_1? How different are are results obtained with different y_1's? Is there a clever way to choose y_1?\n- How does the method compare to DDIM inversion on Flux performed up to some midway timestep t? Namely, the inversion is up to some t*, and then the sampling starts from this t* using the text prompt describing the desired edit.\n- How does the method work on other flow-based models, like SD3?\n- Why does the L2 metric in Table 1 not capture the apparent discrepancies between the edited and original images in Figs. 13,15? As mentioned above, the edited images look almost completely unrelated to the original images.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed Controlled Rectified Flow formulation is mathematical elegant.\n- The SDE variant of the method (and of regular flow as a special case) is quite nice, though it is not explored experimentally.\n- The paper is well written and easy to follow.", "weaknesses": "- The vast majority of the comparisons presented in the paper are against baselines implemented for different models (some of which are not even specified). Comparing between an editing method working with Flux and an editing method working with SD1.5, for instance, is meaningless. It does not reveal whether the differences in performance are due to the editing method or due to the fact that the underlying model is different (e.g. Flux is much stronger than SD1.5). The only exception is Fig. 7, which shows only two qualitative comparisons on Flux, but the baselines are very weak: SDEdit is known not to be consistent with the original image, and DDIM inversion is known to perform much better for editing when running the process only up to some intermediate t<1 (see e.g. Fig. 10 in [1]). I would expect to see at least the baseline of DDIM inversion for Flux going up to some midway point.\n- The qualitative results are underwhelming, especially on LSUN Bedroom. Fig. 4 in the main text seems cherry picked, as in Figs. 13,15 in the appendix, none of the edited results look remotely similar to the input stroke image. It is not clear why this is not captured by the L2 metric in Table 1. The age and gender editing results are also quite weak. For instance, in Figs. 16,17, all edits look very far from the source. \n- The proposed inversion process pushes towards some noise y_1, which is arbitrarily fixed a-priori. This means that the parameter eta interpolates between some fixed generic image corresponding to this y_1, and the original image, as seen in Fig. 3. However, the effect that the choice of this y_1 has on the editing is not explored. It seems to me that when the noise vector y_1 encodes a completely different geometric structure than the image being edited, the edited result may significantly deviate from the original image. On the other hands, when y_1 happens to encode a similar structure, the edited result may be closer to the original image. The paper doesn't provide an ablation study illustrating the effect of y_1 and doesn't provide a clever method to choose it.\n\n[1] Huberman-Spiegelglas et al., \"An edit friendly DDPM noise space: Inversion and manipulations\", CVPR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5XLi4nkYG", "reviewer": "Reviewer_QdiR", "summary": "Given a generative ODE that links pairs of noise and image, one can solve the forward/reverse ODEs to invert an image to noise or reconstruct noise into an image.\n\nThis paper modify the drift term of ODE to guide the process to terminate at high-likelihood noise/image. The guidance term is derived from LQR problem, which of solution is a minimum energy path to convert \"any\" image (noise) to given sample of random noise (image). This guidance is equal to conditional vector field that is computed analytically, without any optimization.\n\nThe authors propose a controlled forward/reverse ODE with drift term computed by linearly interpolating the conditional vector field and the original vector field. By adjusting the interpolation coefficient, the model achieves a trade-off between high-likelihood solutions and consistency to the original image.\n\nFurthermore, the paper interprets the controlled ODE within the context of stochastic differential equations (SDEs), highlighting the robustness of SDEs to initial conditions and offering a deeper insight into the proposed method.\n\nExperimental results validate the method's effectiveness in inversion-editing tasks, showing improved performance over existing techniques.", "questions": "- Have authors tried to restore blurry or downsampled image using the controlled forward/reverse ODE? Although it may not reconstruct the true signal exactly, it seems possible to get much clearer image.\n- Have authors tried non-rigid editing task? (e.g. dog -> jumping dog)\n- Have authors tried local editing (e.g. red flower and blue car -> yellow flower and blue car) rather than global context editing (e.g. man -> woman, photo -> pixar)?\n- Have authors reduced the NFE lower than 28? The reviewer just wonder if fast sampling is available even with controlled vector field, which may break the OT path established by reflow. Just for the reference, [1] used fixed point iteration to make more accurate inversion process and demonstrated that the method enables real-time editing using Flux.1 with 4NFEs. This does not mean that comparison with [1] is required.\n\n**Reference**\n\n[1] Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Models, arxiv 2023", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Theoretical analysis provides valuable insights for the proposed method.\n- The proposed framework of controlled ODE provides a design space for other applications, by adding regulation to the conditioned vector field.\n- Experimental analysis is well established and demonstrating the effectiveness of the proposed method compared to baselines.", "weaknesses": "- Missing comparison or discussion with fixed-point based inverse methods (ReNoise[1], AIDI[2]), which are applicable Euler solver. Their motivation is to reduce the error induced in inversion process with fixed point iterations and improve the editing performance.\n- By design, controlled ODE will generate majority sample since it is guided toward mode. Thus, the proposed method may lose diversity even though we can control it via parameter $\\gamma$. For example, edited faces always look at camera (Figure 5,6,7,10). But, this would be another open question.\n- Proposed controlled forward ODE may break the OT path of Rectified Flow, as it compute conditioned vector field $u_t(Y_t|y_1)$ with fixed noise sample $y_1\\sim \\mathcal{N}(0, \\mathrm{I})$. \n\n\n**References**\n\n[1] Effective real image editing with accelerated iterative diffusion inversion, ICCV 2023\n\n[2] ReNoise: Real Image Inversion Through Iterative Noising, ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)