Text-to-Image Rectified Flow as Plug-and-Play Priors
Authors: Xiaofeng Yang
Chen Cheng
Xulei Yang
Fayao Liu
Guosheng Lin
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Generation
Rectified Flow
Flow Matching
Primary area: generative models

Abstract: Large-scale diffusion models have achieved remarkable performance in generative tasks. Beyond their initial training applications, these models have proven their ability to function as versatile plug-and-play priors. For instance, 2D diffusion models can serve as loss functions to optimize 3D implicit models. Rectified Flow, a novel class of generative models, has demonstrated superior performance across various domains. Compared to diffusion-based methods, rectified flow approaches surpass them in terms of generation quality and efficiency. In this work, we present theoretical and experimental evidence demonstrating that rectified flow based methods offer similar functionalities to diffusion models — they can also serve as effective priors. Besides the generative capabilities of diffusion priors, motivated by the unique time-symmetry properties of rectified flow models, a variant of our method can additionally perform image inversion. Experimentally, our rectified flow based priors outperform their diffusion counterparts — the SDS and VSD losses — in text-to-3D generation. Our method also displays competitive performance in image inversion and editing. Code is available at: https://github.com/yangxiaofeng/rectified_flow_prior.

Published as a conference paper at ICLR 2025
TEXT-TO-IMAGE RECTIFIED FLOW AS PLUG-AND- PLAY PRIORS
Xiaofeng Yang', Cheng Chen', Xulei Yang', Fayao Liu², Guosheng Lin 'College of Computing and Data Science, Nanyang Technological University, Singapore Institute for Infocomm Research, A STAR,, Singapore yang. xi xiaofengentu.ccommmmmmmmm......m edu. sg. ge wew.ininnuuu....
ABSTRACT
Large-scale diffusion models have achieved remarkable performance in genera- tive tasks. Beyond their initial training applications, these models have proven their ability to function as versatile plug-and-play priors. For instance, 2D diffu- sion models can serve loss functions optimize 3D implicit models. Recti- fied Flow, a novel class of generative models, has demonstrated superior perfor- mance across various domains. Compared to diffusion-based methods, rectified flow approaches surpass them in terms of generation quality and efficiency. In this work, we present theoretical and experimental evidence demonstrating that rectified flow based methods offer similar functionalities to diffusion models they can also serve as effective priors. Besides the generative capabilities of dif- fusion priors, motivated by the unique time-symmetry properties of rectified flow models, a variant of our method can additionally perform image inversion. Exper- imentally, our rectified flow based priors outperform their diffusion counterparts the SDS and VSD losses - in text-to-3D generation. Our method also dis- plays competitive performance in image inversion and editing. Code is available at: https///stttu...commmm//iiiiccct floww. flowpprrrooorrrrrrrrrrr
INTRODUCTION
Recent advances in diffusion models (Ho al. 2020, Song & Ermon 2019 Rombach et al. 2022
have revolutionized generative tasks in image, video, and music production (Dhariwal & Nichol
2021 Guo et al. 2023b Huang al. 2023), often surpassing GANs (Goodfellow al. 2014 Karras et al. 2019). Beyond excelling in tasks for which they are specifically trained, various
studies (Graikos et al. 2022 Poole et al., [022 Wang et al. 2023 Yu al. 2024) indicate that these models can also serve plug-and-play priors for related tasks, 1.e., using the pretrained diffusion models as loss functions. For instance, Dreamfusion (Poole et al. 2022) introduces the SDS loss, utilizing pretrained text-to-image diffusion model to generate 3D objects. Several subsequent works (Hertz et al. 2023; Wang et al.) 2023 Yu et al., 20244] Katzir et al., 2024 Yang et al.) (003)) have refined the SDS loss to further enhance generation quality and diversity. applying diffusion priors to a variety of downstream applications, including 2D/3D editing and 4D generation.
Rectified flow methods (Liu et al. 2022 Albergo & Vanden- 2022). which connect data
and noise via trraight-line trajectories, are emerging as promising alternatives to diffusion models.
Early works like InstalFlow (Liu et al. 2024) demonstrate high-quality image synthesis in few steps. Recent advancements (Ma et al. [0244] Esser et al. 2024) achieve state-of-the-aa results in text-to- image generation. Given this growing research focus, it remains to be investigated whether large- scale pretrained rectified flow models can function as effective priors, similar to diffusion-basse methods.
In this work, we explore how to distill knowledge from pretrained text-to-image rectified flow mod- els and use them as priors for various tasks. To ensure a broad applicability, our approach is based on the generalized Stochastic Interpolants framework (Albergo et al. (023), rather than being directly tied to the rectified flow formulation. By incorporating specific interpolation factors of rectified flow, we adapt our method to this model type.
"Coresponding Author

---

Published as a conference paper at ICLR 2025
Real-world image: "A boat in river"
boat - rock
river - field
winter scene"
2D Editing with iRFDS
steaming mug hot chocolate with whipped cream
A 3d model of adorable cottage with thatched roof
green turtle swimming clear pond
A vintage porcelain doll with frilly dress
Text-tn-3D Generation with RFDS-Rev
Figure 1: Demonstration of potential use cases of the proposed rectified flow priors. Our methods can be used on 2D inversion and editing and text-to-3D generation.
We begin by proposing RFDS (Rectified Flow Distillation Sampling), a baseline method analogous to the SDS loss in diffusion models. We derive the baseline RFDS by calculating the gradients of the input image by reversing the flow-matching training process. We observe a phenomenon similar to that seen in diffusion-based methods: by removing the Jacobian of the rectified flow network, RFDS can generate meaningful images or 3D objects given text conditions.
Moreover, inspired by the straight line trajectories of rectified flow, we notice that the baseline RFDS can be extended to optimize input noise by taking the "negative gradient" of the original RFDS. This extension, referred
iRFDS (inverse RFDS), is particularly valuable for tasks such as image inversion and editing, where real images are mapped back to their latent noise representations before being edited with modified text prompts. Experiments show that iRFDS achieves high-quality results in image editing and inversion.
Finally, we propose RFDS-Rev (RFDS Reversal) overcome the limitations the baseline RFDS. Similar to the SDS loss, the original RFDS loss tends to produce outputs with limited details and diversity. We analyze this behavior through both intuitive reasoning and mathematical explanation. To address this issue, RFDS-Rev employs an iterative process that alternates between using iRFDS to recover the original noise and applying RFDS refine the input. This dual-step process signifi- cantly improves the quality of the generated content, enabling RFDS-Rev to outperform the baseline RFDS method and establish new benchmarks in downstream applications.
Additionally, we demonstrate that our method is applicable to score-matching methods, such dif- fusion models. The diffusion models, trained with score-mane matching objectives, can be reformulated deterministic PF-ODEs, making them compatible with proposed methods. Notably, when we formulate score-mate sooe-matching models in terms of PF-ODEs, our proposed RFDS baseline becomes identical to the SDS loss. Furthermore, our introduced iRFDS and RFDS-Rea Rev expand upon previ-
2

---

Published as a conference paper at ICLR 2025
SDk-like, score- soore-matching-- based knowledge distillation approaches, broadening their scope and applicability.
Our experiments are conducted 2D image inversion and editing for iRFDS and text-to-3D gen- cration for RFDS and RFDS-Rev. Notably, the LEDS-Rev method achieves a new ttae-offtthear performance in text-to-3D benchmarks among all 2D lifting methods, surpassing broad spectrum of diffusion priors. Some generation results can be found in Fig.
To summarize contributions:
We propose the first series of studies that utilize pretrained rectified flow models as priors. Our methods span wide array of applications, from generative tasks such as text-to-3I generation the inversion and editing of real images.
Compared diffusion-based priors, methods significantly outperform them in text-to- 3D generation and achieve highly competitive results in image inversion and editing.
2
PRELIMINARIES: FLOW-BASED MODELS AND RECTIFIED FL.WW
The idea of rectified flow is proposed in multiple concurrent works (Liuu al. 2022 Lipman al, 2022 Albergo & Vanden-Fijnden, 2022). Here, we follow the interpretation of Stochastic Inter-
polants (Albergo & Vanden- Eijnden 2022. Ma et al. 2024) to briefly introduce the basic concepts.
The Stochastic Interpolants framework starts by defining general process of interpolating noise and image:
3t 0r J,C.
Here, x. ~ p(x) represents the data sampled from the data distribution, while ~ NO0, I) denotes noise drawn from a standard Gaussian distribution. a₁ and 0, are two pre-defined variables used to control the interpolation trajectory and only related to Rectified flow is special case of the interpolation in Eq. by defining a and a, to change linearly with time In conditional flow a₁ 1 t and oz t. In rectified flow (Liu et al. 2022), the settings are reversed, with 0₂ = and o; = t.
matching (Lipman et al. 2022), the parameters
The process of obtaining I, can also be formulated as probability flow ODE (PF-ODE) with velocity field:
a = v(xt; L), v t) = di E x.
(2)
24 = x + de EE|E xe x²)
(3)
where á, and d, the gradients of a and de with respect t. We refer readers to Stochastic Interpolants (Albergo et al. 2023) and SiT (Ma et al. 2024) for detailed proofs of the above two equations.
Practically, flow-hased generative models learn the velocity v by parameterizing it with neural net- works using parameters & and optimizing flow-matching objective:
min [
E ua (Ft, á. o.E [| dt.
However, it is important to note that the primary objective of this work is not to sample æ (c.g., generating images using rectified flow networks). Instead, our aim to distill knowledge from such pretrained networks networks-specingg speeificalll by employing the pretrained network as a loss function- to hance the functionality of image generation networks across a wider array of applications, including image editing and text-to-3D generation.
Our proposed method also related to widely studied Ieffusion-basedd priors. We discuss these related works in Appendix Sec

---

Published as a conference paper at ICLR 2025
3 METHODS
In this section, we discuss the three proposed distillation methods: the RFDS (Rectified Flow Distillation Sampling) baseline method, the iRFDS (inverse RFDS) method for image inversion and finally the RFDS-Rev (RFDS-Reverse) method to improve the generation quality of baseline RFDS.
Formulation. The problem of optimization using pretrained rectified flow model Vi as loss func- tions can be formatted follows: we would like optimize g by 0* argming 0. g())). The form of æ can be Neural Radiance Field or 3D) Gaussian Splatting (Kerbl et al. 2023) in the 3D generation problem, where the parameter 0 denotes the parameters of the 3D models. In the 2D case, is directly 0. In the remainder of this section, we prove how to define the loss L and find its gradients with respect to 0.
3.1 RFDS
Consider the training loss function of flow-based generative models in Eq. 4 Using the pretrained network as loss function can be seen as optimizing the input rather than the model parameters. Specifically, when place the optimization variable æ in the aforementioned equation, it trans- forms as follows:
L. [II (x d.æ dec [|2] dr, with g()) and at = 043: + aec.
(5)
Here, the network parameter & is fixed and is randomly sampled noise. To find 0, the gradients of the above equation with respect can be written as:
((vs(xe t) â.æ die)
dx
x, =
(us(ar, á.æ die)
dæ
00
Flow Residual
Incobian
It can be further simplified as:
d1 df 8(-oics die))
Gong (stg.
Oste
Vo Frrtta (b., 2. 2 (ma(#e))) ()
Rosidall
Genertoo Jootiaa
As highlighted in diffusion-based methods (Poole al. 2022), calculating the network Jacobian computationally expensive and reacts inadequately to low levels of noise. We observe similar behaviors models based on How-matching. Consequently, we also choose to disregard the rectified flow network Jacobian by setting it to the identity matrix, akin to the approach taken with diffusion priors. This modification leads to the final RFDS loss:
iæ a0
VoLns E. = w(t) (no dua die)
(8)
Flow Revidual
Generator Jacobian
Here w(t)) represent value related timestep and it absorbs all constant values in 7 The RFDS loss is able to use the pretrained rectified flow model as loss function to optimize
given image æ. Given rectified flow based model, we can easily obtain its corresponding RFDS
loss by setting di and o, to -1 and or vice versa, depending on the way it formulates Eq. [ We include detailed algorithm for optimization using RFDS in Appendix Sec
3.2 IRFDS
A particularly interesting characteristic of the velocity prediction objective is its time-symmetry property (Liu et al. 2022). This allows the models to flow from noise to image and also from image back to noise. Motivated by this property, observe that method can be easily extended to
4

---

Published as a conference paper at ICLR 2025
optimize the noise. If our optimization objective changes from the input image noise E, Eq. now becomes:
g(0) to the
O( á.æ deg (are
die)
(,,,
(vo(22, á.æ die)(
(9)
de
dx
OE
de
Network Iscabian
de
Therefore, the final form of iRFDS can be represented as:
V VeCins (d, E w/ (1) (us( () 0, die)
(10)
Flow Residual
where the value w' (t) should have opposite signs wtt) due the relations between oe and at as stated in Sec. 2
Optimizing and recovering the original noise from image is crucial for image inversion and edit-
ing problems. By first retrieving the original noise and subsequently modifying the caption, we demonstrate the experiment section that our method can effectively edit real-world images. A detailed algorithm of iRFDS is included in Appendix Sec
3.3 RFDS-RES REV
RFDS-Reversal (RFDS-Rev) designed to enhance the baseline REDS terms of genera- tion quality. We observe that the RFDS method encounters issues similar to those found with the SDS loss, such as lack of object details. While several approaches, like the VSD loss, have been proposed improve the SDS loss and some applicable recti- fied flows, none shows effective results with rectified flow models based on our experiments.
Algorithm 1:
The RFDS-Rec Rev Algorithm.
To enhance the baseline REDS, we draw in- spiration from the learning mechanisms of Re- flow (Liu et al. 2022 2024). Recall that Reflow straightens the trajectory from noise image, leading to one-to-on one matching of the image and noise. In contrast, the mode-seeking na- ture of the original RFDS, which optimizes the image based on random noises, leads an aver- aged velocity and consequently blurred images. As shown on the left side of Fig. 2 when two noise points are sampled, although both give gradients towards the high-density areas in the image domain, the averaged gradient does not necessarily point in the correct direction. To address this issue, first introduce critical
Initialize the learnable parameter 0
while Not Converge do
Sample random timestep t
Sample random noise e
for n steps do
Freeze 0, optimize E with iRFDS (Eq. 10
Optimize 0 with optimized e based on
RFDS (Eq. 8
RETURN 0
assumption:
Assumption 1: For model finetuned with Reflow, given sample x. from data distribution p(æ), there is at most one corresponding from noise distribution (O, I) that corresponds to C..
This assumption is straightforward, the training process for Reflow enforces linear, non-crossing flow (Liu et al. 2022). Based on this assumption, we propose RFDS-Reversal (RFDS-Rev), a two- stage method to improve the baseline RFDS. Specifically, first stage performs noise reversal. In this stage, we use iRFDS to perform image inversion and optimize each randomly sampled noise such that their reversal occupies relatively static position within the noise distribution. Once the reversal is determined, in the second stage, we apply RFDS to the reversed noise. The algorithm of REDS-Rev we observe that running iRFDS for just one step yields very satisfactory results. Unless specified otherwise, the experiments involving RFDS-Res Rev employ n = as the default configuration. Al- though RFDS-Rev initially designed with the assumption of models fine-tuned using Reflow, our experimental results reveal that enhances the performance of the baseline REDS on models both with and without Reflow training.
depicted on the right in Fig. 2 the
summarized in Algorithm [ experiments,
5

---

Published as a conference paper at ICLR 2025
Sampling Path
Gradlent on Image
Average Gradlent
Gradient on Noise
Noise distribution
Data distribution
Sampled noise
Oplimized noise
g
pxx)
p(x)
p
RFDS
IRFDS
RFDS-Rev
Figure 2: Illustration of our proposed methods.
Understanding REDS-Rev from the Angle of Euler Sampling. In addition to the intuitive deriva- tion above, we demonstrate that RFDS-Rev can also be mathematically derived from the difference between the RFDS baseline and the commonly used Euler sampling in rectified flow generation. Specifically, we observe that RFDS is closely related to Euler sampling. with the key difference being the noise sampling strategy. Our proposed RFDS-Rev bridges this gap by performing an inversion to recover the original noise map. The detailed proof is provided in Appendix Sec B
3.4 APPLYING IREDS AND RFDS-REY TO DIFFUSION MODELS
As discussed previous works (Song al. 2020 Albergo al. 2023). diffusion models trained with score-matching objectives have deterministic PF-ODE form and can be expressed in the form of Eq. 2 We observe that by converting the score function to velocity field, all three of our proposed methods become applicable to diffusion models. Additionally, we prove that our proposed RFDS baseline is identical to the vanilla SDS loss when applied diffusion models. We include detailed proof in Appendix Sec
3.5 CLASSIFIER FREE GUIDANCE (CFG)
The above equations are derived using the rectified flow network Vg. In practice, since the base rectified flow models (Liu al. 2024 Esser al. 2024) usually trained with classifier free guidance (Ho Salimans] 2022). we observe that using a classifier free guidance version of the network bg helps to improve the performance. Ablations included in Appendix Sec
4 EXPERIMENTS
In this section, conduct extensive experiments to compare our proposed methods with diffusion- based approaches. We perform experiments on two types of rectified flow-based methods: Stable Diffusion v3 (SD3), trained with flow-matching but without Reflow finetuning, and InstaFlow, fine- tuned with Reflow. In terms of 2D performance, Stable Diffusion v3 is one of the state-of-the-an -ixaeeo-mmae models, trained with more data and using larger backbones. InstaFlow achieves results similar to those of Stable Diffusion v2.1 (SD2.1) (Rombach et al., 2022). First, we focus eval- uating RFDS and RFDS-Rev in both 2D and 3D text-guided generation scenarios. Subsequently, we explore the performance of iRFDS in image inversion tasks. The diffusion-based methods are implemented using S 1 and the Threestudio codebase (Guo al. 2023a). We include the imple- mentation details in the Appendix for further reference.
4.1 RFDS vs. REDS-REY VS. DIFFUSION PRIORS VS. DIFFUSION REDS-RET REV
Toy Experiments on Optimization of 2D Case. We begin by conducting an intuitive experiment in 2D setting, where we set 0. This simplified setup removes the complexities of the 3D model, allowing us to demonstrate the potential upper limit of 3D generation quality. Results are shown in Fig. Results reveal that our RFDS baseline, when integrated with InstaFlow, performs

---

Published as a conference paper at ICLR 2025
REDS (InstaFlow)
REDS-Rey (InstaFlow)
REDS (SD3))
REDS-Rex Rev (SIX3)
SDS (Diffusio))
VSD (Diffusioo)
REDS-Rev (Dilfussio)
Figure 3: Comparison. Text Prompts: "A castle next to mountain", "An astronaut is riding
horse" and "A monster truck".
REDS (InstaFlow)
REDS-Rev (InstaFkw)
REDS (SD3)
REDS-Rev (SD3)
SDS (Diffuuson)
VSD (Diffuusio))
Figure 4: 3D Comparison. Text Prompts: "An antique wooden rocking horse", "A bear holding a sign that says Hello world" and "Hot popcorn jump out from the red striped popcorn maker".
comparably the diffusion-based SDS loss. Moreover, when combined with Stable Diffusion v3, the RFDS baseline significantly enhances generation quality.
We also apply our proposed method, REDS-Rev, to two rectified flow-based models and the diffu- sion model Stable Diffusion v2.1, as discussed in Se4 To better approximate the real use cases, we use only one step of iRFDS improve efficiency for all experiments. Our results indicate that RFDS-Rev significantly enhances generation performance on InstaFlow, moderately improves per- formance on SD3, and slightly improves performance on the diffusion model. These findings are consistent with the assumptions made when proposing the RFDS-Rev method. The Reflow process straightens the noise-to-data trajectory, making our assumptions particularly applicable to models finetuned with Reflow. Although SD3 not finetuned with Reflow, method still improves gen- eration quality. However, in acore-matching models like diffusion, noise is applied stochastically to corrupt the data, making it challenging to trace the original noise with just step of iRFDS. For

---

Published as a conference paper at ICLR 2025
Table 1: Results of Text-to-3D on Bench dataset. Our proposed methods achieve state-of-the-art performance among 2D lifting methods, beating the diffusion based SDS and VSD priors.
Method
Dreamfusion (SDS) PromificDreamer (VSD)
RFDS
REDS-Rev
RFDS
RFDS-Rev
Base Model
Diffusion
Diffusion
InstaFlow
InstaFlow
SD3
SD3
Single Object
24.9
51.1
46.9
57.6
49.4
59.8
Surroundings
19.3
42.5
34.5
44.6
39.3
54.5
Multiple Objects
17.3
45.7
28.3
44.6
42.9
55.2
Average
20.5
46.4
36.5
48.9
43,9
56.5
Tipuot
Nall Inversion (Diffindon))
DDS (Diffision)
IRFISS (Ours) (IrstaPloo)
IRFIS (Quan)
IRFIS (Qurs) (Uiffrssoo)
A-eel Skysera CESTESEESSSSS TEXt nIVE.:
A Gal sitnng on the CRESS.
Gabrn Surtoundoot by lonsss Winer SCTTE:
A sitting the table sofa.
Figure 5: Comparison of 2D editing. Our iRFDS method able to perform prompt-faithful editing.
this reason, in the 3D experiments, we apply our methods only flow-matching based approaches, and compare them with SDS-like diffusion- approaches.
Text-to-3D Generation by Lifting 2D Models. An important application of rectified flow dif- fusion priors is lifting models from 2D to 3D. We present quantitative and qualitative results for text-to-3D generation. The qualitative results displayed in Fig. 4 We observe that both of our proposed methods are capable of generating high-quality 3D objects, with the RFDS-Rev method enhancing the baseline RFDS in terms of object detail and color accuracy. Qualitatively, when ap- plied to similar quality base models (InstaFlow and SD2.1), the RFDS baseline shows competitive performance compared to the SDS loss used diffusion-basee methods. Compared VSD, RFDS- Rev achieves comparable level of detail in generated objects. Notably, VSD experiences issues with "cxploding" artifacts. contrast, the RFDS-Re -ev loss does not exhibit this problem. When apply method to the more powerful SD3 base model, observe that it outperforms diffu- sion priors. Moreover, RFDS-Res further enhances the generation quality compared to the RFDS baseline SD3. Additional qualitative results of RFDS-RR Rev are included in Appendix M
We also conduct quantitative experiments on the text-to-3D benchmark Bench He et al. 2023 The dataset contains 300 text prompts for text-to-3D generation, making it the largest text-to-31)
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparison of 2D editing.
Method
DDS
Null Inversion
iRFDS (InstaFlow)
IRFDS (SD3)
IRFDS (Diffusion)
CLIP Score
0.294
0.298
0.306
0.297
0.296
User Preference (Semantic)
7.3%
20.1%
43.6 %
16.8%
12.2%
User Preference (Consistency)
10.7%
32.5%
33.4%
13.1%
10.3%
benchmark available. Results are demonstrated in Table [ We report the quality score, derived by rendering multi-view images and calculating the CLIP (Radford et al.) 2021) and ImageReward (Xu et al.. 2024) scores between the rendered images and the corresponding text prompt. We observe that our proposed methods significantly outperform diffusion-basee methods. Furthermore, the RFDS- Rev method sets new state-of-the-arr performance among all 2D lifting methods in this benchmark.
4.2 IRFDS vs. DIFFUSION METHODS
In this section, we study the performance of iRFDS in 2D image inversion and subsequent text- guided editing. Image inversion is also widely explored topic in diffusion-based methods, known for its capability to edit real-world images. We compare iRFDS against a popular method null inversion (Mokady
2023 on real-world images. Additionally, we evaluate our method against the diffusion prior based optimization method, the DDS loss (Hertz et al. 2023), which
directly optimizes the original images.
The results of the inversion and editing experiments are summarized in Fig. 5 Among the dif- ferent variants of our iRFDS methods, we find that the model finetuned with Reflow produces the best results. Further visualizations can be found in Appendix Sec. [ We also conduct quantitative experiments to calculate the CLIP score between the target caption and the image after editing. Ad- ditionally, we carry out user study evaluate the editing performance from the users' perspective in terms of Semantic coherence and Consistency of unrelated areas. The experiments are performed on 15 real images randomly collected from the internet, with each image being edited three times ing different captions. The results are summarized Table. 2 We observe that our proposed method iRFDS + InstaFlow achieves the best performance in terms of CLIP scores and user preference.
The above results are achieved using the simplest implementation, where the learned noise serves the starting noise for image generation. The performance of our proposed iRFDS can be fur- ther enhanced by integrating the learned noise into an intermediate flow generation step-a sim- ple yet effective technique commonly employed in DDIM inversion (Hertz et al.) (022) and null- inversion (Mokady et al. 2023). This method improves background consistency and ensures more accurate color reproduction. In experiments, we observed that inserting the noise at 10% of the total steps achieves the best balance between consistency and editing performance. Detailed comparisons are provided the Appendix Sec K
We also provide comparison of image reconstruction ability. The experiments are conducted on the same dataset, evaluating the PSNR scores between the inversion results and the original images. Our proposed iRFDS baseline achieves a PSNR of 28.52. By applying the aforementioned insertion trick, the PSNR improves to 28.96, surpassing the null-inversion score of 28.81. visual comparison is provided in Appendix Sec L
4.3 ABLATION EXPERIMENTS
iRFDS optimization steps in REDS-Rev (Algorithm []. Fig. 6 shows results from both 2D and 3D cases. We observe that performing the iRFDS for 1 step achieves the best efficiency and performance trade-off.
w/ Network Jacobian vs. w/o Network Jacobian. As illustrated in Fig. 7 we observe that keeping the network Jacobian can hardly generate meaningful images even in the 2D case. This observation consistent for both SD3 and InstaFlow.
Additional Ablation Results. We also include additional ablation results in Appendix Sec G The
experiments include: ablation the effect of CFG, RFDS-Rew Rev vs. directly applying the same idea of VSD to rectified flow and a comparison of computational cost between our methods and diffusion-basee methods.

---

Published as a conference paper at ICLR 2025
InstaRow
2-00 (RFDS)
(RFDS
A pair of hiking boots on a forest trail, surrounded by autumn laavss.
InstaFlow
R
n= An antique glass per bottle
(=FDSS
n=1
n=2
Figure 6: Experiments of the noise optimization step in RFDS- Top: 2D case. Bottom: 3D case. Optimizing the iRFDS for one step offers the optimal balance between efficiency and performance.
W TIAAA
intulondori
seashell the
fox peexing through the bashes in soowy landssaape
Figure 7: Ablation experiments of ignoring the rectified flow network Jacobian. We observe that is difficult to generate images by keeping the Jacobian of the rectified flow network.
5 SUMMARY OF OBSERVATIONS AND DISCUSSION
Summary of observations. 1), RFDS and RFDS-Rev are effective on flow-matching models, re- gardless of whether Reflow finettuning applied (Fig. 3 Table. TT. 2). Reflow fine-tuning en- hances iRFDS performance on image inversion and editing tasks (Fig. S Table 3). When ap- plied to ccrre-matciing-bsee methods like diffusion models, RFDS equivalent to the SDS loss. RFDS-Rev offers slight improvements over the SDS baseline. While iRFDS may not produce the best editing results with diffusion models, it presents a simple alternative to other inversion methods.
Additional advantages over diffusion based priors. Besides the performance improvements, observe that methods based on rectified flow achieve faster convergence during the optimization of 3D models. Intuitively, if we rearrange Eq. can be regarded as learning toward a direct objective 0g + c. In contrast, diffusion models rely on the difference predicted noises to guide the optimization process. A detailed visual comparison is included in Appendix See H In terms of wall-clock time, generating a single 3D scene using InstaFlow with RFDS takes less than 20 minutes on A6000 GPUs. Diffusion model-based methods, such Stable Diffusion 2.1, utilize the same number of parameters as InstaFlow. However, the SDS loss baseline requires 40 minutes to produce scene with reasonably good quality, while VSD takes over 1.5 hours.
CONCLUSION, LIMITATIONS
In this work, introduce the SDS and VSD counterparts in rectified flow based methods. We pro- pose three methods that utilize rectified flow as plug-and-plas and--lay priors. Our two generative methods, RFDS and REDS-Rev, outperform SDS and VSD terms of text-to-3D generation. Additionally, the iRFDS method demonstrates robust capabilities in image inversion and text-guided editing.
Limitations. Since the 2D model lacks training with camera pose information, our proposed meth- ods encounter similar issues to the SDS and VSD losses in 3D generation, such as the multi-face issue. These limitations can addressed future studies by training pose-aware models using multi-view data, similar to current diffusion based approaches (Ye et al. (2033] Shi et al.) 2023a Li cet al. 2023 Shi et al. 2023b Long et al. 2023). Moreover, currently our proposed iRFDS method does taken the CFG mismatch issue into considerations. A promising future direction will be combining the iRFDS method with null-nnversion (Mokady et al. 2023) to address the CFG issue.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M231.b0021).. This research is also supported by the MoE AcRF Tier 2 grant (MOE-T2EP20223-000))
REFERENCES
Michael S Albergo and Eric Vanden-Eijudent Building normalizing flows with stochastic inter- polants. arXiv preprint arXiv:2209.15577 15571, 2022.
Michael Albergo, Nicholas M Boffi, and Eric Vanden Manden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint (tt.::2333008799,, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, and Robin Rombach. Scaling rectified flow transformers for high-resolution image syn- thesis. In Forty-first International Conference on Machine Learning, 2024. URL https:
nwt. wttp://www......oommmmmoaaaaaaaa
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. Advances Neural Information Processing Systems, 35:14715-14728, 2022.
Yuan-Chen Guo, Ying- Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. httpss//gg..bbb...c/rrrreesssss Ehreestudio, 2023a.
com
project/
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.047235 2023b.
Yuze He. Yushi Bai, Matthieu Lin, Wang Zhao, Yubin Hu, Jenny Sheng, Ran Yi, Juanzi Li, and Yong-Jin Liu. T3bench: Benchmarking current progress in text-to-3d generation. arXiv preprint arXiv:2310.02997, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to-prome image editing with cross attention control. arXiv preprint 0rX1022208.01626, 2022.
Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv:2304.07009 2023.
arXiv preprint
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Qingqing Huang. Daniel S Park, Tao Wang. Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang. Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03911 2023.

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. In The Twelfth International Conference on Learning Represenlations, 2024. URL https:
7 //operreview
forsst/fooommm/ d=d1 IMcml Adk
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1-14, 2023.
Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. arXiv preprint arXiv:2310.02596, 2023.
Yixun Liang, Xin Yang. Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6517-6524 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.027477 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 03003, 2022.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In International Conference on Learn- ing Representations, 2024.
Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.I 15008, 2023.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric "anden-Eijndee and Sain- ing Xie. Sit: Exploring flow and diffusion-basee generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08744 2024.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), -1(4):1-15, 2022.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763, PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Ruoxi Shi, Hansheng Chen. Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: a single image to consistent multi-view diffusion base model. arXiv preprint 011iv:2310.11 10.22/101///11 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308. 165 2023b.
12

---

Published as a conference paper at ICLR 2025
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.1681 16818, 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv 2023.
preprint arXiv:2305.162
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Xiaofeng Yang, Yiwen Chen, Cheng Chen, Chi Zhang, Yi Xu, Xulei Yang. Fayao Liu, and Guosheng Lin. Learn to optimize denoising scores for 3d generation. arXiv:2312.04820 2023.
Xiaofeng Yang, Fayao Liu, Yi Xu, Hanjing Su, Qingyao Wu, and Guosheng Lin. Diverse and stable 2d diffusion guided text to 3d generation with noise recalibration. Proceedings of the AAAI Conference on Artificial Intelligence, 28476. https: //css aaal. org/ index.idexxx/// http://bAAA/aatticll//////////
(57)))669-66577 Mar. 2024. doi: 10.1669/aaii..333.
Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent-l-to-3: Consistent im- age to 3d view synthesis via geometry-awar aware diffusion models. arXiv preprint 10X002330000000, 2023.
Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint arXiv:23 10.022220 2023.
Xin Yu, Yuan-Chen Guo, Yangguang Li,
Ding Liang, Song-Hai Zhang, and XIAOJUAN QI. Text- to-3d with classifier score distillation. In The Twelfth International Conference on Learning Rep- resentations, 2024. htLphtts::/ /tctpa://openreeeeee htt.://tttetooooeocccacatttttaa.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A BROADER IMPACTS.
Our approach significantly expands the potential applications of pretrained rectified flow models. We demonstrate that a text-to-image model can be adapted for both image editing and text-to-3D generation tasks. However, since we utilize pretrained lext-to-image rectified flow models our foundational network, our methods might inherit the biases present in these networks.
B PROOF: UNDERSTANDING RFDS-RA REV FROM THE ANGLE OF EULER SAMPLING.
Euler sampling is one of the most fundamental and widely used sampling strategies in flow matching models (Albergo & Vanden- Eijnden 2022 Lipman et al.; [0022] Liu et al. [024]. A natural question arises: why does Euler Sampling succeed generating high-quality 2D images, while the RFDS
loss does not? Within the framework of a flow matching model, Euler sampling is defined as:
Li+At 2t At)(xx.t).
(11)
In words, the image is generated by moving a small step each time given the predicted velocity. Given the de predicted at each step, we can recover the original image by applying the definition of æt in Eq.
altAr + OLtIC Atv(x,, t) + ay + ozc.
(12)
If re-arrange the above Equation and add -0'+ a. both side, we can have:
a+A - a+tAIT+= = Atv(a,, t) + a; + orE - OHAIE - a(+A(T=.
(13)
By dividing both side with At, we can have the final form of the updating rule of Euler sampler:
0++ Ar. v( t) a. oye.
(14)
The left side of the equation indicates the direction of x. updates. Notably, the right side of the equation the same as the proposed RFDS loss (Eq. R). with one key difference: Fuler Sampling, the noise is fixed initial noise, whereas in RFDS, the noise is randomly sampled. However, in the context of 3D generation, sampling a fixed noise for the entire 3D scene is impractical because 3D optimization is inherently stochastic process, and no fixed noise corresponds to every ren- dered view. Nevertheless, this "fixed noise" can be identified or learned. Our proposed RFDS-Rev addresses this by using iRFDS to perform image inversion each rendered view. identifying the corresponding static noise and ultimately bridging the gap between RFDS and Euler sampling.
C PROOF: RFDS, IRFDS, REDS-REV WITH DIFFUSION MODELS
RFDS is Identical to SDS When Expressed in Terms of Score Function
As proven in Stochastic Interpolants AAlbergo et al. 2023 the velocity v(xzz) can be expressed in terms of score function s learned with matre-matchin objective
v(w)) o, o,s( a,oz) + a
(15)
For the detailed proof of this relation, we refer the readers to Albergo et al. (023) and Ma et al. 2024
By substituting this relation into RFDS (Eq. 8 and considering the relation obtain the same equation the SDS loss
-cp
we directly
a,
Öa ae Generator Jacoboan
(0, w(t) (c+(#))- -
(16)
Score Residual
14

---

Published as a conference paper at ICLR 2025
iREDS Expressed in Terms of Score Function
Similarly, we derive the iRFDS in terms of score function
V. V.Cinal., x. ~ w (cx(
(17)
Score Residual
After the two formulas are derived, we can naturally arrive at RFDS-Rev
D IMPLEMENTATION DETAILS
RFDS and REDS-Rev for 3D generation. We use the 3D model implicit model Instant- NGP (Müller et al., 2022) as the 3D backbone. Each 3D model is optimized for 15000 steps. We use CFG of 50 for all 3D experiments and 2D toy experiments. The model is optimized with a resolution of 256 for the first 5000 steps and then 500 for the final 10000 steps. The experiments are carried out on NVIDIA A6000 GPUs. On InstaFlow, generating a single 3D scene takes approxi- mately 30 minutes of wall-clock time with RFDS- Rev and 20 minutes with the RFDS baseline. On SD3, the same task requires around hour with RFDS-Re Rev and 40 minutes with the RFDS baseline. We use w((t) 1 SD3 and w(t) -1 on InstaFlow. Choosing iRFDS step size is important to achieve high quality generation with RFDS-Res For InstaFlow, we use a stepsize 1. For SD3, we observe that stepsize of o, produce reasonable results.
iREDS for image inversion and editing. The inversion starts from a randomly sampled Gaussian noise. We optimize the noise for 1000 steps using iRFDS and CFG I with learning rate of 1+1--3 To facilitate effective noise optimization, we add one additional loss to enforce the noise follows Gaussian distribution. Specifically, we add loss enforce the mean and variance of the current noise to be zero and one respectively. After the noise is optimized, we change the caption to the target caption and run the forward flow for 5 steps using CFG 1.5. We use w' (t) 1 on SD3 and w/ (t) on InstaFlow.
E ADDITIONAL RELATED WORKS
Diffusion as plug-and-play priors. Our work is greatly motivated by the development of diffusion- based priors. The earliest work of diffusion prior (Graikos et al.) 2022) requires to backpropagate through the diffusion U-Net. Dreamfusion (Poole et al. 2022) recognizes that such a backpropagate process greatly hurts the performance of diffusion priors. By ignoring the U-Net Jacobian, the SDS loss proposed in Dreamfusion can be used for 3D generation and image editing. However, the initial version of the SDS loss suffers greatly from the lack of details and diversity. DDS loss (Hertz al., 2023) proposes an improved version of the SDS to improve image editing by taking the difference between the current SDS and the source image SDS. VSD loss (Wang al.) (023) improves the SDS loss for 3D generation problems. Specifically, it first trains a LoRA of the current 3D model and then takes the difference between the diffusion SDS and LoRA SDS. Although lots of methods have been proposed for diffusion models (Liang et al. 2024 Yang et al.) 2024 Katzir et al., 2024) our work marks as the first work that studies how to effectively use rectified flow as priors.
F ALGORITHM FOR RFDS AND IRFDS
We list the detailed algorithm for RFDS and iRFDS in Algorithm
2 and Algorithm 3
G ADDITIONAL ABLATION EXPERIMENTS
Ablation Results on CFG. The scale classifier-free guidance (CFG) (Ho Salimans] 2022) plays a crucial role in diffusion-based methods, such SDS and VSD. We observe very similar phenomenon with the rectified flow priors. As demonstrated in Fig. g both of the proposed methods require CFG greater than 10 to learn reasonable shapes. However, when the CFG becomes exces- colors. In contrast, REDS-Res -
sively large, the 3D objects generated by RFDS exhibit remains robust large CFG values, when the CFG exceeds 2000.
15

---

Published as a conference paper at ICLR 2025
Algorithm 2:
The RFDS Algorithm.
Algorithm 3: The iRFDS Algorithm.
Initialize the learnable parameter
Initialize the learnable parameter e
while Not Converge do
Get initial image
Sample random timestep t
3 while Not Converge do
Sample random noise
Sample random timestep
Optimize 0 with based on RFDS (Eq.
Optimize using fixed æ based on
iREDS (Eq. 10
RETURN 0
RETURN
RFDS InstaFlow
RFDS- InstaFlow
RFDS SD3
REDS-Rey Rev SD3
CPG 2000
200
CRG
CRG 10
CRG 2
Figure 8: Ablation experiments of classifer free guidance scale on text-to-3D generation. Prompt: A DSLR image of hamburger.
RFDS-Rev vs. RFDS-VSD, As mentioned in the main text, some of the existing methods aimed improving the diffusion models can be used directly on rectified flow based methods. We explore to combine VSD with the baseline RFDS, denoted as RFDS-VSD, Specifically, we train rectified flow LoRA model based on the current rendered images and then calculate the gradients by taking the difference between RFDS and REDS-LORA following the VSD setting. Results are shown in Fig. 9 Our experiments, conducted using the InstaFlow backbone, demonstrate that RFDS- Rev produces significantly better results compared to RFDS- VSD,, despite RFDS VSD requiring more computational resources. Notably, implementing VSD the SD3 model presents significant challenges for most currently available commercial GPUs due to its requirement for fine-tuning the base model, which demands excessive GPU memory.
16

---

Published as a conference paper at ICLR 2025
RHOSAV
RFDS-Y'D
1335 hoor
RATES
RHDS-VSD
Mosbin pROcad
Figure 9: Ablation experiments of RFDS-Rew RFDS-VSD Top: 2D case. Bottom: 3D case.
H COMPARISON OF CONVERGENCE SPEED.
we observe that our rectified flow based methods lead to much faster conver- gence speed when doing text-to-3D generation.
As listed in Fig. 10
I COMPARISON OF THE COMPUTATIONAL COST.
We evaluate the computational costs by examining the number of forward and backward passes of the diffusion or rectified flow network required optimization iteration. Results are list Table. The RFDS baseline has the same computational demands as the SDS loss. Due to the calculation of CFG (Ho & Salimans 2022), they both require two forward passes. RFDS-Res ReDS-Rev requires only one additional forward pass, whereas VSD needs two additional forward passes and one additional costly backward pass.
Table 3: Computational cost of one iteration based the number of forward and backward passes of the network.
Method
SDS Poole et
2023
Wang all 2023
DDS Hertz et al, 2023
REDS
IREDS
RFDS-Rey
Category
Diclusion
Diffusion
Diffusion
Rectifid Flow
Rectifed Flow
Rectifed Flow
Computation
Foeward
2
3
Backward
0
J MORE RESULTS OF IMAGE INVERSION AND EDITING USING IREDS.
We show results of 2D editing in Fig.
K IMPROVE THE PERFORMANCE OF IRFDS BY INCORPORATING NOISE INTO THE INTERMEDIATE GENERATION STEP.
As discussed in the main text, the performance our proposed iRFDS can be further enhanced by integrating the learned noise into an intermediate flow generation step. A visual comparison of this approach is presented in Fig. I2
L COMPARISON OF IMAGE RECONSTRUCTION BETWEEN IRFDS AND NULL-INVERSION
visual comparison of the image reconstruction ability between our proposed iRFDS and null-
inversion is presented in Fig. 13
17

---

Published as a conference paper at ICLR 2025
Stap 1000
Stsp-000
Step 7000
Step 10000
RSD3
RFDS Rev (SDX)
REDS-Rey (lrstaPow))
SOS Dilrcion
VSD (Difitaoo))
Figure 10: Comparison of convergence speed in 3D generation. Caption: A DSLR image of hamburger. The 3D model trained with the same learning rate. We observe that the rectified flow based methods converge much faster compared with diffusion-based methods.
M MORE RESULTS OF TEXT-TO-3D GENERATION
We show qualitative results of text-to-3I) generation in Fig.
14
Fig.
15
Fig.
16
and Fig.
17
18

---

Published as a conference paper at ICLR 2025
Imptt
Nail inverssion (1ilissio)))
DDS (Tilission)
BRFDS (Qurs)
(InstaFow)
BRETDS Ous SSF
IRETS (Ouus) (TIIlussoo)
A castle next river road
villa the
A eat tiger Sitting on the table
church beside lake, sunset
Figure 11: More results on 2D editing.
N IRFDS USER STUDY DETAILS
The user study is carried out with google doc. The users are ask to select the best editing results from the 4 methods. A screenshot of the user study is shown in Fig 18
O MORE COMPARISON WITH OTHER STATE-OF-THE-ARY 3D GENERATION METHODS
We further compare our proposed method (RFDS- Rev + SD3) with other satte-of-the-at 3D gener- ation methods, including LucidDreamer (Liang al. 2024). DreamCraft3) (Sun et al. 2023 and Dreamrr (Yi et al.. 2023). The results are presented in Fig Our findings demonstrate that our proposed method is versatile, as it can be applied both NeRF and 3DGS backbones. Our method achieves highly competitive performance across different settings.
19

---

Published as a conference paper at ICLR 2025
Original Image
iRFDS (Step 0)
iRFDS (Step 0.1)
Null-inversion
A eeg cat sitingg on the grass.
A cat tiger sitting the table
bext rock in a niver
Figure 12: Inserting the iRFDS learned noise into the intermediate generation step improves back- ground and color consistency.
InFDS
IRFDS 0.1)
Original Image
PSNR 20
PSNR 28.03
PSNR 20.52
A boal in a
Original Image
IRFDS
IRIDS 0.1)
Nill Lnveesoo
PSNR 28.52
A sitting on a table
Figure 13: Comparison of image reconstruction.
20

---

Published as a conference paper at ICLR 2025
An intricate ceramic vase with peonies painted on it
sparkling diamond tiara
A bright, yellow rubber duck
A well-loved stuffed teddy bear
A pair of worn- blue jeans
An old, frayed straw hat.
bright fire hydrant
A vibrant sunflower with green leave Text-to-3D Generation with RFDS-Rev
Figure 14: More results on text-to-3D generation. Model:SD3
21

---

Published as a conference paper at ICLR 2025
vibrant orange pumpkin sitting on hay bale
blue denim jacket hanging on a metal coat rack
A pair of white sneakers a black mat
A blue ceramic mug filled with steaming coffee
A green wellington boot in mud
A white swan tranquil lake
A hot air balloon in clear sky
A green frog lily pad Text-lo-3D Generation with RFDS-Rev
Figure 15: More results on text-to-3D generation. Model:SD3
22

---

Published as a conference paper at ICLR 2025
A 3d model of an adorable cottage with thatched roof
florist making bouquet with fresh flowers
wom-out leather briefcase
A faux-fur -ur leopard print hat
A chipped porcelain teacup
A partly broken shell of a tortoise
A chameleon perched on a tree branch
pink ceramic vase filled with frosh white lilies
Text-to-3D Generation with RFDS-Res Ree
Figure 16: More results on text-to-3D generation. Iodel:InstaFlow
23

---

Published as a conference paper at ICLR 2025
white picket fence around gardlen
hot air balloon in clear sky
yellow school bus a city strect
A colorful parrot on a jungle
An old hrass key langs for the clsstdd antique chest could unlock
A colorful amay of spices in tiny jars
A footballer is kicking a socoer ball
A frazee rash super sport car Text-to-3D Generation with REDS-Rev
Figure 17: More results on text-to-3D generation. Iodel:InstaFlow
24

---

Published as a conference paper at ICLR 2025
Given source imaee a boat rivec' which method has better semantie correspondence when changing 'boat" 'rock'?
consistency unrelsted areas when changing 'boet' "rock?
Option
Option
Option
Option
Option
Figure 18: User study page
RFDS-Rev (Ours)
LuridDmamer
DreamCraft30
Gaussian
DSLR R tmage at a hamburger
small porcelatn white rabbit tigurtnn
NeRF
pair of scissors craft table
NeRF
An alarm clock is glaring at tousled pillow
Figure 19: Comparison with other state-of-the-art 3D generation methods
25


Reviews:
[{"review_id": "P3I8IexAzF", "reviewer": "Reviewer_jHkr", "summary": "The submitted work propose a way of utilizing rectified flow models a prior, that implies the internal knowledge of the model as in the form of an objective. Throughout the manuscript, the authors propose three different algorithms in this regard, where they are named as RFDS (Rectified Flow Distillation Sampling), iRFDS (inverse RFDS) and RFDS-Rev (RFDS Reversal). While discussing the analogy between the SDS (Score Distillation Sampling), which serves as a prior for models trained with score-matching objective (e.g. diffusion models), they demonstrate how these algorithms can be utilized as a loss objective reflecting the knowledge of the rectified flow model and the image inversion & editing task. Furthermore, the authors propose RFDS-Rev algorithm as the improved version of the baseline algorithm (RFDS).\n\nTo demonstrate the effectiveness of the set of the proposed algorithms, authors show the effectiveness of them on a variety of tasks. Initially, the authors demonstrate how does the proposed method improve over SDS in diffusion models in terms of generation quality. Following, they illustrate the applications of the proposed algorithms in inversion based tasks (e.g. image editing) and text-to-3D generation with a 2D prior from rectified flow models. For all of the demonstrated algorithms, authors provide qualitative and quantitative results that shows the effectiveness of the proposed framework.", "questions": "- The authors are strongly encouraged to discuss more on the quality degradation issues mentioned in weaknesses. If this is an effect of the proposed algorithm, it should be discussed throughly. \n- Authors should consider  extending the quantitative results on the reconstruction properties of the iRFDS. Despite the fact that the current evaluation assesses the edit in terms of success in reflecting the semantics, the qualitative results seems that the algorithm fails in preserving edit independent details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper proposed the first algorithm that utilizes rectified flow models as priors, to both enable implicit information encoded in the rectified flow model and inversion based image editing with such models.\n- In addition to the baseline method provided, authors also propose an extension named RFDS-Rev, that improves over the baseline objective RFDS, that combines the algorithms proposed together and promises improved generation quality.\n- Proposed method showcases satisfactory results on Text-to-3D generation, which shows the effectiveness of the proposed method over providing a prior on multiple views.\n- Authors introduce a simplifying assumption enabling to efficiently using rectified flow models as priors, by simplifying the generator Jacobian. This serves as a modification for improving the efficiency of the proposed method.\n- The paper provides sufficient amount of experiments demonstrating the effect of the design decisions made on the algorithms, which also serve as insightful observations (number of steps, effect of the Jacobian).", "weaknesses": "- In the examples provided, there is a significant saturation effect on the provided results (see Fig. 5, row 2 and Fig. 6, examples from SD3). It is unclear if that effect is a result of the proposed method or a property of the rectified flow models. \n- While the image editing results seem semantically correct, there seems to be significant changes in the provided images (See Fig. 5) compared to methods such as Null-text Inversion. Despite the fact that the authors provide a user study and CLIP score based comparisons, the faithful reconstruction property is not discussed and it is unclear that how successful the provided approach is.\n- In generation results provided in Fig. 7 (fox example), the generation results seem degraded in terms of quality and provides artifacts. It also seems that RFDS in general produces these artifacts. If the alternative method RFDS-Rev recovers those, authors need to provide related discussions and examples on its performance on the performed tasks (such as inversion based editing). If this algorithm is not applicable for such a scenario, it should be mentioned as a limitation of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GCPviHfPvc", "reviewer": "Reviewer_cYSo", "summary": "This paper proposes to use rectified flow for SDS instead of diffusion models.\n\nThe propose method has three components:\n* Rectified flow distillation sampling (RFDS) = typical SDS but with rectified flow\n    * Equations are the same with SDS except having *flow residual* instead of denoising loss\n    * It uses random noises to compute the *flow residual*.\n* Optimizing the *noise* (iRFDS) with the same flow residual\n* Full algorithm (RFDS-Rev) with RFDS and iRFDS\n    * iRFDS starting from a random noise\n    * RFDS with the optimized noise\n\niRFDS and RFDS-Rev are applicable to diffusion models, especially with classifier-free guidance.", "questions": "1. Why do the bottles change across n=1 and n‎ = 2 while boots stay the same in Figure 6? The only difference I catch between boots and bottles is 2D and 3D.\n2. Why does the network Jacobian harm the results?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper tackles a long-standing problem: SDS. SDS with rectified flow is not good enough and the proposed method generates sharp results.\n2. The proposed method is easy to understand and mostly sound.\n3. The proposed method is generalizable to a wide range of flow-based methods.\n4. Preliminary is thorough enough to provide the knowledge base.\n5. Figure 2 greatly helps understanding the intuition of RFDS-Rev.\n6. Experiments are well-organized from 2D to 3D.", "weaknesses": "(Ordered by importance. Resolving them will raise my rating.)\n\n1. Subsection 3.2 should provide the theoretical justification for the reason why optimizing the noise helps RFDS.\n2. Choice of the competitors for text-based image editing is not sound because it covers only inversion variants. Answering following questions may improve soundness: Why should we compare only with inversion variants? Why prompt-to-prompt variants (e.g., DDPM inversion + P2P) should be ignored?\n3. The text-to-3D results still suffer from Janus problem. Discussion in this direction would enrich the paper.\n4. Question of the user study is ambiguous: Given the source image, \"a boat in a river\", which method is better by changing boat -> rock? There are different aspects of being \"better\", e.g., consistency except the boat, and they should be separately evaluated.\n5. Results of generated 3D assets (Figure 4) should be rendered in multiple views to be evaluated. Supp. materials have viewpoint-varying videos. Mentioning a Supp. would suffice.\n6. Introduction is verbose. It would be clearer if non-essential sentences are removed.\n\nMisc.\n\nL169 using pretrained rectified flow models v_phi -> using a pre-trained rectified flow model v_phi", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2GF1GHCOsm", "reviewer": "Reviewer_6Xia", "summary": "This paper uses the SOTA rectified-flow-based model as plug-and-play priors to lift the text-to-2D model into a text-to-3D model, with RFDS and RFDS-Rev. By removing the Jacobian of the rectified flow network, RFDS can generate meaningful images or 3D objects given text conditions. RFDS-Rev iteratively applies iRFDS for flow reversal to determine the original noise, and RFDS for knowledge distillation to refine the input.", "questions": "See weaknesses", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper analyzes the refined process of the rectified flow.\n\n2. Using the rectified flow as the priors is interesting.\n\n3. The experiments are sufficient.", "weaknesses": "* Writing needed to be improved, especially, from Lines 100-107, which is important but the logic is somewhat unclear.\n\n* The focus of the paper is a little confusing, including Image inversion, editing, and text-to-3D generation. In my view, text-to-3d must be the key contribution as it use the 2D model as the priors.\n\n* What is the difference between RFDS Loss and SDS loss. It seems that RFDS is the version of the flow-based model.\n\n* What is the speed impact of the iterative application of iRED in REDS-rev?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LhrYFCwlsG", "reviewer": "Reviewer_6eY8", "summary": "This paper presents a framework that distills knowledge from the pre-trained text-to-image rectified flow models, which serve as a strong prior and apply to various downstream tasks such as image editing and text-to-3D. Similar to ‘Score Distillation Sampling’ (SDS) loss, authors propose a novel loss termed ‘Rectified Flow Distillation Sampling’ (RFDS) that is fully compatible with Rectified Flow models and partially with Diffusion models. Furthermore, they suggest iRFDS and RFDS-Reversal, the variant of their proposed loss, that can be applied to the image inversion and RFDS-based enhancement mechanism, respectively. The authors conducted extensive experiments to validate the efficiency and efficacy of their method.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed RFDS, which poses similarity to SDS, presents an effective way to adapt pre-trained Rectified Flow models to tasks such as text-to-3D. This contribution is fundamental since there is a growing trend toward Rectified Flow models in the community.\n\n2. The authors conducted extensive experiments with various Rectified Flow/Diffusion baselines, which proves the versatility and efficacy of the proposed RFDS framework.\n\n3. The proposed method shows solid performance and faster convergence speed compared to baselines, which is noticeable.\n\n4. This paper is easy to follow and comprehend.", "weaknesses": "1. As far as my understanding is correct, RFDS-Rev redirects the gradients (velocities) of the rendered image to points close to the same data distribution mode given different noisy samples to handle common issues such as blurriness that could arise from averaging different-mode directing gradients. For this reviewer, it is unclear how iRFDS can enhance the performance of models without Reflow since these models would not have perfect straight-line paths. Approximated starting points (initial noise) would not guarantee a ‘static position’ within the noise distribution, and I think the more detailed explanation or empirical evidence would further make this work concrete. \n\n2. The results in image editing (Fig.5) do not seem convincing since semantics that are supposed to remain still are also being changed, such as the global contrast of the image, textures, and object shapes. This makes me doubtful if the proposed iRFDS can indeed optimize the valid initial noise given data. Maybe a simple reconstruction test (from optimized noise to the original data) would further provide a concrete evaluation.\n\n3. I think the explanation of faster convergence speed compared to SDS/VSD is weak since it is an important property in the field of 3D generation methods using diffusion priors. Further elaboration or evidence of this feature would help readers fully comprehend their work (e.g., comparing convergence curves with respect to the number of iterations or wall-clock time).\n\n4. Weak baseline methods. While I appreciate the paper's contribution (the first attempt to use Rectified Flow as priors) and the results are quite promising, baseline methods are rather weak (DreamFusion and VSD are quite outdated at the moment). It would be great if the authors compared against state-of-the-art text-to-3D methods, e.g., LucidDreamer [1], DreamCraft3D [2] or even GaussianDreamer [3] using different 3D representations.\n\n[1] LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching, Liang et al., CVPR 2024\n\n[2] DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior, Sun et al., ICLR 2024\n\n[3] Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models, Yi et al., CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)