Steering Rectified Flow Models in the Vector Field for Controlled Image
  Generation
Authors: name: Maitreya Patel; name: Song Wen; name: Dimitris N. Metaxas; name: Yezhou Yang
Published: 2024-11-27 19:04:40

Abstract: Diffusion models (DMs) excel in photorealism, image editing, and solving
inverse problems, aided by classifier-free guidance and image inversion
techniques. However, rectified flow models (RFMs) remain underexplored for
these tasks. Existing DM-based methods often require additional training, lack
generalization to pretrained latent models, underperform, and demand
significant computational resources due to extensive backpropagation through
ODE solvers and inversion processes. In this work, we first develop a
theoretical and empirical understanding of the vector field dynamics of RFMs in
efficiently guiding the denoising trajectory. Our findings reveal that we can
navigate the vector field in a deterministic and gradient-free manner.
Utilizing this property, we propose FlowChef, which leverages the vector field
to steer the denoising trajectory for controlled image generation tasks,
facilitated by gradient skipping. FlowChef is a unified framework for
controlled image generation that, for the first time, simultaneously addresses
classifier guidance, linear inverse problems, and image editing without the
need for extra training, inversion, or intensive backpropagation. Finally, we
perform extensive evaluations and show that FlowChef significantly outperforms
baselines in terms of performance, memory, and time requirements, achieving new
state-of-the-art results. Project Page: https://flowchef.github.io.