Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators
Authors: Daniel Geng
Andrew Owens
Venue: ICLR 2024 poster (2024)
Keywords: Optical Flow
Image Editing
Guidance
Diffusion Models
Primary area: generative models
TL;DR: We use diffusion guidance through an off-the-shelf optical flow network as a zero-shot method to edit images given a desired motion.

Abstract: Diffusion models are capable of generating impressive images conditioned on text descriptions, and extensions of these models allow users to edit images at a relatively coarse scale. However, the ability to precisely edit the layout, position, pose, and shape of objects in images with diffusion models is still difficult. To this end, we propose _motion guidance_, a zero-shot technique that allows a user to specify dense, complex motion fields that indicate where each pixel in an image should move. Motion guidance works by steering the diffusion sampling process with the gradients through an off-the-shelf optical flow network. Specifically, we design a guidance loss that encourages the sample to have the desired motion, as estimated by a flow network, while also being visually similar to the source image. By simultaneously sampling from a diffusion model and guiding the sample to have low guidance loss, we can obtain a motion-edited image. We demonstrate that our technique works on complex motions and produces high quality edits of real and generated images.

Published as a conference paper at ICLR 2024
MOTION GUIDANCE: DIFFUSION-BASED IMAGE EDIT- ING WITH DIFFERENTIABLE MOTION ESTIMATORS
Daniel Geng, Andrew Owens
University of Michigan
https://dangeng. github. oo/mot _n dance
ABSTRACT
2Xiv:2401.18088.0 [cs.CV] Jan 2024
Diffusion models are capable of generating impressive images conditioned on text descriptions, and extensions of these models allow users to edit images at a rel- atively coarse scale. However, the ability to precisely edit the layout, position, pose, and shape of objects in images with diffusion models is still difficult. To this end, we propose motion guidance, zero-shot technique that allows user specify dense, complex motion fields that indicate where each pixel in an image should move. Motion guidance works by steering the diffusion sampling process with the gradients through an off-the-shelf optical flow network. Specifically, we design a guidance loss that encourages the sample to have the desired motion, as estimated by flow network, while also being visually similar to the source im- age. By simultaneously sampling from a diffusion model and guiding the sample to have low guidance loss, we can obtain a motion-edited image. We demonstrate that our technique works on complex motions and produces high quality edits real and generated images.
INTRODUCTION
Recent advances in diffusion models have provided users with the ability to manipulate images in a variety of ways, such as by conditioning on text (Hertz al., 2022: Tumanyan et al., 2023), instructions (Brooks et al., 2023), or other images (Gal et al., 2022). Yet existing methods often struggle to make many seemingly simple changes to image structure, like moving an object a specific position or changing its shape.
emerging line of work has begun to address these problems by incorporating motion prompts, such as user-pre sse--provided displacement vectors that indicate where a given point on an object should move (Chen et al., 2023; Pan et al., 2023a; Shi et al., 2023; Mou et al., 2023; Epstein et al., 2023). However, these methods have significant limitations. First, they are largely limited to sparse motion inputs, often only allowing constraints to be placed on one point at a time. This makes it difficult to precisely capture complex motions, especially those that require moving object parts in nonrigid ways. Second, existing methods often require text inputs and place strong restrictions on the under- lying network architecture, such as by being restricted to editing objects that receive cross-attention from a text token, by requiring per-image finetuning, or by relying on features from specific layers of specific diffusion architectures.
To address these issues, we propose motion guidance, a simple, zero-shot method that takes ad- vantage of powerful, off-the-shelf motion estimators. This method allows users to edit images by specifying dense, and possibly complex, flow field indicating where each pixel should move in the edited image (Fig. 1) ). We guide the diffusion sampling process using a loss function incorporating an off-the-shel optical flow estimator, similarly to classifier guidance Dhariwal & Nichol (2021). As part of each diffusion sampling step, we estimate the motion between the generated image and the input source image, and measure the extent which it deviates from the user-provided flow field. We then augment the noise estimate with gradients through our loss, in the process backprop- agating through the optical flow estimator. Simultaneously, be visually similar to the source image by encouraging corresponding pixels to be photoconsistent, allowing model to trade off between the fidelity of motion and visual appearance.
encourage the generated image to
In contrast to other approaches, our method does not require any training, and does not require a specific diffusion network architecture. We show that our method works for both real and generated

---

Published as a conference paper at ICLR 2024
Source
Image
Motion Guided
Target Flow
Flow Legend
a) "a teapot floating in water"
b) "a photo of cat"
c) "a photo of lion"
d) "a photo of laptop"
e)
photo toplary"
"a painting lone tree"
(real image]
[real image)
Figure I: Flow Guidance. Given source image and target flow, we generate new image that has the desired flow with respect the original image. Our method is zero-shot, achieving this performing guidance through an optical flow network, and works on both of this paper were automatically selected for, and random results can be found in Appendix A3.
synthetic images. Note, qualitative results the main body
images, and that it supports wide range of complex target motions, such flow fields that have been extracted from a video. We make the following contributions:
We propose motion guidance, zero-shot technique that allows users to specify motion field to edit an image.
We show that off-the-shelf optical flow networks provide a useful guidance signal for diffusion. Through qualitative and quantitative experiments, we show that model can handle wide range complex motion fields, including compositions of translations, rotations, homographies, stretching, deformations, and even flow fields extracted from video.
RELATED WORK
Diffusion Models.
Diffusion models (Sohl-Dickstein et al., 2015; Ho al., 2020; Song et al., 2021) are powerful generative models that learn to reverse forward process where data is iteratively converted to noise. The reverse process is parameterized by neural network that estimates the noise, eø (Xe, f,y) y), given the noisy datapoint X,, the timestep t, and optionally some conditioning y, for example an embedding of a text prompt. This noise estimate then used to gradually remove the noise from the data point under various update rules. such as DDPM DDIM (Song et al., 2020).
Diffusion models admit technique called guidance, in which the denoising process is perturbed toward a desired outcome. This perturbation may come from the diffusion model itself as in classifier-free freee guidance (Ho & Salimans, 2022; Nichol et al., 2021), classifier as in ImageNet classifier guidance Dhariwal & Nichol (2021), or in general the gradients an energy function. Many forms of guidance function have been proposed including CLIP embedding distance (Wallace et al., 2023; Nichol et al., 2021). LPIPS similarity (Lee et al., 2023), bilateral filters (Gu & Davis, 2023), internal representations of diffusion models themselves (Epstein et al., 2023), and "readout heads" (Luo et al., 2023). Ho et al. (2022) propose to do guidance on a one-step approximation of the clean data, which they term reconstruction guidance, and Bansal et al. (2023) apply this
Guidance.
2

---

Published as a conference paper at ICLR 2024
"a teapot floating in water"
b)
h)
Moving and Deforming Objects. We show various motion edits on a single source image (a), demonstrating our method can handle diverse deformations including scaling and stretching. We provide a legend for flow visualization in Figure 1.
Figure 2:
idea achieve guidance on various off-the-shelf models such segmentation, detection, facial recognition, and style networks. Our method proposes using off-the-shelf optical flow models as guidance to achieve motion-bane image editing.
Image Editing. Diffusion models, due to their strong image priors, are attractive starting point for many image editing techniques. Some methods, such as SDEdit (Meng et al., 2022), propose to run the forward diffusion process partially, and then denoise the partially corrupted image. Other methods modify the denoising step as in Lugmayr et al. (2022) and Wang et al. (2023). Still others finetune diffusion model for image manipulation tasks (Brooks et al., 2023; Zhang & Agrawala, 2023). Another class of diffusion-based based image manipulation techniques uses the features and at- tention inside the denoising models themselves edit images. These techniques work due the observation that features can encode attributes, identity, style, and structure. Methods in this vein include Prompt-to-Promp (Hertz al., 2022), Plug-and-Play (Tumanyan al., 2023), and Self- Guidance (Epstein et al., 2023).
The majority of these techniques enable impressive editing of image style, but are unable to modify the image structure. A more recent line of work seeks to achieve editing of structure. These methods are closest to ours. DragGAN (Pan et al., 2023a) proposes an optimization scheme over StyleGAN features that allows users to "drag" points in an image to new locations. However, these results are constrained to narrowly trained GANs. In order to achieve much wider applicability concurrent work has adapted the DragGAN optimization scheme diffusion models (Shi et al., 2023; Mou et al., 2023), however these approaches either require fine-tuning with LoRA (Hu et al., 2021) for each image have not been shown work complex, densely defined deformations. In recent work, Epstein et al. (2023) perform motion editing with classifier guidance, using the activations and textual-visune cross- cooss-tttention of the diffusion model specify object position and shape. This limits the complexity of deformations and constrains manipulable objects to those that have a corre- sponding textual token in the prompt. In contrast, our guidance comes from off-the-shelf optical flow network and thus does not suffer from these problems, nor does it rely on particular architec- ture for the diffusion model. Overall our method seeks build upon past and concurrent work, by enabling dense, complex, pixel-level motion manipulation for a wide range of images in a zero-shot manner.
METHOD
Our goal is to allow user to edit source image x" Rwxxx3 by specifying target flow field f xxxx2 that indicates how each pixel should move. To do this, we develop techniques that guide diffusion model's denoising process using an off-the-shelf optical flow network.
3

---

Published as a conference paper at ICLR 2024
3.1 GUIDANCE IN DIFFUSION MODELS
Diffusion models (Sohl-Dickstein et al., 2015; Song et al., 2021) generate images by repeatedly denoising sample of random noise. They are commonly represented (Ho et al., 2020) as functions that predict the noise in a noisy data point x, at step I of the diffusion process, using an optional conditioning signal y. Classifier guidance (Dhariwal & Nichol, 2021) seeks generate samples that minimize a function L(x), such as an object classification loss, by augmenting the noise estimate with the gradients of L:
eo
(, ex(x;;4,)
Eo(x;; L, v) = c0(xx; t,, V,,x,
(1)
where Eg is the new noise function and or is a weighting schedule. A key benefit of this approach is that it does not require retraining, and it does not explicitly place restrictions L, except for differentiability. Recent work has shown that this approach can be straightforwardly extended to use functions I that accept only clean (denoised) images input (Bansal et al., 2023).
3.2 MOTION GUIDANCE
We will use guidance to manipulate the positions and shapes of objects in images. We design a guidance function L(x) (Eq. 1) that measures how well generated image, x, captures the desired motion. We encourage the optical flow between x" and x to be f. Given a (differentiable) off-the- shelf optical flow estimator F(.-)), .). we will minimize the loss
(2) While we found that performing guidance on this loss could produce the desired motions, we also observed that the edited objects would often change color or texture. This is because motion estima- tion models are invariant to many color variations', thus there is little to constrain the appearance of the object to match that of the source image. Inspired by Pan et al. (2023b) and Geng et al. (2022), to address this we add a loss that encourages corresponding pixels to have similar colors
L (x) [ - fll
Lealor (x) ||x warp(x, F(x", x))]I
where warp(x, f) indicates a backward warp of an image x using the displacement field f. The full loss we use for guidance is then
L(x) = Aow Lnw (x) + moooo (x),
(4)
where Anow and Asolor are hyperparameters that trade off between the model's adherence to the tar- get motion and the visual fidelity to objects in the source image, and Meoler is mask to handle occlusions, explained below.
3.3 IMPLEMENTING MOTION GUIDANCE
We find a number of techniques useful for producing high quality edits with motion guidance.
Handling Occlusions. When objects move, they occlude pixels in the background. Since these background pixels have no correspondence in the generated image, the color loss (Eq. 3) is coun- terproductive. Similar to Li et al. (2023), we adopt the convention that the target flow specifies the motions of the foreground. i.e., that when an object moves, it will occlude the pixels that it overlaps with. To implement this, we create an occlusion mask from the target flow and mask out the color loss the occluded regions. Please see Appendix A1 for details.
Edit Mask. For given target flow, often many pixels will not move at all. In these cases, we can reuse content from the source image by automatically constructing an edit mask m from the target flow, indicating which pixels require editing. This mask consists of all locations that any pixel is moving to or from. To apply the edit mask during diffusion sampling, we assume access to a sequence of noisy versions of the source image x" for each timestep, which denote x²- For real images this sequence can be constructed by injecting the appropriate amount of noise into x" at each timestep t, or alternatively can be obtained through DDIM inversion (Song et al., 2020). For images sampled from the diffusion model, we can also cache the x; from the reverse process. Then, similarly Song et al. (2021) and Lugmayr al. (2022), during our denoising process at each timestep & we replace pixels outside of the edit mask with the cached content: x, +
m O X + (1 m) x
'This quite useful for flow estimation videos, where lighting may change rapidly frame frame.
4

---

Published as a conference paper at ICLR 2024
Handling Noisy Images. In standard classifier guidance, noisy images x, are passed into the guidance function. In our case, this results in a distribution mismatch with the off-the-shelf optical flow model, which has only been trained on clean images. To address this we adopt the technique of Bansal et al. (2023) and Ho et al. (2022), and compute the guidance function on a one step approximation of the clean Xo given by
Xt
VI agca (Xe: t)
X0
(5)
v 0tt
resulting in the gradient Vx, (x)) with a more in-domain input the guidance function.
Recursive Denoising. Another problem we encounter is that the optical flow model, due to its complexity and size, can be quite hard to optimize through guidance. Previous work (Lugmayr et al., 2022; Wang et al., 2023; Bansal et al., 2023) has shown that repeating each denoising step for a total of K steps, where K is some hyperparameter, can result in much better convergence. We adopt this recursive denoising technique to stabilize our method and empirically find that this resolves many optimization instabilities, in part due an iterative refinement effect (Lugmayr al., 2022) and also in part due to the additional steps that the guidance process takes, effectively resulting more optimization steps over the guidance energy.
Guidance Clipping. We also find that clipping guidance gradients prevents instabilities during denoising. Concurrent work by Gu & Davis (2023) introduces an adaptive clipping strategy, but find that simply clipping gradients by their norm to a pre-set threshold, cy. works well in our case.
4 RESULTS
We evaluate our method's ability to manipulate image structure, both qualitatively and quantitatively, on real and generated images. Additional results can be found in Appendix A3.
4.1 IMPLEMENTATION DETAILS
We use RAFT (Teed & Deng, 2020) our flow model. To create target flow fields, we com- pose a set of elementary flows and use segmentation model (Kirillov et al., 2023) for masking. Target flow construction and hyperparameters are discussed in detail in Appendix Al. We addi- tionally implement a simple GUI to generate various flow fields, which we describe in Appendix Figure Al and A2. For our experiments we use Stable Diffusion (Rombach al., 2021). Rather than performing diffusion directly on pixels, Stable Diffusion performs diffusion in latent space, with an encoder and decoder to convert between pixel and latent space. To accommodate this, we precompose the decoder with the motion guidance function, L(D(-)), so that the guidance function can accept latent codes. Additionally, we downsample our edit mask to 64 x 64, the spatial size of the Stable Diffusion latent space.
4.2 QUALITATIVE RESULTS
Main qualitative results can be found in Figures 2,2,3, and 4. Following previous work (Ramesh et al., 2021; Bansal et al., 2023) we generate multiple samples per example and choose the best result according to the guidance energy. Details can be found Sec. AI, and random and top-5 qualitative results can be found in Sec. A3. We point out several strengths of our method:
Disocelusions. Because our method is built on top of powerful diffusion model, it is able to reason about disoccluded regions. For example, in Figure Ic our model moves the lion to the left and is able to fill in the previously occluded face.
Diverse Flows. Our method successfully manipulates images using translations (Figures 1c, Ih, 3a ), rotations (Figures 1b, 4a ), stretches (Figures 1a, 1f, 1g and scaling (Figures 2e, 2f, 4b). ). It also handles complex deformations (Figures le, 3c, 4c) ), homographies (Figures 1d, 3b), and mul- tiple objects (Figures 4e, Ie). Moreover, our method is able handle relatively large movements (Figures 2b, 2c, 2d, 3a) ). We are able to achieve successes on diverse flows due to the generality of both the optical flow and diffusion model we use.
5

---

Published as a conference paper at ICLR 2024
Diverse Images. We show that our approach works wide range of input images with diverse objects and backgrounds, including non- pon-poooorcaaistic images such as paintings or sketches (Figures 1f, 4b, 5a, 5b, 5c ), despite the fact that the optical flow network We attribute this success due to the fact that optical flow is relatively low level signal, and does not require understanding of high level semantics estimate. Additionally, show that method works for both synthetic images as well as real images (Figures Ig, Ih, 3c, 4e, 7b, 7c, 7d).
trained on these styles.
Soft Optimization. In contrast to forward warping, our method optimizes soft constraint: satisfy the flow while also producing plausible image. Because of this, our method works even for coarsely defined target flows as shown Figures Ic, 1h, 5b. This is particularly useful when we do motion transfer in Section 4.6 and Figure 7, where the extracted flow is only roughly aligned with the source image. Another benefit soft optimization is that our method works even when target flows ambiguous, such as when flows specify that two objects should overlap in Figure le.
Text Conditioning. Because our method is based on guidance, it is independent of text condi- tioning. This useful for editing real images, for which no caption exists. In all figures, samples conditioned on text will have the prompt written in italics below the example, and real images are labeled as "[real image]" and are unconditionally generated.
4.3 ABLATIONS
In Figure 3 we qualitatively ablate out key components of our guidance function.
No Recursive Denoising. Without recursive denoising our method converges much less fre- quently, often resulting in samples with heavy artifacts as in Figure 3c. Even in cases where the guidance mostly succeeds, artifacts can still occur as in Figure 3a.
No Color Loss. Without the color loss, method generally moves objects to the correct loca- tions, but objects tend to change color as Figure 3a. These samples still achieve a low flow loss because of the color invariance of the optical flow network, highlighting the need for the color loss. In some cases we see catastrophic failure, as in Figure 3c
No Flow Loss. Removing the flow loss also removes any knowledge the method has of the target flow. To address this also modify the color loss (Eq. 3), ), replacing the computed flow with the target flow. Without the flow loss our method able to move objects to the correct location but often ballucinates things in disoccluded areas, such the half apple in Figure 3a or the waterfall in Figure 3c. This is because the color loss does not produce any gradient signal in disoccluded areas due the warping operation. In the disoccluded areas the diffusion model effectively acts freely with no guidance at all, resulting in these hallucinations.
Source Image
Target Flow
No Recuraive Dencising
No Color Lass
Flow Loes
No Occlasion Mask
Oura Motion Guided
"an Apple en wooden lable"
photo o lyptop"
Seal omagel
Figure 3: Ablations. We qualitatively ablate out techniques we use to achieve motion guidance. For a discus- sion please see Section 4.3. 3. We provide legend for the flow visualization in Figure I.

---

Published as a conference paper at ICLR 2024
No Occlusion Masking. Without occlusion masking the color loss is incorrect in occluded areas. This is because the target flow in occluded areas is invalid, so the warping operation mismatches pixels. As result we tend ground or take on the color of the disoceluded background, as can be seen in Figure 3a or Figure 3c. In more extreme cases, the object fail to into the occluding region at all, as in Figure 3b.
"ghosting" of moved objects, where objects blend into the back-
4.4 BASELINES
We present comparisons between our method and baselines in Figure 4. For more extensive set of comparisons please see Figure A7 in the appendix.
InstructPix2Pix. InstructPix2Pix (Brooks et al., 2023) distills Prompt-to-Prompt (Hertz et al., 2022) into diffusion model that is conditioned on textual editing instructions. Because this model only lext-conditioned, it is not possible to give it access to the target flow, but we try to faithfully summarize the flow in an instruction. As can be seen, despite our best efforts, InstructPix2Pix never able move objects significantly although it does occasionally alter the image in rather unpredictable ways. This failure can be attributed to two factors. Firstly, text is not an ideal method for describing motion. For example it is quite hard encapsulate the motion in Figure 4e in text without being excessively verbose. And secondly, feature-copying methods such as Prompt-to- Prompt often fail make significant structural edits images.
Forward Warp with SDEdit. We also use baselines that explicitly use the target flow. Specif- ically, we forward warp the latent code by the target flow and use SDEdit (Meng et al., 2022), conditioned on the text prompt (if given) at various noise levels "clean" the warped image. While SDEdit succeeds to degree in Figure 4b and Figure 4d, the results are of lower quality and the failure cases contain severe errors.
Forward Warp with RePaint. In addition to cleaning the forward warped images using SDEdit, try the diffusion based inpainting method RePaint (Lugmayr al., 2022), conditioned the text prompt (if given), to fill in disoccluded areas in the forward warp. The inpainted areas are generally realistic looking, but often not exactly correct. For example in Figure 4d, additional teapot is generated in the disoccluded area.
Source Image
Target Flow
InstrussPidPPi
PW Sacde 75%
FW
SOGGR
RePaint
Dure Modion Guidbonc
peleiing of surfoan
Teato the suctbwer NaNT Degee
"orve
Pou veega
Trooes the Whrdow doadyy under 1he laro.."
Figure 4: Baselines. We show qualitative examples from various baselines and our method. The instruction used for InstructPix22Pxx is shown beneath each InstructPix2Pix sample. For discussion please see Section 4,4. We provide a legend for the flow visualization in Figure 1.

---

Published as a conference paper at ICLR 2024
Source Image
Target Flow
GAN Inversion
DragGAN
Ours Motion Guidance
a)
orayon drawing of an elephant"
b)
loo"
"a sketch of man face a tophar" Figure 5: Comparison DragGAN. DragGAN works only domains for which StyleGAN has been trained on. Attempts to edit real images that are out-of-domain, even they are invertible, results failures that our model handles well. Here we show results on StyleGANs trained for (a) elephants (b) lions (c) faces. We provide legend for the flow visualization in Figure 1.
DragGAN. Recent work has proposed an iterative two-step optimization procedure over GAN features to "drag" user specified handle points to target points. While the method, DragGAN (Pan et al., 2023a), is impressive has only been demonstrated on StyleGANs (Karras et al., 2020) trained narrow domains, such dogs, lions, human faces, limiting the scope of such method. To demonstrate this, we show out-of-domain cases in which DragGAN fails in Figure 5, Following DragGAN, invert image StyleGAN latent code we PTI (Roich al., 2021). We then subsample the target flow to obtain handle and target points that are required by DragGAN. Drag- GAN GPU memory usage scales linearly with the number of handle points, so densely sampling the flow is not feasible, even on an NVIDIA A40 GPU with 48 GB of memory. DragGAN also supports an edit mask, so we provide it with the same automatically generated edit mask that our method uses. We find that while the GAN inversion is successful even on out-of-domain cases, DragGAN motion edits contain many artifacts.
4.5 QUANTITATIVE RESULTS
We are interested not just optimizing for specific target flow, but also generating images that are faithful to the source image and that are coherent. Therefore we evaluate our method on two metries which are designed to reflect the trade-offss we are interested in. One metric the Flow Loss, Lnow from Equation 2, which should measure how accurately generated image adheres to the target flow. Because our guidance function actually uses Laow with RAFT, we present results using both RAFT and GMFlow (Xu al., 2022) in case we compute the CLIP Similarity as the cosine distance between the CLIP visual embeddings the source and generated images. We treat this measure of faithfulness to the source image well as measure of general image quality.
overfitting. For second metric
(a) KITI (RAFT)
180 KITI (GMFlow)
(c) Ours (RAFT)
(4) Ours (GMFlow)
1.2
1.2
10
InstructPu2Rx
Flow loss (CMPPPM owr
InstructPrMMM
Flow Loss (RAT)
PW-SDEOR
Flow Loss (GMPIOW)
PW SOBAR 75% 0
How Loss (0011))
PW- Strah 71%
PW-50EdF:
PA SDEsh 75%
0.9
Dura
FW-SDDEP 50%s
FW+ 25%
SW++SD 50%
PV-50EdR: 25%
PW-SDDdd 25%
0.7
FW'RRePaint
PW RaPaant
Fi RaPu
WWW-Rthuntt
950
O 955
O 960
950
960 CUP Visual Similarity
D 955
97 CUP Visual Similarity
95
96 D 97
CLIP Visual Simbarity
CUP Visual Similarity
Figure 6: Quantitative Metrics. We show performance of baselines and our method on the Flow Loss and CLIP Similarity metrics, two different datascts, Our curated datasct and automatically generated KITTI dataset. Please see Section 4.5 for further discussion.
8

---

Published as a conference paper at ICLR 2024
We evaluate two different datasets. The first dataset is composed of examples with handcrafted target flows, a subset of which can be seen in Figures 1, 2, 3, 4, and 7. This dataset has the advantage of containing interesting motions that are of practical interest. In addition, highly specific instructions for the InstructPix2Pix baseline for a fair comparison. However, this extent. We ameliorate this by performing an additional evaluation automatically generated dataset based on KITTI (Geiger et al., 2012), which contains egocentric driving videos with labeled bounding boxes cars. To build our dataset we construct target flows consisting of random translations cars. For more details on the datasets please see Appendix A2.
write
dataset eurated
We show results in Figure 6. As can be seen, our method offers an attractive trade-off between satisfying the target flow and keeping faithful to the source image. The RePaint baseline, which is essentially forward warping and then inpainting, achieves low flow loss due to the explicit warp operation but suffers from artifacts due to aliasing and inpainting and thereby results in low CLIP similarity. The SDEdit baseline, which also uses forward warping, can achieve low flow loss if the noise level is low, but also falls short on the visual similarity metric.
4.6 MOTION TRANSFER
Sometimes is hard to specify a flow by hand. In these cases we find that our method can success- fully "transfer" motion from a video to an image. We give examples in Figure 7, where we extract the flow from a video of the earth rotating and use as our target flow. We find that even the extracted flow does not overlap perfectly with the target image the desired motion can be achieved. We attribute this to the fact that our guidance is soft optimization and the diffusion model overall ensures high-level semantic consistency.
freat image]
Motion Edited
frest image)
Motion Edited
freal image)
Motion Edited
Figure 7: Motion Transfer. Our method can use the motion extracted from a source video to motion-cdit completely different image. Note we use fewer recursive denoising steps for these samples than for other samples because found it to improve results slightly. We provide legend for the flow visualization in Figure 1.
DISCUSSION
Limitations. While our method can produce high quality motion-condin conditiond edits, is also sus- ceptible to various weaknesses. In particular, inherit the deficiencies of diffusion models and guidance based methods, such as slow sampling speed. The use of universal guidance also intro- duces instability in the sampling process, which is only partially addressed by the recursive denois- ing strategy (Sec. 3.3). In addition, we also inherit the limitations of our optical flow method, and find that certain target flows are not possible. We give comprehensive discussion of our limitations in Appendix A4.
5.1 CONCLUSION
Our results suggest that we can manipulate the positions and shapes of objects in images by guiding the diffusion process using an optical flow estimator. Our proposed method is simple, does not require text rely work as a step in two research directions. The first direction is in integrating differentiable motion estimation models into image manipulation models. Second, work opens the possibility of repurposing other low-level computer vision models for image generation tasks through diffusion guidance.
particular architecture, and does not require any training. We see our

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This project was supported by a Sony Research Award. Daniel Geng is supported by National Science Foundation Graduate Research Fellowship under Grant No. 1841052. Daniel would also like to thank Berlin for wonderful summer, during which this research was conducted.
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models, 2023. 2, 4, 5, 13, 20
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023. 3,
Shien Chen, Chieh Hubert Lin, Hung- Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304.4400x 2023.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Neural Information Processing Systems, 2021. 1,
A. Dosovitskiy, P. Fischer, E. 1lg. P. Häusser, C. Hazırba,, V. Golkov, v.d. Smagt, D. Cremers, and TT Brox. Flownet: Learning optical flow with convolutional networks. In IEEE Interna- tional Conference on Computer Vision (ICCV), 2015. URL http://indh informatik.
ceil-frebuurg dee/Fublication
22///011/// 20015/DFIC IB15. 13
Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. Neural Information Processing Systems (NeurIPS), 2023. 1, 2, 3, 13
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 2rXiv:2208.01618 2022.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference Computer Vision and Pattern Recognition (CVPR), 2012. 9
Daniel Geng, Max Hamilton, and Andrew Owens. Comparing correspondences: Video prediction with correspondence-vine losses. In CVPR, 2022. 4
Zeqi Gu and Abe Davis. Filtered-guided diffusion: Fast filter guidance for black-box diffusion models, 2023. 5,
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. coompt-to-prompe image editing with cross attention control. 2022. 1, 3, 21
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint
arxiv:2006. /239, 2020. 2,
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.0m 03458, 2022. 2,5
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685. 2021.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
10

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXhv:2304.026433 2023. 5, 13
Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffusions. In Teirth-seventh Conference on Neural Information Processing Systems, 2023. 2
Wei Li, Xue Xu, Xinyan Xiao, Jiachen Liu, Hu Yang, Guohao Li, Zhanpeng Wang, Zhifan Feng, Qiaoqiao She, Yajuan Lyu, and Hua Wu. Upainting: Unified text-to-image diffusion generation with modsl guidance, 2022. 13
Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics.
arXiv preprint arXiv:2309.07906, 2023. 4, 13
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models, 2022. 4, 5,
Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, and Aleksander Holynski. Readout guidance: Learning control from diffusion features. arXiv preprini arXiv:2312.021555 2023. 2
N. Mayer, E. Ilg. P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estima- tion. In IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2016. URL http:// /mtp../mm.///.../////////m/// Anformatik. uni wwriifreebuurr... de wwweee/allaatioss///eeeeeeeeeeee MIFDB16 6. 112iv:151..02134. 13
.de
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In Interna- tional Conference on Learning Representations, 2022. 3,7
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models, 2023. 1,3
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2021.
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image mani- fold. arXiv preprint arXiv:2305.1097. 2023a. 1, 3, 8
Zhaoying Pan, Daniel Geng, and Andrew Owens. Self-supervised motion magnification by back- propagating through optical flow. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL httpt:tt///// //tr///aavvvaa////a org/ abs/ 2211. 17056. 4
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot texl-to-image generation. In International Conference on Machine Learning. pp. 8821-8831 PMLR, 2021. 5,
Daniel Roich, Ron Mokady, Amit Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Trans. Graph., 2021. 8
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffu- sion models with deep language understanding, 2022. 13
Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdif- fusion: Harnessing diffusion models for interactive paint-based image editing. arXiv preprint arXiv:2306. 14435, 2023. 1,3
11

---

Published as a conference paper at ICLR 2024
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.). Pro- ceedings of the 32nd International Conference Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL. html.
ptttps://proceeding/ ml presss
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502,, October 2020. URL https: https::/aaxiv..rrettee http/v.xivvv..o/../...a wwtpi..rgg/ab///1110000 02502. 2.4
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Interna- tional Conference Learning Representationss 2021. URL https: https::/epeerrvieee net/
2,
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pp. 402-419. Springer, 2020. 5
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-ple diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Patlern Recognition (CVPR), pp. 1921-1930, June 2023. 1,3
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent opti- mization improves classifier guidance, 2023. 2
Yinhuai Wang, Jiwen Yu, and Jian Zhang, Zero-shot image restoration using denoising diffusion null-space model. The Eleventh International Conference on Learning Representations, 2023. 3,
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8121-8130, 2022. 8
Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 3
12

---

Published as a conference paper at ICLR 2024
A1 IMPLEMENTATION DETAILS
Hyperparameters We use Stable Diffusion v1.4 with a DDIM sampler for 500 steps, and we generate images resolution of 512 x 512. All experiments are conducted on a single NVIDIA A40 GPU. For our motion guidance function (Eq. 4) we found that setting Acolor to 100 and Arow to 3 worked well. In addition, in our implementation we scale the guidance gradients by a global weight of 300. We set the gradient clipping threshold Cg to be 200 and take K 10 recursive denoising steps. For just the motion transfer results we set K = 10, 5, 2 for the cat, pug, and panda respectively, For the RAFT optical flow model use the public checkpoint trained on FlyingChairs (Dosovitskiy et al., 2015) and DrrinngThingg3D.. (Mayer et al., 2016) with 5 iterative updates.
Guidance Schedule Previous work (Saharia et al., 2022; Epstein et al., 2023; Li et al., 2022) has shown that sample quality can improve with well-chosen guidance schedules, that is, applying guidance with varying strength through out denoising. We find that applying guidance as normal for the first 400 steps and turning it completely off for the final 100 steps is beneficial in our case and we use this schedule for all samples. Intuitively, in the final 100 steps the low-level structure of the image is already sampled and little can be done to alter the motion of the sample. Turning off guidance during this phase allows the model to focus on synthesizing realistic, high quality details.
Occlusion Masking In order to create an occlusion mask we need additional information or as- sumptions beyond just the flow. For example, depth could indicate to us which objects would oc- clude others. Similar to Li et al. (2023), in the absence of this information we make the simplifying assumption that the target flow is applied a foreground object. Then in order to construct an occlu- sion mask we find all background regions that will be overwritten by the target flow by performing a warp and add these pixels to the occlusion mask, which is used to mask the color loss.
Reranking Following previous work (Ramesh al., 2021: Bansal et al., 2023) we generate multi- plc samples and automatically filter these samples. Specifically, we generate 32 samples per example and re-rank based on the final motion guidance loss value. We find that re-ranking particularly useful for filtering out poorly converged samples, which we discuss in more detail in Appendix A4. We present both top-5 and random samples in Appendix A3. Note we do not use re-ranking for our quantitative evaluations in Section 4.5.
Target Flow Construction We craft target optical flows by composing translations, rotations, scaling, stretching, and shearing. For more complex flows, such as with the river or the topiary samples, we interpolate between discrete displacements with sine functions. For homographies, as in the laptop example, we define starting and ending locations for corners and fit a homography H to these points. We then extract a flow by computing Hx x, the displacement for each point under the homography.
In most cases mask out the flow using mask extracted using SAM (Kirillov al., 2023). This mask is dilated by a few pixels by applying an all ones convolution, equivalent to a box blur, and then thresholding. We find this helpful because SAM often slightly under estimates the extent of an object. All flow outside of the dilated mask is set to zero. We do not mask for the motion transfer samples.
For ease of use, we create a simple GUI that combines the segmentation step and the generation of optical flows. Our GUI allows users create translations, rotations, stretches, scaling, and complex deformations simply by clicking a dragging. For more details and examples, see Figures Al and A2.
A2 DATASET DETAILS
For the dataset based on KITTI we use frames with bounding box car annotations. We first filter out frames with no cars, frames with cars that are too small or too large, and frames with occluded cars. Occlusion information is annotated by the dataset. For each frame we choose car and crop the frame to that car, and resize to obtain a 512 x 512 image. We then construct a flow representing translation of the bounding box, with uniformly random direction and magnitude uniformly sampled between 10 and 50 pixels. We use a total of 226 examples. For the InstructPix2Pix baseline automatically construct one of sentences such as "move the car to the left" or "move the car to the right and down." We visualize results of motion guidance this dataset in Figure A3.
instruction prompt by mapping the randomly sampled direction to
13

---

Published as a conference paper at ICLR 2024
Original Image
SAM Masking
GUI Flow
Motion Edited
Figure A1: Overview of GUL Our GUT allows with image, mask the objects they motion-edit, construct an optical flow be used with our method.
Flow Legend
Translation
Rotation
Stretching
Scaling
Interpolated Stretching
Figure A2: Implemented GUIs. Our GUI supports the synthesis of optical flows by clicking and dragging control arrows. We describe the interface for each of the above examples. Translation: Global translations parameterized by the direction and magnitude of the arrow dragged out by the user. Rotation: The initial click defines the center of rotation. The angle of rotation cursor away from the center. Stretching: The initial click determines the center of stretching. The dragged arrow determines the angle at which to stretch, and the factor hy which to stretch. The notch on the anoow indicates unity, where the stretching factor is equal to 1. Dragging indicates squeezing, and dragging arrow away from the notch and center indicates squeezing. Scaling: The center scaling determined by the initial click of the user. the arrow's ss distance from the center of scaling. The circle indicates unity, where the scaling factor is equal to 1. Dragging arrow inside the circle equates to shrinking. and dragging an arrow outside of the circle equates expanding. Interpolated strefching: We take the "stretching" UI and allow arrows, each defining stretch (or squecze). Treating these arrows "control points," we can interpolate between these stretches and squeezes to synthesize complex deformations.
parameterized by
distance user drags the
arrow between the center and the notch
factor by which to scale determined by
to draw multiple
14

---

Published as conference paper at ICLR 2024
Target Flow
Image
Motion Guided
Figure A3: KITTI Dataset Examples. We show examples of motion guided edits on the automatically gen- eraled KITTI dataset. Two failure cases are shown the very right. One in which the car is distorted rather than moved the railroad tracks, and one which the identity of the changes to satisfy the target flow. Because exact motion can difficult interpret from the flow visualization alone, we also draw target location of the bounding box in the top row.
Source Image
Target Flow
Ours Motion Guided
Source Image
Target Flow
Ours - Motion Guided
"a photo a humenoid sold background"
(nea image)
painting
pfrooo
"a photo of modern house"
photo of leptop"
Figure A4: Additional examples. We present additional results using motion guidance method. These include target flows that (a) rotations (b) stretches scaling (d, e) translations and (f) homographies.
For our curated dataset we sample images from Stable Diffusion using custom prompts and collect real images from Unsplash. We then handcraft interesting target flows for each image. For the InstructPix2Pix baselines we then manually annotate an instruction describing the motion the target flow should apply to the source image. In addition, for our evaluations we further augment the dataset by scaling each target flow by total of factors, with the aim of covering a wide range of flow magnitudes. This results in 204 examples.
A3 ADDITIONAL RESULTS
We show examples of motion guidance in Figure A4. In addition, show multiple samples for source image and target flow pairs with both ranked sampling (see Appendix Al for details) qualitative examples using
in Figure A5 and random sampling in Figure A6. We also show baseline methods and our method in Figure A7. Finally, we show samples and their flow with respect the source image over the course of the denoising process in Figure A8.
15

---

Published as a conference paper at ICLR 2024
Source Image
Target Flow
Top-5 Samples
photo lpptop"
photo of lio""
"a psinting of sunfooerr
to toapor flaating
to flatiig
"a teapot flosing water"
photo toplary"
painting
freal imagei
(raal imagej
imagei
Figure A5: Top 5 Ranked Samples. We show a selection of the top 5 samples after reranking with our guidance loss, from set of 32 samples. The samples are ordered left to right, better to worse ranked. For a random set of samples using the same images and target flows please Figure A6.
16

---

Published as a conference paper at ICLR 2024
Source Image
Target Flow
Random Samples
photo lpptop"
photo of lao""
"a psinting of sunfooerr
to toapor flaaiing
to flatiig
"a teapot flosing water"
photo toplary
painting
freal imagei
(raal imagej
imagei
Figure A6: Random Samples. We show a random set 5 samples for each image and target flow pair. These samples are of slightly lower quality than the re-ranked examples from Figure A5, as can be seen in the teapot handle, the texture of the topiary bush, the pug. or the painting of tree.
17

---

Published as a conference paper at ICLR 2024
Source Torget Flow
InstrusPha2P
FW 3Demh 78%
PW 60Eal 60%5
FW 90eaii 20%
FW RuPait
Oura Motio Ourdance
tapfooo of moson house"
"aato the roof bie touse"
c
photo a upptpp"
Tolco Sie tepine kiy
di
"wyghwn Roe
Jaaoe! Soublig it waleo"
Trroe the Iapoo lyt"
N
photo
Figure A7: Additional Baseline Images. We qualitatively compare haselines to our method on additional examples. For further discussion please see Section 4.4.
18

---

Published as a conference paper at ICLR 2024
Source Image/ Target Flow
400
900
100
Figure A8: Denoising Samples and Flows. We show samples through the denoising process, along with the computed flows with respect clean data point at timestep f, as well as its flow with respect to source image as estimated by RAFT. Note that hecause our schedule removes guidance for the final 100 steps, which we found improved results, the final flow for t 0 is not similar to the target flow previous timesteps.
source image. Specifically, we show the one-step approximation
19

---

Published as a conference paper at ICLR 2024
Source Image
Target Flow
Ours Motion Guidance
s)
Guidance
Wky whes on brick wel
-40
Cuidance Endryy
-100
-100
fusai image)
-160
poorly converged
100
200
300
400
500
Timestep
+)
Guidence Fallure Cases
ta photo of Von"
Figure A9: Failure Cases. We show various failure cases of our method. (a) Our method cannot handle out-of-distribtioo larget flows, such as a vertical flip. (b) In some cases method alters the content of the source image, such the surfboard and the removal of the right leg of the surfer. (c) Guidance unstable and randomly result in poorly converged images, such as lion example. In (d) we plot guidance energies over the denoising process for the sample in (c) in brown, along with five other samples that successfully converged. The dip timestep 400 is the result of our guidance schedule.
sometimes
A4 LIMITATIONS AND FAILURE CASES
Our method can effectively apply motion-based manipulations both real and synthesized images for a wide range of target optical flows, but it also has limitations, which we discuss here.
Target Flows Our method works for wide range of flows, as demonstrated in Fig- ures 2, 3, 4, 5, 7, and A4, such as compositions of translations, rotations, stretches, shears, homographies, and deformations. However, we also found that there are some target flows that do not produce good results. For example, we could not get horizontal vertical flips work. We hypothesize this is because these target flows out-of-domain for off-the-shelf optical flow networks. We give an example in which we try to optimize a vertical flip target flow Figure A9a.
Diffusion and Guidance Because our model is based on diffusion guidance we inherit the draw- backs of diffusion models and guidance. For one, method is slow to sample from. Much of our overhead comes from recursive denoising, which scales linearly with sampling time. Depending on the number of recursive denoising steps, take 15 minutes in our best case with 2 recursive steps and in the worst we take around 70 minutes with 10 recursive steps, which is on par with previous work (Bansal al., 2023). In addition we find that guidance is sometimes unstable, which exac- erbated by the large and complex optical flow models we are backpropagating through. We show an example of guidance failure in Figure A9c and show the corresponding guidance energy over the reverse diffusion process in Figure A9d, along with the energy plots for successful samples. We hypothesize this instability comes from trade-off on guidance strength. large guidance strength results in a lower guidance loss, but also increases the magnitude of the perturbation, and thereby the chances that sampling may diverge.
Forgetting and Identity Our method is also prone to "forgetting" the content of the source image. Minor cases of this can be seen in Figure A4f, in which the content of the laptop screen changes, or in Figure which the reflection in the window slightly changes. This problem can be exacerbated by real images for which we have caption we condition on, shown in Figure A9b.
20

---

Published as a conference paper at ICLR 2024
Source Image
Text-Base Text-Based Edits
a)
crachet crsyon diawiing
b)
Jraal Imagei
painting painting horse style
Figure A10: Text-Conditioned Image-to- Image Translation. Setting the target flow to zero and applying motion guidance while using a new text prompt can enable text-based edits while keeping the structure of the source image.
image)
Image)
Warp
Freal image)
Motion Edited
Forward Warp
Figure A11: Motion Transfer Compared Forward Warp. We the misalignments between target flow and the source image hy forward warping.
A5 ADDITIONAL APPLICATIONS
Text Conditioned Image Image Translation. Setting the target flow zero everywhere allows us to modify an image while keeping its high level structure by providing an appropriate prompt, as shown in Figure A10. We note that this is not simply setting the weight on the flow loss to be zero. Rather, we are constraining the sample to have zero flow with respect to source image. This allows to produce quite different images that share the same structure as the source image, in the "line drawing" in Figure Al0a. While we produce high quality results, find that our technique is not as robust as other existing methods for text conditioned image-to- image translation such network must compute the optical flow between two images of different styles, which is typically quite nut-of-domain for the model. We hope that future progress in motion estimators may improve the robustness of this technique.
Soomp-tto-Promp
(Hertz et al., 2022). We believe this is because the flow
A6 ADDITIONAL MOTION TRANSFER ANALYSIS
Motion transfer is particularly difficult because the flow does not align with the source image very well. We demonstrate this by forward warping the source images by the target flow from Figure Al1. Our results do not look like the forward warped results because optimize soft objective: produce a likely image from the data distribution learned by the diffusion model, that simultancously has low guidance energy.
21

---

Published as a conference paper at ICLR 2024
f= 0.1
f 0.2
f= 0.3
f= 0.4
f= 0.5
f= 0.6
0.7
0.8
f= 0.9
f= 1.0
Figure A12: Motion Guidance on Scaled Flows We show motion guidance on the waterfall example from Figure 3, but we scale the flow by factor of f for each image.
A7 VIDEOS FROM MOTION GUIDANCE
We show sequence of motion edits the waterfall example from Figure 3. Each sample uses the same target flow, but scaled by some factor f. These demonstrate our model working for more fine-grained flow than the large motions presented in the main body of the paper. In addition, this indicates the possibility of using our method to construct videos. However, we note that because the images sampled independently, there is no coherence how our method fills in disocclusions, resulting in slightly choppy video. We leave this problem for future work.
22


Reviews:
[{"review_id": "v1kQQjQuCA", "reviewer": "Reviewer_wQXZ", "summary": "This paper proposes a way to edit images using by applying a desired motion field. This is done by guiding a pretrained diffusion model at inference time according to a user-specified advection field. The approach is similar to classifier guidance---a differentiable guidance loss utilizes an off-the-shelf flow prediction deep network, encouraging the flow from the source image to the generated image to resemble the target flow at each diffusion step. The authors compare to several GAN-- and diffusion--based approaches.", "questions": "What is the intuition for why the edit mask is necessary to make this approach work? Shouldn't the fact that the flow field is identity on the unchanged regions be sufficient?\n\nI would be curious to see what the one step approximation derived by equation (5) looks like as well as the results of passing it through the flow estimation model rather than the final denoised image. Is it a sufficiently good approximation and not out of distribution?\n\nDoes using a pretrained neural flow prediction model offer a significant  advantage over differentiating through an actual optical flow algorithm? For instance, works like \"Supervision-by-Registration\" [Dong et al. 2018] have done this for other tasks---I'm wondering how well a similar approach would work here.\n\n---------\n\nPost-rebuttal:\n\nThank you to the authors for their clarifications. I did not realize that [Shi et al. 2023] and [Mou et al. 2023] are concurrent work. Given this fact and the other addressed points, I have updated my score and recommend acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Being able to inject user control into the output of a diffusion model is an important goal. Using a dense advection field as guidance is an intuitive way to manipulate images and could be potentially be useful for enforcing other spatial constraints on diffusion model outputs like temporal consistency and so on. The paper is clearly written, and a comprehensive ablation study validates many of the design choices.", "weaknesses": "My main concern has to do with lack of comparisons with recent works. While the authors discuss some of these in related works, they do not share any qualitative or quantitative comparisons. In particular, [Shi et al. 2023] and [Mou et al. 2023] seem to enable very similar functionality. While I recognize the authors' explanation that their approach is more efficient in that it requires no additional training or fine-tuning and is only done at inference time, I think it would be necessary see how these other methods perform on the same examples.\n\nI would also be interested in seeing how this approach handles more complicated, higher-frequency flow fields. It seems like one big advantage of being able to control a diffusion model via a dense flow field is to enforce fine-grained motion. However, all the examples feature quite coarse and global transformations.\n\nFinally, I am a little concerned about possible hallucination or identity loss resulting in these transformations. This is particularly evident in the motion transfer examples of Figure 7---as a result of the of the motion edit, the subject in the image also changes considerably. Is there a way to mitigate this effect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T352XbZmTP", "reviewer": "Reviewer_iTDk", "summary": "This paper proposes a new motion guidance technique for diffusion models. The specified motion field can be arbitrary, but the reconstructed images are always plausible thanks to the diffusion model and the color loss. The range of supported motions is diverse, starting from simple translation and up to complex non-rigid motions. The method is extensively evaluated on several datasets, both qualitatively and quantitatively.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- An extensive related work section with good literature overview.\n- The method supports many different types of transformations: translation, rotation, scaling, stretching, and other complex deformations.\n- The method does not require any additional training and runs directly on the required inputs. It is also designed in such a way that it is independent on the used architecture.\n- The results look visually pleasing.\n- Many qualitative ablation studies support main contributions.\n- The paper is well-written and easy to understand.", "weaknesses": "- Unfortunately, all experiments and ablations are done mostly qualitatively. This brings a question whether the results are cherry-picked. I'd like to see more quantitative evaluations. Maybe the authors could evaluate the consistency of predictions over longer videos.\n- In all experiments, the authors used optical flow (probably from a video). However, how can this method be used if the end user has only one image and wants to edit it? There is one experiment (Fig. 7), which shows that it can be estimated from a different video (and I appreciate it). However, I'd like to see if a hand-drawn optical flow could be used.\n- eq. (3): later in Sec. 3.3 it is written that the color loss is masked out in \"the occluded regions\". However, this is not represented in eq. (3). Please update the equation to be precise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YEqnABMkZT", "reviewer": "Reviewer_wHkp", "summary": "This paper presents a new method for editing images with diffusion models and optical flow guidance. The proposed method does not require any network training or finetuning, and does not modified the diffusion network architecture. Instead, it applies an additional loss to guide the diffusion sampling process, making the edited image follow motion guidance without changing image contents. Experiments show that the proposed method can handle various type of motion guidance and generate high-quality results.", "questions": "* I think the proposed method can be easily extended for video generation by providing a sequence of flow guidance. It would make the paper stronger if the authors could provide such an example.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The core of the proposed image editing method is a loss function that incorporates a pretrained optical flow network for motion supervision, and does not need any network training or finetuning. Therefore, it requires no data collection or network modification. \n\n* The proposed method is simple yet flexible and supports a wide range of motion guidance. When manipulating the image contents, the proposed method is able to hallucinate missing pixels like disoccluded areas or backgrounds.  The qualitative results are impressive. \n\n* The authors conduct extensive experiments to demonstrate the effectiveness and flexibility of the proposed method. The experiments are solid and convincing. \n\n* The paper is overall well-written and easy to follow.", "weaknesses": "* The proposed method requires dense optical flows as the supervision signal. This is not a user-friendly input, as it is not clear how to obtain such flows without programming. DragGAN [Pan et al. 2023] is more user-friendly; users just need to simply drag a small number of pixels to manipulate images. In addition,  DragGAN provide feedback to users almost immediately, while the proposed method is slow to execute. \n\n* In Figure 5, the authors conduct a comparison experiment with DragGAN on out-of-domain images. However, StyleGANs are trained on narrow domains and DragGAN works better if the input image falls in the training image domain.  Although GAN inversion still works for out-of-domain images, I think it would be better if the authors can compare with DragGAN using in-domain images (e.g., human facial images) to make the experiment stronger and more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c52AT96ANl", "reviewer": "Reviewer_6Wgo", "summary": "The authors propose a run-time image editing method based on diffusion models using an optical flow as input. The method can synthesize a realistic looking image given the reference image and optical flow fields. The synthesis is done via a new motion guidance, which is composed of a flow reconstruction loss between a provided flow map to 2D optical flow estimate from reference image to the generated image, and a color loss to ensures the generated image is consistent to input image when warping back. The authors provide extensive visual examples and ablations to show the efficacy of this approach", "questions": "* For the recusive denoising, is the K constant number for all the experiments? I did not find what the parameter is used in the implementation details section.\n* In flow loss discussion on page 6, the authors mentions \"Without the flow loss our method is able to move objects to the correct location but often hallucinates things in disoccluded areas\". Though it is true from the fig.3, the disocclusion regions looks quite wrong with flow loss, I did not quite understand why it is the case. The disocclusion region should not really impact the optical flow since they don't really have correspondence. One possible explanation is that they could potentially create ambiguity between the input image and synthesize image for the estimated flow, and adding flow floss could drive it to remove that potential ambiguity. I wonder whether that's the potential case? \n* Both Eq. (2) & (3) seems require the gradient w.r.t. the whole flow estimator if I understand correctly. That could be really slow? I wonder the compute time for each. For eq (3), will it make any difference if using f instead of F(x^{\\star}, x) ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is technically sound and simple. The approach should be easy to implement at runtime, and can be pretty general for most image synthesis backbones. \n* The authors provided very clear ablations study for many implementation details in the paper, including how to do denoising steps, editing mask, handling occlusions, which are crucial for the success. I like the provided examples which very clearly highlight the issues of each item if not being used. \n* The method performs favorable compared to alternative baselines. The method provides great visual consistency with respect to the input image and flow map.", "weaknesses": "* As the author indicated in Sec 4.6, it is hard to create pixel-wise flow map by hand, which I personally will be the biggest limitation for this method to be applied in real-world use-case. I am not exactly how the authors demonstrate all the examples in results section, but from the results, they definitely look a real pixel aligned motion field regarding the input image, which is almost impossible to get in applications. From the motion transfer results, the authors discuss using \"fewer recursive denoising steps\" to get slightly better results. I wonder that's related to the gap in pixel aligned optical flow. I think it worth providing a careful analysis here to help readers understand the potential gap and insights to address these issues which is how this method used in real-world use-cases. I wonder whether there are also tasks (e.g. video stylization) that can better map the input of this method. \n* A suggestion, not a weakness. The demonstration of flow map can be more clear if provided with the color mapping chat with the flow (motion to color).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)