Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos
Authors: name: Hadi Alzayer; name: Zhihao Xia; name: Xuaner Zhang; name: Eli Shechtman; name: Jia-Bin Huang; name: Michael Gharbi
Published: 2024-03-19 17:59:58

Abstract: We propose a generative model that, given a coarsely edited image,
synthesizes a photorealistic output that follows the prescribed layout. Our
method transfers fine details from the original image and preserves the
identity of its parts. Yet, it adapts it to the lighting and context defined by
the new layout. Our key insight is that videos are a powerful source of
supervision for this task: objects and camera motions provide many observations
of how the world changes with viewpoint, lighting, and physical interactions.
We construct an image dataset in which each sample is a pair of source and
target frames extracted from the same video at randomly chosen time intervals.
We warp the source frame toward the target using two motion models that mimic
the expected test-time user edits. We supervise our model to translate the
warped image into the ground truth, starting from a pretrained diffusion model.
Our model design explicitly enables fine detail transfer from the source frame
to the generated image, while closely following the user-specified layout. We
show that by using simple segmentations and coarse 2D manipulations, we can
synthesize a photorealistic edit faithful to the user's input while addressing
second-order effects like harmonizing the lighting and physical interactions
between edited objects.