PixelMan: Consistent Object Editing with Diffusion Models via Pixel
  Manipulation and Generation
Authors: name: Liyao Jiang; name: Negar Hassanpour; name: Mohammad Salameh; name: Mohammadreza Samadi; name: Jiao He; name: Fengyu Sun; name: Di Niu
Published: 2024-12-18 19:24:15

Abstract: Recent research explores the potential of Diffusion Models (DMs) for
consistent object editing, which aims to modify object position, size, and
composition, etc., while preserving the consistency of objects and background
without changing their texture and attributes. Current inference-time methods
often rely on DDIM inversion, which inherently compromises efficiency and the
achievable consistency of edited images. Recent methods also utilize energy
guidance which iteratively updates the predicted noise and can drive the
latents away from the original image, resulting in distortions. In this paper,
we propose PixelMan, an inversion-free and training-free method for achieving
consistent object editing via Pixel Manipulation and generation, where we
directly create a duplicate copy of the source object at target location in the
pixel space, and introduce an efficient sampling approach to iteratively
harmonize the manipulated object into the target location and inpaint its
original location, while ensuring image consistency by anchoring the edited
image to be generated to the pixel-manipulated image as well as by introducing
various consistency-preserving optimization techniques during inference.
Experimental evaluations based on benchmark datasets as well as extensive
visual comparisons show that in as few as 16 inference steps, PixelMan
outperforms a range of state-of-the-art training-based and training-free
methods (usually requiring 50 steps) on multiple consistent object editing
tasks.