Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free
  Real Image Editing
Authors: name: Vadim Titov; name: Madina Khalmatova; name: Alexandra Ivanova; name: Dmitry Vetrov; name: Aibek Alanov
Published: 2024-09-02 15:21:46

Abstract: Despite recent advances in large-scale text-to-image generative models,
manipulating real images with these models remains a challenging problem. The
main limitations of existing editing methods are that they either fail to
perform with consistent quality on a wide range of image edits or require
time-consuming hyperparameter tuning or fine-tuning of the diffusion model to
preserve the image-specific appearance of the input image. We propose a novel
approach that is built upon a modified diffusion sampling process via the
guidance mechanism. In this work, we explore the self-guidance technique to
preserve the overall structure of the input image and its local regions
appearance that should not be edited. In particular, we explicitly introduce
layout-preserving energy functions that are aimed to save local and global
structures of the source image. Additionally, we propose a noise rescaling
mechanism that allows to preserve noise distribution by balancing the norms of
classifier-free guidance and our proposed guiders during generation. Such a
guiding approach does not require fine-tuning the diffusion model and exact
inversion process. As a result, the proposed method provides a fast and
high-quality editing mechanism. In our experiments, we show through human
evaluation and quantitative analysis that the proposed method allows to produce
desired editing which is more preferable by humans and also achieves a better
trade-off between editing quality and preservation of the original image. Our
code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.