Real-World Image Variation by Aligning Diffusion Inversion Chain
Authors: name: Yuechen Zhang; name: Jinbo Xing; name: Eric Lo; name: Jiaya Jia
Published: 2023-05-30 04:09:47

Abstract: Recent diffusion model advancements have enabled high-fidelity images to be
generated using text prompts. However, a domain gap exists between generated
images and real-world images, which poses a challenge in generating
high-quality variations of real-world images. Our investigation uncovers that
this domain gap originates from a latents' distribution gap in different
diffusion processes. To address this issue, we propose a novel inference
pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes
diffusion models to generate image variations from a single image exemplar. Our
pipeline enhances the generation quality of image variations by aligning the
image generation process to the source image's inversion chain. Specifically,
we demonstrate that step-wise latent distribution alignment is essential for
generating high-quality variations. To attain this, we design a cross-image
self-attention injection for feature interaction and a step-wise distribution
normalization to align the latent features. Incorporating these alignment
processes into a diffusion model allows RIVAL to generate high-quality image
variations without further parameter optimization. Our experimental results
demonstrate that our proposed approach outperforms existing methods with
respect to semantic-condition similarity and perceptual quality. Furthermore,
this generalized inference pipeline can be easily applied to other
diffusion-based generation tasks, such as image-conditioned text-to-image
generation and example-based image inpainting.