A Noise is Worth Diffusion Guidance
Authors: name: Donghoon Ahn; name: Jiwon Kang; name: Sanghyun Lee; name: Jaewon Min; name: Minjae Kim; name: Wooseok Jang; name: Hyoungwon Cho; name: Sayak Paul; name: SeonHwa Kim; name: Eunju Cha; name: Kyong Hwan Jin; name: Seungryong Kim
Published: 2024-12-05 06:09:56

Abstract: Diffusion models excel in generating high-quality images. However, current
diffusion models struggle to produce reliable images without guidance methods,
such as classifier-free guidance (CFG). Are guidance methods truly necessary?
Observing that noise obtained via diffusion inversion can reconstruct
high-quality images without guidance, we focus on the initial noise of the
denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we
uncover that small low-magnitude low-frequency components significantly enhance
the denoising process, removing the need for guidance and thus improving both
inference throughput and memory. Expanding on this, we propose \ours, a novel
method that replaces guidance methods with a single refinement of the initial
noise. This refined noise enables high-quality image generation without
guidance, within the same diffusion pipeline. Our noise-refining model
leverages efficient noise-space learning, achieving rapid convergence and
strong performance with just 50K text-image pairs. We validate its
effectiveness across diverse metrics and analyze how refined noise can
eliminate the need for guidance. See our project page:
https://cvlab-kaist.github.io/NoiseRefine/.