FAM Diffusion: Frequency and Attention Modulation for High-Resolution
  Image Generation with Stable Diffusion
Authors: name: Haosen Yang; name: Adrian Bulat; name: Isma Hadji; name: Hai X. Pham; name: Xiatian Zhu; name: Georgios Tzimiropoulos; name: Brais Martinez
Published: 2024-11-27 17:51:44

Abstract: Diffusion models are proficient at generating high-quality images. They are
however effective only when operating at the resolution used during training.
Inference at a scaled resolution leads to repetitive patterns and structural
distortions. Retraining at higher resolutions quickly becomes prohibitive.
Thus, methods enabling pre-existing diffusion models to operate at flexible
test-time resolutions are highly desirable. Previous works suffer from frequent
artifacts and often introduce large latency overheads. We propose two simple
modules that combine to solve these issues. We introduce a Frequency Modulation
(FM) module that leverages the Fourier domain to improve the global structure
consistency, and an Attention Modulation (AM) module which improves the
consistency of local texture patterns, a problem largely ignored in prior
works. Our method, coined Fam diffusion, can seamlessly integrate into any
latent diffusion model and requires no additional training. Extensive
qualitative results highlight the effectiveness of our method in addressing
structural and local artifacts, while quantitative results show
state-of-the-art performance. Also, our method avoids redundant inference
tricks for improved consistency such as patch-based or progressive generation,
leading to negligible latency overheads.