FreeU: Free Lunch in Diffusion U-Net
Authors: name: Chenyang Si; name: Ziqi Huang; name: Yuming Jiang; name: Ziwei Liu
Published: 2023-09-20 17:56:18

Abstract: In this paper, we uncover the untapped potential of diffusion U-Net, which
serves as a "free lunch" that substantially improves the generation quality on
the fly. We initially investigate the key contributions of the U-Net
architecture to the denoising process and identify that its main backbone
primarily contributes to denoising, whereas its skip connections mainly
introduce high-frequency features into the decoder module, causing the network
to overlook the backbone semantics. Capitalizing on this discovery, we propose
a simple yet effective method-termed "FreeU" - that enhances generation quality
without additional training or finetuning. Our key insight is to strategically
re-weight the contributions sourced from the U-Net's skip connections and
backbone feature maps, to leverage the strengths of both components of the
U-Net architecture. Promising results on image and video generation tasks
demonstrate that our FreeU can be readily integrated to existing diffusion
models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion,
to improve the generation quality with only a few lines of code. All you need
is to adjust two scaling factors during inference. Project page:
https://chenyangsi.top/FreeU/.