Style-Friendly SNR Sampler for Style-Driven Generation
Authors: name: Jooyoung Choi; name: Chaehun Shin; name: Yeongtak Oh; name: Heeseung Kim; name: Sungroh Yoon
Published: 2024-11-22 08:29:25

Abstract: Recent large-scale diffusion models generate high-quality images but struggle
to learn new, personalized artistic styles, which limits the creation of unique
style templates. Fine-tuning with reference images is the most promising
approach, but it often blindly utilizes objectives and noise level
distributions used for pre-training, leading to suboptimal style alignment. We
propose the Style-friendly SNR sampler, which aggressively shifts the
signal-to-noise ratio (SNR) distribution toward higher noise levels during
fine-tuning to focus on noise levels where stylistic features emerge. This
enables models to better capture unique styles and generate images with higher
style alignment. Our method allows diffusion models to learn and share new
"style templates", enhancing personalized content creation. We demonstrate the
ability to generate styles such as personal watercolor paintings, minimal flat
cartoons, 3D renderings, multi-panel images, and memes with text, thereby
broadening the scope of style-driven generation.