Score Distillation Sampling with Learned Manifold Corrective
Authors: name: Thiemo Alldieck; name: Nikos Kolotouros; name: Cristian Sminchisescu
Published: 2024-01-10 17:51:46

Abstract: Score Distillation Sampling (SDS) is a recent but already widely popular
method that relies on an image diffusion model to control optimization problems
using text prompts. In this paper, we conduct an in-depth analysis of the SDS
loss function, identify an inherent problem with its formulation, and propose a
surprisingly easy but effective fix. Specifically, we decompose the loss into
different factors and isolate the component responsible for noisy gradients. In
the original formulation, high text guidance is used to account for the noise,
leading to unwanted side effects. Instead, we train a shallow network mimicking
the timestep-dependent denoising deficiency of the image diffusion model in
order to effectively factor it out. We demonstrate the versatility and the
effectiveness of our novel loss formulation through several qualitative and
quantitative experiments, including optimization-based image synthesis and
editing, zero-shot image translation network training, and text-to-3D
synthesis.