Repulsive Score Distillation for Diverse Sampling of Diffusion Models
Authors: name: Nicolas Zilberstein; name: Morteza Mardani; name: Santiago Segarra
Published: 2024-06-24 14:43:02

Abstract: Score distillation sampling has been pivotal for integrating diffusion models
into generation of complex visuals. Despite impressive results it suffers from
mode collapse and lack of diversity. To cope with this challenge, we leverage
the gradient flow interpretation of score distillation to propose Repulsive
Score Distillation (RSD). In particular, we propose a variational framework
based on repulsion of an ensemble of particles that promotes diversity. Using a
variational approximation that incorporates a coupling among particles, the
repulsion appears as a simple regularization that allows interaction of
particles based on their relative pairwise similarity, measured e.g., via
radial basis kernels. We design RSD for both unconstrained and constrained
sampling scenarios. For constrained sampling we focus on inverse problems in
the latent space that leads to an augmented variational formulation, that
strikes a good balance between compute, quality and diversity. Our extensive
experiments for text-to-image generation, and inverse problems demonstrate that
RSD achieves a superior trade-off between diversity and quality compared with
state-of-the-art alternatives.