Interpreting the Weight Space of Customized Diffusion Models
Authors: name: Amil Dravid; name: Yossi Gandelsman; name: Kuan-Chieh Wang; name: Rameen Abdal; name: Gordon Wetzstein; name: Alexei A. Efros; name: Kfir Aberman
Published: 2024-06-13 17:59:56

Abstract: We investigate the space of weights spanned by a large collection of
customized diffusion models. We populate this space by creating a dataset of
over 60,000 models, each of which is a base model fine-tuned to insert a
different person's visual identity. We model the underlying manifold of these
weights as a subspace, which we term weights2weights. We demonstrate three
immediate applications of this space -- sampling, editing, and inversion.
First, as each point in the space corresponds to an identity, sampling a set of
weights from it results in a model encoding a novel identity. Next, we find
linear directions in this space corresponding to semantic edits of the identity
(e.g., adding a beard). These edits persist in appearance across generated
samples. Finally, we show that inverting a single image into this space
reconstructs a realistic identity, even if the input image is out of
distribution (e.g., a painting). Our results indicate that the weight space of
fine-tuned diffusion models behaves as an interpretable latent space of
identities.