Interpreting the Weight Space of Customized Diffusion Models
Authors: Amil Dravid
Yossi Gandelsman
Kuan-Chieh Wang
Rameen Abdal
Gordon Wetzstein
Alexei A Efros
Kfir Aberman
Venue: NeurIPS 2024 poster (2024)
Keywords: Weight Space
Model Editing
Diffusion Models
Latent Space
Personalization
Primary area: diffusion_based_models
TL;DR: Using a dataset of fine-tuned diffusion models, we define a subspace in diffusion model weight space that enables controllable creation of new diffusion models.

Abstract: We investigate the space of weights spanned by a large collection of customized diffusion models. We populate this space by creating a dataset of over 60,000 models, each of which is a base model fine-tuned to insert a different person's visual identity. We model the underlying manifold of these weights as a subspace, which we term $\textit{weights2weights}$. We demonstrate three immediate applications of this space that result in new diffusion models -- sampling, editing, and inversion. First, sampling a set of weights from this space results in a new model encoding a novel identity. Next, we find linear directions in this space corresponding to semantic edits of the identity (e.g., adding a beard), resulting in a new model with the original identity edited. Finally, we show that inverting a single image into this space encodes a realistic identity into a model, even if the input image is out of distribution (e.g., a painting). We further find that these linear properties of the diffusion model weight space extend to other visual concepts. Our results indicate that the weight space of fine-tuned diffusion models can behave as an interpretable $\textit{meta}$-latent space producing new models.

Interpreting the Weight Space of Customized Diffusion Models
Amil Dravid 122
Yossi Gandelsman
Kuan-Chieh Wang"
Rameen Abdal
Gordon Wetzstein
Alexei A. Efros!
Kfir Aberman
I UC Berkeley Snap Inc.
Stanford University
weights (w2w) Space
Diffusion Unet
weights
Single Image Inversion
Identity Editing
Sampling
Figure I: weights2weight (w2w) space enables controllable creation of new customized diffusion models. We model a manifold of customized diffusion models as a subspace of weights that encodes different instances of broad visual concept (e.g.. human identities, dog breeds, etc.). This forms a space that supports inverting the subject (e.g., identity) from single image into a model, editing the subject encoded in the model, and sampling concept. Fach of these operations results in a new model that can consistently generate the subject.
models that encode instances of the visual
Abstract
We investigate the space of weights spanned by large collection of customized diffusion models. We populate this space by creating a dataset over 60,000 models, each of which is a base model fine-tuned insert different person's 's visual identity. We model the underlying manifold of these weights as a subspace, which term weightssweightts. We demonstrate three immediate applications of this space that result in new diffusion models sampling, editing, and inversion. First, sampling set of weights from this space results in a novel identity. Next, we find linear directions in this space corresponding to model with
model encoding
semantic edits of the identity (e.g., adding beard), resulting in the original identity edited. Finally, we show that inverting a single image into this space encodes realistic identity into a model, even if the input image is out of distribution (e.g., painting). We further find that these linear properties of the diffusion model weight space extend to other visual concepts. Our results indicate that the weight space of fine-tuned diffusion models can behave as an interpretable meta- mete-l space producing new models.!
Equaa contribution
Project page: https: https://snap-reesercc. gittggittttut.. 1o/ io/weighttsvveihts
Code: https:/ //gttbb.// com cetaaapreeeerch//eee rew.errrrrreeiigtsssssiiiiiiiiie
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Sampling
Editing
Inversion
Latent space
image
reconstructed image
z Rehubby
latent image
- image editing
w2w space
model 0 E w2w-> identity
0 A0 clubbry identity editing
image
reconstructed identity
paroo..
A(v] person...
Figure 2: The weights2weightts space operates as a meta-latent space. Unlike a traditional generative latent space, w2w space controls the model itself rather than single image instances. New identity-encoing models can be sampled from the space and edited by linearly traversing along semantic directions in weight space. Additionally, single image can be inverted into the space produce model that consistently generates that identity.
Introduction
Generative models have emerged powerful tool to model our rich visual world. In particular, the latent space of single-step generative models, such as generative adversarial networks (GANs) [18, 28), has been shown to linearly encode meaningful concepts in the output images. For instance, datasets of latent vectors were used to discover linear directions in the GAN latent space encoding different attributes (e.g., gender or age of faces) [20, 60]. Even earlier, datasets of images and keypoints were leveraged to discover subspaces of facial shape and appearance [6, 53].
We aim to extend this even further, using datasets of model weights instead datasets of images or latents. Can we discover such interpretable subspaces in the model weights themselves? Recently introduced personalization approaches, such Drcambooth [54] or Custom Diffusion [34], may hint that this is the case. These methods aim to learn an instance of a subject, such as a person's visual identity. Rather than searching for a latent code that represents identity in the input noise space, these approaches customize diffusion models by fine-tuning on subject-specific images, which results in ddentity-specifff model weights. We therefore hypothesize that a latent space can exist in the weights themselves.
To test our hypothesis, we fine-tune over 60,000 personalized models on individual identities obtain points that lie on a manifold of customized diffusion model weights. To reduce the dimensionality of each data point, use low-rank approximation (LoRA) [23] during fine-tuning and further apply Principal Components Analysis (PCA) to the set of data points. This forms our final space: weights2weights (w2w). Unlike traditional generative models like GANs, which model the pixel space of images, we model the weight space of these personalized models. Thus, each sample in space corresponds to an identity-speciff model which can consistently generate that subject. We provide a schematic in Fig. 2 that contrasts typical latent space with our proposed w2w space, demonstrating the differences and analogies between these two representations. w2w space can be thought of as meta-latent space, enabling controllable creation of new models instead of just images like traditional latent space.
Creating this space unlocks a variety of applications that involve traversal in w2w (Fig. 1). First, we demonstrate that sampling model weights from w2w space corresponds to new model encoding novel subject. Second, we find linear directions this space corresponding to semantic edits of the identity. Finally, we show that enforcing weights to live in this space enables diffusion model to learn a subject given a single image, even is out of distribution.
We find that w2w space is highly expressive through quantitative evaluation on editing customized models and encoding new identities given single image. Qualitatively, we observe this space supports sampling models that encode diverse and realistic identities, while also capturing the key characteristics of -uito--distribution identities. We finally demonstrate that similar weight subspaces exist for other visual concepts such as dog breeds and car types.

---

Model weights Fine-Tuning
Dimensionality Reduction
Projecting back to weights space
"Young"
hyperplane
"Old"
Identities Dataset
Model Weights Space
w2w Space
Sampling and Editing
Figure 3: Building weights2weights (w2w) space. We create dataset of model weights where each model is personalized to a specific identity using low-rank updates (LoRA). These model weights lie on weights manifold that we further project into lower-dimensions subspace spanned by its principal components. We train linear classifiers to find disentangled edit directions in this space.
2 Related Work
Image-based generative models. Various models have been proposed for image generation, includ- ing variational autoencoders (VAEs) [31], flow-based models [12, 32, 49), generative adversarial networks (GANs) [18], and diffusion models [22, 43, 62]. Within the realm of high-qualit photo- realistic image generation, GANs [25, 28, 29] and diffusion models [22, 43, 52, 63] have garnered significant attention due to their controllability and ability to produce high-quality images. Leverag- ing the compositionality of these models, methods for personalization and customization have been developed which aim to insert user-defined concepts via fine-tuning [16, 34, 40, 54). Various works try to reduce the dimensionality of the optimized parameters for personalization either by operating in specific model layers [34] in text-emhedding space [16], by training hypernetworks [55], and by constructing a linear basis in text embedding space [73].
Latent space of generative models. Linear latent space models of facial shape and appearance were studied extensively in the 1990s, using PCA-based representations (e.g. Active Appearance Models [9], 3D Morphable Models [6]) well as operating directly in pixel and keypoint space [53]. However, these techniques were restricted to aligned and cropped frontal faces. More recently, generative adversarial networks (GANs), particularly the StyleGAN series [26, 27, 28, 29]. have showcased editing capabilities facilitated by their interpretable latent space. Furthermore, linear directions can be found in their latent space to conduct semantic edits by training linear classifiers applying PCA [20, 60], among other methods for discovering semantic directions [8, 66). Several methods aim to project real images into the GAN latent space in order conduct this editing [1, 51, 64, 77J. Beyond the latent space, works such as [4] found that directions could be discovered in the neuron activation space, suggesting the interpretability of weights.
Although diffusion models architecturally lack a GAN-like latent space, some works aim to discover similar spaces in these models. This has been explored in the UNet bottleneck layer [35, 41], noise space [10, 78]. and rexbedibd space [5]. Concept Sliders [17] explores the weight space for semantic image editing by conducting low-rank training with contrasting image or text pairs.
Weights as data. Past works have exploited the structure within weight space of deep networks for various applications. In particular, some have found linear properties of weights, enabling simple model ensembling and editing via arithmetic operations [24, 56, 59, 69]. Other works create datasets of neural network parameters for training hypernetworss [14, 19, 42, 55, 68), predicting properties of networks [58], and creating design spaces for models [46, 47].
Method
We start by demonstrating how we create a manifold of model weights as illustrated Fig. 3. We explain how we obtain low-dimensional data points for this space, each of which represents an individual subject from a broad class (i.e., identity). We then use these points to model a weights manifold. Next, we find linear directions in this manifold that correspond to semantic attributes and use them for editing the identities. Finally, we demonstrate how this manifold can be utilized for constraining ill- ill-posed inversion task with a single image to reconstruct its identity.
3

---

3.1 Preliminaries
In this section, we first introduce latent diffusion models (LDM) [52]. which we will use create a dataset of weights. Then, we explain the approach for obtaining identity-speciif models from LDM via Dreambooth [54] fine-tuning. We finally present version of fine-tuning that uses low- dimensional weight updates (LoRA [23]). We will use the fine-tuned low-dimensional per-identity weights as data points to construct the weights manifold in Sec. 3.2.
Latent diffusion models [52]. We will extract weights from latent diffusion models to create w2w space. These models follow the standard diffusion objective [22] while operating on latents extracted from a pre-trained Variational Autoencoder [15, 31, 50J. With text, the conditioning signal is encoded by a text encoder (such as CLIP [44]), and the resulting embeddings are provided to the denoising UNet model. The loss of latent diffusion models is:
ex.c,e,t[ux ||e cø (xe, c, t)1)3)
(1)
where eø is the denoising UNet, X is the noised version of the latent for image, is the conditioning signal, is the diffusion timestep, and wy is time-dependent weight on the loss.
To sample from the model, random Gaussian latent xy is deterministically denoised conditioned on a prompt for a fixed set of timesteps with a DDIM sampler [63]. The denoised latent is then fed through the VAE decoder to generate the final image.
Dreambooth [54]. To obtain an identity-speciffe model, we use the Dreambooth personalization method. Dreambooth fine-tuning introduces novel subject into a pre-trained diffusion model given only a few images of it. During training, Dreambooth follows two-part objective:
Ex,,,c, /wwlle co 1 + Auy cx( c', (1)))
(2)
where the first term corresponds to the standard diffusion denoising objective using the subject- specific data x conditioned on the text prompt "[identifier] [class noun]" (e.g., "[v/ person"), denoted c. The second term, weighted by X, corresponds to prior preservation loss, which involves the standard denoising objective using the model's own generated samples x' for the broader class c' (e.g.. "person"" This prevents the model from associating the class name with the specific instance, while also leveraging the semantic prior on the class.
Low Rank Adaptation (LoRA) [23]. Dreambooth requires fine-tuning all the weights of a model, which is high dimensiona space. We turn to more efficient fine-tuning scheme, LoRA, that modifies only a low-rank version of the weights. LoRA uses weight updates AW with a low intrinsic rank. For a base model layer W mx the LoRA update for that layer AW can be decomposed into A AW- = BA, where B E Rmxr and A E Rx"= are low-rank matrices with r < min(m, n). During training. for each model layer, only the A and B are updated. This significantly reduces the number of trainable parameters. During inference, the low-rank weights are added residually to the weights of each layer in the base model and scaled by a coefficient a E R: W aAW.
3.2 Constructing the weights manifold
Creating a dataset of model weights. To construct the weights2weights (w2w) space, we begin by creating dataset of model weights 0,. We conduct Dreambooth fine-tuning on latent diffusion models in order to insert new subjects with the ability to control image instances using text prompts. This training is done with LoRA in order to reduce the space of model parameters. Each model is fine-tuned on a set of images corresponding to one human subject. After training, we flatten and concatenate all of the LoRA matrices, resulting in data point 0, E Rd which represents one identity. After training over N different instances, we have our final dataset of model weights D (01,02 02.,0 0N). representing diverse array of subjects.
Modeling the weights manifold. We posit that our data D S Rd lies on a lower-dimensional manifold of weights that encode identities. A randomly sampled set of weights in Rdd, would not be guaranteed to produce a valid model encoding identity as the d degrees of freedom can be fine-tuned for any purpose. Therefore, we hypothesize that this manifold is a subset of the weight space. Inspired by findings that high-level concepts can be encoded as linear subspaces of representations [13, 37, 45, 48), we model this subset as a linear subspace Rm where m < d, and call it weightssweightss (w2w) space. We represent points in this subspace as a linear combination

---

of basis vectors w {wi.... W. wA R In practice, we apply Principal Component Analysis (PCA) on the N models and keep the first m principal components for dimensional reduction and forming our basis of m vectors.
Sampling from the weights manifold. After modeling this weights manifold, model that lies on it, resulting in a new model that generates a novel identity. We sample model represented with basis coefficients {B1, 3m J. where each coefficient 3x is sampled from a normal distribution with mean and standard deviation σы. The mean and standard deviation are calculated for each principal component i from the coefficients among all the training models.
sample new
3.3 Finding Interpretable Weight Space Directions
We seek a direction n E Rd defining a hyperplane that separates between binary identity properties embedded in the model weights (e.g., male/female), similarly to hyperplanes observed in the latent space of GANs [60]. We assume binary labels are given for attributes present in the identities encoded by the models. We then train linear classifiers using weights of the models data based on these labels, imposing separating hyperplanes in weight space. Given an identity parameterized by weights 8, we can manipulate a single attribute by traversing in a direction n, orthogonal to the separating hyperplane: Oodi 0 + on. An edit operation in w2w spaee produces a new model with the original subject edited, allowing the model to generate infinitely many new images of the edited subject.
3.4 Inversion into w2w Space
Traditionally, inversion of generative model involves finding an input such as latent code that best reconstructs given image [38, 70]. This corresponds finding projection of the input onto the learned data manifold [77]. With w2w space, we model manifold of model weights rather than images. Inspired by latent optimization methods [1, 77], we propose a gradient- gradient-based method of inverting single identity from an image into our discovered space.
Given single image x, we follow constrained denoising objective:
max Ex Ex,,,e ea (Xe. ()11) s.t. 0 E w2w
Specifically, constrain the model weights to lie in w2w space by optimizing set of basis coefficients [31, Bm) rather than the original parameters. Unlike Dreambooth, we do not employ prior preservation loss, since the optimized model lies in the subspace defined by our dataset of weights, and inherits their priors
Experiments
We demonstrate w2w space on the visual concept of human identities for variety of applications. We begin by detailing implementation details. Next, we use w2w space for 1) sampling new models encoding novel identities, 2) editing identity attributes in consistent manner via linear traversal in w2w space, 3) embedding new identity given a single image, and 4) projecting out-of uutt-f-distriuutinn identities into w2w space. Finally, we analyze how scaling the number of models in our dataset of model weights affects the disentanglement of attribute directions and preservation of identity.
4.1 Implementation Details
Creating an identity dataset. We generate synthetic dataset of ~65,000 identities using [67], where each identity is associated with multiple images of that person. Each identity is based on an image with labeled binary attributes (e.g., male/female) from CelebA [36]. Each set of images corresoonding to an identity is then used as data to fine-tune latent diffusion model with Dreambooth. Further details on this dataset and train/test splits are provided Appendix
Encoding identities into model weights. We conduct Dreambooth fine-tuning using LoRA with rank on the identities. Following [56], we only fine-tune the query and value projection matrices in the cross-attention layers. We utilize the Realistic Vision-v51 checkpoint based on Stable Diffusion
5

---

1.5, Conducting Dreambooth fine-tuning on each identity training set results in dataset of ~65,000 weights 0 where 0 E IR 100,000
Finding semantic attribute directions. We utilize binary attribute labels from CelebA to train linear classifiers the dataset of model weights curated. We run Principal Component Analysis (PCA) on the ~~5,0000 training models and project to the first 1000 principal components in order to reduce the dimensionality. The orthogonal edit directions are calculated via the analytic least squares solution on the matrix of projected training models D E |665,00011000 and then unprojected to the original dimensionality of the model weights: 0 E I 100000000
4.2 Sampling from w2w Space
Sampled Identity
Nearest Neighbor
We present images generated from models that sampled from the weights manifold (i.e., w2w Space) in Fig. 4. We follow the sampling procedure from Sec. 3.2, and generate images from the sampled model. As shown, each new model encodes a novel, realistic, and consistent identity. Additionally, we present the nearest neighbor model among the training dataset of model weights. We use cosine similarity on the models' principal component representations. Comparing with the nearest neighbors shows that these samples are not just copies from the dataset, but rather encode diverse identities with different attributes. Yet, the samples still demon- strate some similar features to the nearest neigh- bors. These include jawline and eye shape (top row), facial hair (middle row), and nose and eye shape (bottom row). Appendix A includes more such examples.
Figure 4: Identity samples from w2w space. We show the samples from w2w space do not overfit to nearest-neighbor identities, although they incor- porate facial attributes from them. The identities diverse and consistent across generations.
4.3 Editing Subjects
We demonstrate how directions found by the linear classifiers can be used to edit subjects encoded in the models. It is desired that these edits are 1) disentangled (i.e., do not interfere with other attributes of the embedded subject and preserve all other concepts such as context) 2) identity preserving (i.c., the person is still recognizable) 3) and semantically aligned with the intended edit.
Baselines. We compare against a naïve baseline of prompting with the desired attribute "/v. person with small eyes"), and then Concept Sliders [17], an instance-specife editing method which we adapt to subject editing. particular, we train their most accessible method, the bext-based slider, which trains LoRAs to modulate attributes in a pretrained diffusion model based on contrasting text prompts. We then apply these sliders to the personalized identity models.
Woman
Chubby
Narrow Eyes
Sliders Prompting
W2w
Figure 5: Qualitative comparison. w2w edits preserve identity while being disentangled and semantically aligned. Concept Sliders [17] tends to exaggerate effects which induces artifacts and
degrades identity, while prompting the subject with the desired edit has unexpected effects.

---

Table 1: Edits in w2w space preserve identity, are disentangled, and semantically aligned.
ID Score f
LPIPS 4
CLIP Score 1 Sliders
Prompting
Sliders
w2w
Prompting
Sliders
w2w
Prompting
w2w
Gender
0.39+
0.33+
0. 0.4500..0
0.30+
0.30+
0..1=
1.98=0.75
3. 3.50=o.0s
4.1000.00
Chubby
0.29
0.33
0. 0.45
a..0
0.38
2006
1.12.0.01
2.16
Eyes
0. 0.52+0.0
0.53+0.04
0. 0.72+0.0
0.32+6.08
+.30
0.19
0.17=0.17
0.01=0.22
0.59
Original
Flat Brows
Bangs
Straight Hair
Original
+ +Jawline
+ Eye Bags
+ Narrow Eyes
0155
DISp
DnS
Figure 6: Composing edits in w2w space. Each column represents fixed seed samples from edited model. Multiple edits in w2w space minimally degrade the original identity or interfere with other concepts, while maintaining edit appearance across different samples.
Evaluation protocol. We evaluate these three methods for identity preservation, disentanglement, and edit coherence. To measure identity preservation, we first detect faces in the original generated images and the result of the edits using MTCNN [74]. We then calculate the similarity of the FaceNet [57] embeddings. We also use LPIPS [76] computed between the images before and after the edit to measure the degree of disentanglement with other visual elements, and CLIP score [21], to measure if the desired edit matches the text caption for the edit.
To generate samples, fix set of prompts and random seeds which used as input to the held-out identity models. Then, we choose a set of -ppecic manipulations. For prompt-base editing, augment the attribute description to the set of fixed prompts (e.g., "chubby [v/ person"). For Concept Sliders and w2w, we apply the weight space edit directions to the personalized model with a fixed norm which determines the edit strength. The norm is calculated using the maximum projection component onto the edit direction among the training set of model weights.
w2w edits are identity preserving and disentangled. We evaluate over range of identity-specife attributes and present three (gender, chubby, narrow eyes) in Tab. 1. Edits w2w preserve the identity of the original subject measured by the ID score. These edits are semantically aligned with the desired effect as indicated by the CLIP score while minimally interfering with other visual concepts, as measured by LPIPS. We note that the CLIP score can be noisy in this setting text captions can be too coarse to describe attributes as nuanced as those related to the human face. We supplement this with user study presented in Appendix B.
Qualitatively, w2w edits make the minimal amount of changes to achieve semantic and identity- preserving edits (Fig. 5). For instance, changing the gender of the man does not significantly change the facial structure or hair, unlike Concept Sliders or prompting with text descriptions. Prompting has inconsistent results, either creating no effect or making drastic changes. Concept Sliders tends to make caricaturized effects, such as making the man cartoonishly chubby and baby-like.
Composing edits. Edit directions in w2w space can be composed linearly as shown in Fig. 6. The first column represents samples from the original model, and each subsequent column represents samples from the edited models. Each row shares the same fixed random generation seed. The composed edits persist in appearance across different generations, binding to the identity. Furthermore, the edited weights result in new model, where the subject has different attributes while still maintaining as much of the prior identity. This in contrast to editing in a traditional latent space, where an edit only corresponds to single image. Additionally, as we operate weight manifold, minimal changes are made other concepts, such as scene layout or other people. For instance, in Fig. 6, adding edits to the woman does not interfere with the person standing by her.
personalized ddentity-specifie
7

---

Inper
Inversion
CChuhby
Input
Invension
+Points Nose
Isput
Inversion
Figure 7: Single image inversion reconstructs identity and enables editing in w2w space. We present generated samples from the inverted models. These inverted identities can be composed in novel contexts and edited using our discovered semantic directions weight space. These edits persist in appearance across generation seeds and prompts.
Input
Pmjection
Input
Projection
Input
Projection
Figure 8: Projecting antto--distribution identities. We show that our inversion method can convert unrealistic identities into realistic renderings with in-domain facial features. Each image represents generated sample from the inverted model. The resulting identities can be composed in novel scenes, such playing tennis or rendered into other artistic domains.
4.4 Inverting Subjects
Evaluation protocol. We measure w2w space's ability to represent novel identities by inverting a set of 100 random FFHQ [28] face images. We follow inversion objective from eq. 3. We then provide a set of diverse prompts to generate multiple images and follow the identity preservation metric from Sec. 4.3 to measure subject fidelity, Implementation details provided in Appendix C.
We compare our results to two approaches that use Dreambooth with rank-1 LoRA. The first is trained on single image. The second is trained multiple images of each identity. We generate such images by following our identity dataset construction from Sec. 4.1. This approach can viewed as pseudo-upper bound modeling identity as it uses multiple images.
w2w space provides strong identity prior. Inverting single image into w2w space improves on the single image Dreambooth baseline and closes the gap with the Dreambooth baseline that uses multiple identity images (Tab. 2). Conducting Dreambooth fine-tuning with a single image in the original weight space leads to image overfitting and poor subject reconstruction as indicated by lower ID score. In contrast, by constraining the optimized weights to lie on a manifold of identity weights, w2w inversion inherits the rich priors of the models used to discover the space. As such, it can extract a high-fidelity identity that consistent and compositional across gener- ations. We present qualitative comparisons against Dreambooth and single-iman image- Dreambooth Appendix C. We additionally compare against other personalization methods in that section.
Inverted models are editable. Fig. demonstrates that a diverse set of identities can be faithfully represented in w2w space. After inversion, the encoded identity can be composed in novel contexts and poses. For instance, the inverted man (rightmost example) can be seen posing with a celebrity or rendered as a statue. Moreover, semantic ed- its can be applied to the inverted models while maintaining appearance across generations.
Table 2: w2w Inversion closes the gap with Dreambooth.
Method
GingleImmaee
ID Score
DB-LoRA DB-LoRA w2w
0.69 0.01
0.43 + 0.03
0.64 + 0.01
4.5 initrf-Distributtion Projection
w2w space captures distriitittiittonnnn identities. We follow the w2w inversion method from Sec. 4.4 to project images unrealistic identities (e.g.. paintings, cartoons, etc.) onto weights manifold, and present these qualitative results in Fig. 8. By constraining the optimized model to live w2w space, the inverted identities are converted into realistic renditions of the stylized identities, capturing prominent facial features. In Fig. 8, notice how the inverted identities generate similar

---

1.0-
0.7
Entanglement
0.8
1D Score
0.2
Black Hair Pale Skin
Young- Bold
0.1
w2W inversion
Roard
0.0
DB LoRA
10
# of Models
of Models
Figure 9: Scaling dataset of models further dis- Figure 10: Scaling the number of models im- entangles classifier directions. We highlight the proves identity preservation. As the span trend in disentanglement of three examples where of w2w space increases, inversion can recon- attributes may be strongly correlated among iden- struct single-image identities more faithfully, ap- tities. As the number of models is increased, the proaching the pseudo-upper bound of multi-image features are less entangled.
Dreambooth (DB-LoRA).
blonde hairstyle and nose structure in the first example, defined jawline and lip shape in the second example, and head shape and big nose in the last example. As also shown in the figure, the inverted identities can also be translated to other artistic domains using text prompts. We present variety of domains projected into w2w space in Appendix D.
4.6 Effect of Number of Models Spanning w2w Space
We ablate the number of models used to create w2w space and investigate the expressiveness of the resulting space. In particular, we measure the degree of entanglement among the edit direction and how well this space can capture identity.
Disentanglement vs. the number of models. We find that scaling the number of models in our dataset of weights leads to less entangled edit directions in w2w space (Fig. 9). We vary the number of models in our dataset of weights and reapply PCA to establish a basis. We then measure the absolute value of cosine similarity (lower is better) between all pairs of linear classifier directions found for CelebA labels. We repeat this as we scale the number of model weights used to train the classifiers. We report the mean and standard deviation for these scores, along with three notable semantic direction pairs. We observe a trend in decreasing cosine similarity. Notably, pairs such as "Black Hair Pale Skin," "Young- Bald," and "Male Beard" which may correlate in the distribution of identities, become less correlated as we scale our dataset of model weights.
Identity preservation vs. the number of models. We observe that in our dataset of weights, identities are more faithfully represented in w2w space (Fig. 10). We follow the same procedure as the disentanglement ablation, reapplying PCA to establish a basis based on the dataset of model weights. Next, following Sec. 4.4, we optimize coefficients for this basis and measure the average ID score over the 100 inverted FFHQ evaluation identities. As each model in our dataset encodes a different instance of an identity, growing this dataset increases the span of w2w space and its ability to capture more diverse identities. We plot the average multi-image Dreambooth LoRA (DB-LoRA) ID) score from Sec. 4.4, which is agnostic to our dataset of models. This establishes pseudo-upper bound on identity preservation. Scaling enables w2w represent identities given a single image with performance approaching that of traditional Dreambooth with LoRA, which uses multiple images and trains in a higher dimensional space.
scale the number of models
Extending to Other Domains
We extend our hypothesis of interpretable linear weight subspaces in diffusion models other visual concepts beyond human identities. We apply the weights2weights framework to form a subspace for models encoding different dog breeds. To create a dataset for fine-tuning, we generate images with Stable Diffusion based on each of the 120 dog classes from ImageNet [11]. We then conduct Dreambooth fine-tuning on each set of dog breed images to create dataset of 120 dog-encoding
models, subsequently applying PCA. To find edit directions, we use GPT-4 [2] to create labels for

---

Small
Large
Van
Sports Car
Figure 11: weights2weights linear subspaces be created for other visual concepts. We follow the same procedure of applying PCA and finding edit directions with linear classifiers on datasets of models encoding dog breeds and models encoding car types.
each dog breed (e.g., wavy hair or not) and then train linear classifiers on the model weight principal component projections like Sec. 3.3. We apply the same framework to different car categories using models fine-tuned on images from a dataset of 197 different car types [33]. We present results for traversing edit directions in these two subspaces in Fig. 11. Each column represents samples from an edited model. Each row shares the same fixed random generation seed.
Our results provide further evidence that diffusion models can encode visual concepts linearly. This enables the creation of new models in a controlled manner via simple interpolation. For instance, in Fig. 11, we rewrite the model's 's learned concept of a small golden retriever to make it bigger, or the model's encoding of red van to make it a sports car. Additionally, unlike older PCA-based methods [6, 53, 61, 65] which rely aligned pixels or keypoints of human faces, weights2 can extend other domains beyond human identities. We refer the reader to Appendix G for more results of applying w2w space to other visual concepts.
Limitations
Input
Inversion
As with any data-driven method, w2w space in- herits the biases of the data used to discover it. For instance, co-occurring attributes in the identity-encoding models would cause linear classifier directions to entangle them (e.g. gen- der and facial hair). However, as we scale the number of models, spurious correlations will drop as evidenced by Fig. These semantic directions are also limited by the labels present in CelebA. Additionally, the span of the w2w space is dictated by the models used to create it. Thus, w2w space can struggle to represent more complex identities as seen in Fig. 12. Inversion these cases amounts to projecting onto the closest identity on the weights manifold. Despite this, our analysis on the size of the model dataset reveals that forming space using a larger and more diverse set of identity-encodiin models can mitigate this limitation.
Figure 12: weights2weights fails capture iden- tities with undersampled attributes.
Discussion and Broader Impact
We presented paradigm for representing diffusion model weights as a point in a subspace defined by other customized models weights2weights (w2w) space. This enabled applications analogous those of a generative latent space inversion, editing, and sampling but producing model weights rather than images, resulting in what we term a meta-latent space. We demonstrated these applications on model weights encoding human identities and extended this space to other visual concepts. Although these applications could enable malicious manipulation of real human identities and model weights, we hope the community uses the framework explore visual creativity as well utilize this interpretable space for controlling models for safety.
10

---

Acknowledgements
The authors would like to thank Grace Luo, Lisa Dunlap, Konpat Preechakul, Sheng- Yu Wang. Stephanie Fu, Or Patashnik, Daniel Cohen-Or, and Sergey Tulyakov for helpful discussions. AD is supported by the US Department of Energy Computational Science Graduate Fellowship. Part of the work was completed by AD as intern with Snap Inc. YG is funded by the Google Fellowship. Additional funding came from ONR MURI.
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVE international conference on computer vision, pages 4432-4441, 2019.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal. Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint vrXiv:2303.08774 2023.
[3] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and An- tonio Torralba. Semantic photo manipulation with generative image prior. ACM Transactions on Graphics (TOG), 38(4):1-11, 2019.
[4] David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations, 2019.
[5] Stefan Andreas Baumann, Felix Krause, Michael Neumayr, Nick Stracke, Vincent Tao Hu, and Björn Ommer. Continuous, subject-ppecific attribute control t2i models by identifying semantic directions. arXiv preprint arXiv:2403, /7064, 2024.
[6] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '99, page 187-194, USA, 1999. ACM Press/Addison-Weed
Publishing Co.
[7] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560-
22570, 2023.
[8] Anton Cherepkov, Andrey Voynov, and Artem Babenko. Navigating the gan parameter space for semantic image editing. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 3671-3680, 2021.
[9] Timothy F Cootes, Gareth Edwards, and Christopher Taylor. Active appearance models. In Computer Vision-ECCV'9S 5th European Conference on Computer Vision Freiburg, Germany, June 2-6, 1998 Proceedings, Volume 5, pages 484-498 Springer, 1998.
[10] Yusuf Dalva and Pinar Yanardag, Noisecir: A contrastive learning approach for unsupervised discovery interpretable directions diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 24209-2421 2024.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[12] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2016.
[13] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravee, Zac Hatfield-Dode Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposition. arXiv preprint 200.222209990022.... 2022.

---

[14] Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14300-14310, 2023.
[15] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[16] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022.
[17] Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint 1Xiv:2311.12092 2023.
[18] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[19] David Ha, Andrew M Dai, and Quoc V Le. Hypernetoorks. In International Conference on Learning Representations, 2016.
[20] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances neural information processing systems, 13:9841-98515 2020.
[21] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1116840-6811, 2020.
[23] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[24] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2022.
[25] Minguk Kang, Jun- Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 10124-10134, 2023.
[26] Tero Karras, Miia Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 13:12104-1211 2020.
[27] Tero Karras, Miia Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in neural information processing systems, 34:852-863, 2021.
[28] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[29] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110 8119, 2020.
[30] Diederik P Kingma and Jimmy Ba. Adam: a method for stochastic optimization. In International Conference Learning Representations, 2014.
12

---

[31] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Confer- ence on Learning Representations, 2014.
[32] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances neural information processing systems, 31, 2018.
[33] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[34] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[35] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In The Eleventh International Conference on Learning Representations, 2022.
[36] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 1730--738, 2015.
[37] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[38] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Os Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038 6047, 2023.
[39] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296 4304, 2024.
[40] Yotam Nitzan, Kfrr Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. ACM Transactions on Graphics (TOG), 110: 2022.
[41] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 36:24129-24142 2023.
[42] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint N..iv:2229.188999 2022.
[43] Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit Bermano, Eric Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. In Computer Graphics Forum, volume 43, page e e15063. Wiley Online Library, 2024.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8778 8763. PMLR, 2021.
[45] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 151 1106444,, 2015.
[46] Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In Proceedings of the IEEF/CVE international conference on computer vision, pages 1882-1890, 2019.
13

---

[47] Mija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428- 10436, 2020.
[48] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, 2020.
[49] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMIR, 2015.
[50] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286 PMLR, 2014.
[51] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on graphics (TOG), 42(1):1-13, 2022.
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[53] Duncan A Rowland and David Perrett. Manipulating facial appearance through shape and color. IEEE computer graphics and applications, 15(5):00-76, 1995.
[54] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-223 22510, 2023.
[55] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingho Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personaliza- tion of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6527-6536, 2024.
[56] Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2023.
[57] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 815-823, 2015.
[58] Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems, 34:1644 14:16411-16993 2021.
[59] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311.13600 2023.
[60] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1243-9252, 2020.
[61] Lawrence Sirovich and Michael Kirby. Low-dimensional procedure for the characterization of human faces. Josa a, 4(3):519-555 524, 1987.
[62] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[63] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2020.
14

---

[64] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 1014) 2021.
[65] Matthew Turk and Alex Pentland. Eigenfaces for recognition. Journal of cognitive neuroscience, 3(1):71-86, 1991.
[66] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International conference on machine learning, pages 9786-9796. PMLR, 2020.
[67] Kuan-Chich Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, and Kfir Aberman. Moa: Mixture-of-attention for subject-context disentanglement in personalized image generation. arXiv e-prints, pages FeXiv-2240 2024.
[68] Kuan-Chieh Wang, Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel. Adver- sarial distillation of bayesian neural network posteriors. In International conference on machine learning, pages 5190-5199. PMLR, 2018.
[69] Mitchell Wortsman, Gabriel Ilharco. Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 2022.
PMLR,
[70] Weihao Xia, Yulun Zhang, Yujiu Yang. Jing-Hao Xue, Bolei Zhou, and Ming-Hsuaa Hsuan Yang. Gan inversion: A survey. IEEE transactions on pattern analysis and machine intelligence, 45(3):3121-3138, 2022.
[71] Guangxuan Xiao, Tianwei Yin, William Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-fres multi-subject image generation with localized attention. arXiv preprint arXiv:23005. 10431, 2023.
[72] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0677 067.21,,.7,, 2023.
[73] Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi. Xintao Wang, Ying Shan, and Huicheng Zheng. Inserting anybody in diffusion models via celeb basis. Advances in Neural Information Processing Systems, 36, 2024.
[74] Kaipeng Zhang, Zhanpeng Zhang. Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(1)):1499 1503, 2016.
[75] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[76] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[77] Jun- Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 597-613 Springer, 2016.
[78] Ye Zhu, Yu Wu, Zhiwei Deng, Olga Russakovsky, and Yan Yan. Boundary guided learning-free semantic control with diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
15

---

A Sampling
We present additional examples of models sampled from w2w space in Fig. 13. The sampled models encode a diverse array of identities which are not copied from the dataset of model weights, seen by comparing them to the nearest neighbor models from the training set. However, there are attributes borrowed from the nearest neighbors which visually appear in the sampled identity. For instance, the sampled man the first row shares a similar jawline to the nearest neighbor identity. The sampled identities also demonstrate the same ability the original training identities to be composed into novel contexts. A variety of prompts are used in Fig. 13 and the identities are still consistent.
Sampled Identity
Nearest Neighbor
models from w2w space and their nearest neighbor models. The sampled identities share some characteristies with the nearest neighbors, but are still distinct. These identities can also be composed into novel contexts like a standard customized diffusion model.
Figure 13: Sampled
16

---

B Model Editing in w2w Space
Qualitative Results. We display additional examples of applying edits in w2w space based on the directions discovered using linear classifiers and CelebA labels. In Fig. 14, we demonstrate how the strength of these edits can be modulated and combined with minimal interference. These edits are apparent even in more complex scenes beyond face images. Also, the edits do not degrade other present concepts, such as the dog near the man (top left example).
In Figs. 15 and 16, we demonstrate how multiple edits can be progressively added in disentangled fashion with minimal degradation to the identity. Additionally, since operate in subspace of weight space, these edits persist with consistent appearance across different generations. For instance, even the man exhibits the edits as a painting in Fig. 15.
Bald
Chubby
Black Hair
Beard
Pointy Nose
Age
Curis
Pale
Figure 14: Multiple edits can be controlled continuous manner.
17

---

Original
+Flat Brows
+Bangs
+Straight Hair
+Old
Original
+Chubby
+uustacee
+Big Eyes
+Balding
ETHIRBME
Figure 15: Composing four different edits with minimal identity degradation. These edits bind to the identity and persist in appearance across multiple generation seeds and prompts.
18

---

Original
+Pale
+Small Lips
+Male
Original
+Pointy Nose
+Eye Bags
+Big Lips
Original
Chheekbones
+Curis
+Thick Brows
Figure 16: Additional examples of composing multiple edits. We provide more examples of semantic edits based on labels available from CelebA.
19

---

User Study. We present two-alternative forced choice (2AFC) user study to evaluate the quality of identity edits Tab. 3. Twenty-five users were given ten sets of images. Each contained randomly sampled original image of an identity, and then an image of that identity edited for an attribute using Concept Sliders [17], w2w, and text prompting. Users were then asked to choose between alternate pairs based on three criteria: identity preservation, alignment with the desired edit, and disentanglement. Our results in Tab. show that users have a strong preference towards w2w edits. User instructions and an example question from the study are provided in Figs. 17, 18.
Table User study on identity editing.
Method
Win Rate (%) f
Sliders w2w
28.4
71.6
Prompting w2w
12.8
87.2
Comparing Edits
Read the following instructions.
page, will original identity, followed by attribute edited for that identity using three different methods. You will be asked different questions. Each question will ask you to choose between two of the edit methods. The top of each section will provide the edit description. Choose which image satisfies the most of these three criteria:
Semantic Alignment. The image aligns with textual description for the edit (e.g. "chubby" should mean that the edited identity now looks chubby).
Identity Preservation. The image preserves the original identity. other words, the edited identity looks recognizable compared the original. Some things look for are: eye shape, nose shape, face shape (e circular or oval-llke), skin tone, eyebrow shape, hair color.
Disentanglement. The edit minimally interferes with the rest of the image (t..
diseetangled does not affect clothes, background, etc.)
Figure 17: Instructions provided to users for the identity editing user study.
Between Edit #1 and Edit #2, which most satisfies semantic alignment (chubby).
identity preservation, and disentanglement? Zoom for better inspection.
Original
Edit #1
Edit #2
Edit #3
Edit #1
Edit #2
Figure 18: Example question from the identity editing user study.
20

---

C Inversion
We present additional details on w2w inversion and comparisons against training Dreambooth LoRA on single image vs, multiple images.
To conduct w2w inversion, train single image following the objective from eq. 3. We qualitatively find that optimizing 10,000 principal component coefficients balances identity preservation with editability. This discussed in Appendix F. We optimize for 400 epochs, using Adam [30] with learning rate 0.1, Bn = 0.9, B2 = 0.999 and with weight decay factor 1e-10. For conducting Dreambooth fine-tuning, follow the implementation from Hugging Face using LoRA with rank 1. To create dataset of multiple images for an identity, we follow the procedure from Sec. 4.4.
Implementation Details:
w2w inversion is more efficient than previous methods. Inversion into w2w space results in a significant speedup in optimization seen in Tab. 4, where we measure the training time single NVIDIA A100 GPU. Standard Dreambooth fine-tuning operates on the full weight space and incorporates an additional prior preservation loss which typically requires hundreds of prior images. In contrast, we only optimize a standard denoising objective on a single image within a low-dimensional weight subspace. Despite operating with lower dimensionality, w2w inversion performs closely to standard Dreambooth fine-tuning on multiple images with LoRA.
Table 4: Inversion into w2w space balances identity preservation and efficiency.
Method
# Param
Opt. Time (s)
Identity Fidelity f
DB-LoRA
99,648
220
0.69 0.01
DB-LoRA
99,648
200
0.43 = 0.03
w2w Inversion
10,000
55
0.64 + 0.01
Qualitative Inversion Comparison. In Figs. 19 and 20, we present qualitative comparisons w2w inversion against Dreambooth trained with multiple images and a single image. Although mult-image Dreambooth slightly outperforms w2w inversion in identity preservations, its samples tend to lack realism compared to w2w inversion. We hypothesize that this may be due using generated images for prior preservation and training on synthetic identity images. Dreambooth trained on a single image either generates an artifacted version of the original image or random identities. Notice how inversion into w2w space is even able to capture key characteristies of the child although babies are nearly completely absent the identites based on CelebA used fine-tune our dataset of models.
Single Image DB-LORA
w2w Inversion
Input
DB-LoRA
Figure 19: Inversion into w2w space preserves identity and realism. We compare against Dream- booth fine-tuning with LoRA multiple images and single image.
21

---

Single Image DB-LORA
w2w Inversion
Input
DB-LORA
Single Image DB-LORA
w2w Inversion
Input
DB-LoRA
PES
Single Image DB-LORA
w2w Inversion
Input
DB-LoRA
Figure 20: Inversion into w2w space preserves identity and realism (cont.).
22

---

Comparison against Single Image Personalization Methods. We compare w2w inversion to single shot personalization methods Celeb-Basis [73] and Adddter FacelD* [72], following the same evaluation protocol from Sec. 4.4. These quantitative results are presented in Tab. 5. While Celeb-Bast optimizes in the input text embedding space for given face image, IP-Adapter trains light-weight adapters [39, 75] to condition generation on any input face image.
We further conducted two-alternative foreed choice (2AFC) user study the perceptual quality of w2w identity preservation. Twenty users were given ten sets of images. Each set contained randomly sampled original image of the identity, and then three random images generated using IP- Id-Attreer FacelD, Celeb-Bate Basis, and w2w with the same random prompt. Users were then asked choose between alternate pairs based on three criteria: identity preservation, prompt alignment, and diversity of generated images. Our results in Tab. 6 show that users found generations from w2w models capture identity better while also generating more diverse images that better align with the prompts.
Across both these metrics, w2w performs stronger than Celeb-Basis and IP-Adapter FaceID. Our results indicate that operating in our weight subspace is highly expressive and flexible as it is able to faithfully capture nuanced identity without overfitting to the input image. For instance, Fig. 21, w2w inversion enables diverse generation with various poses, facial expressions, and clothing while maintaining identity.
Table 5: Single-shot personalization com- parison.
Table 6: User study on identity inversion.
Method
Win Rate (%) 1
Method
ID Score
Celeb-Basis
13.4
Celeb-Basis
0.60 0.02
w2w
86.6
IP-Adapter FaceID
0.62 = 0.02
IP-Adapter FacelD
22.8
w2w
0.64 + 0.01
w2w
77.2
Original
"A photo of V, person"
Celeb- Celeb-Basis Basis IP-Adepter
w2w
Figure 21: Qualitative comparison of single-shot personalization methods.
23

---

Below in Figs. 22 and 23, we provide the instructions provided to users for the identity inversion user study in addition to an example question.
Comparing Identities
Read the following instructions. each page, there will original identity, followed three sets images trying prompt.
be asked different questions. question will ask you to choose between two of the methods. The top of each section will provide the prompt used to generate the images. Choose which set of images satisfies the these three criteria reasonable degree: Identity Preservation. edited identity still looks recognizable compared to the original. Some things to look out for are: shape, nose shape, face shape (e.g. circular or oval-like), skin tone, eyebrow shape, color. Try not biased by image quality, alignment of the pose, facial expression.
identity generated
images preserve the original identity. other words, the
2. Prompt Alignment. The set of images should satisfy the prompt used to generate that set of images (e.g.. the prompt "playing tennis outdoors," the set of images should satisfy that).
Image Diversity. The set of images should not just copy the original Image. There should be variety of poses, backgrounds, clothes, and facial expressions (if the prompt does not include those).
Figure 22: Instructions provided to users for the identity inversion user study.
Given the following identity, choose the option that satisfies of these three criteria to a reasonable degree 1) Identity preservation 2) Prompt alignment ("sad") 3) Image diversity. Zoom better inspection.
Original
1)
2)
3)
2
Figure 23: Example question from the identity inversion user study.
24

---

D Out of Distribution Projection
Additional examples of out-of-distribution projections are displayed in Fig. 24. A diverse array of styles and subjects (e.g. paintings, sketches, non-humans) can be distilled into a model in w2w space. After embedding an identity into this space, the model still retains the compositionality and rich priors of a standard personalized model. For instance, we can generate images using prompts such as "/v/ person writing desk" (top example), "/v/ person with a dog" (middle example), or "a painting of [v/ person painting on canvas" (bottom example).
Input
Projection
Input
Projection
Input
Projection
Figure 24: Projection into w2w space generalizes to a variety of inputs. A range of styles and entities can be inverted into realistic identity in this space. Once model is obtained, it can be prompted to generate the identity in a variety of contexts.
25

---

E Identity Datasets
In Fig. 25, we present examples of synthetic identity datasets used conduct our Dreambooth fine-tuning discussed in Sec 4. Each dataset is a set of ten images generated with [67] conditioned on single CelebA [36] images associated with binary attribute labels. Note that we only display subset of images per identity in the figure. The same identity can occur multiple times in different images in CelebA but have different appearances. So, creating these synthetic datasets reduces intra-datase diversity and creates more consistent appearance for each subject. For instance, the first two rows the figure are the same identity, look drastically different. So we instead treat them as different identities associated with different set of images.
For evaluating identity edits from Sec. 4.3, we hold out 100 identities, which results in leaving out 1000 models since multiple models may encode different instances of the same identity. For instance, if we left out the model encoding the identity in the first row of Fig. 25 for evaluation, the model encoding the second row identity would also be left out since it encodes the same identity but a different instance.
CelebA Prior
Generated Dataset Examples
Figure 25: Fine-tuning synthetic examples allows Dreambooth fine-tuning to distill sistent identity. The left column shows a CelebA image used to condition generation of set of identity-consistett images in the right column associated with that identity using [67]. The consistent appearance of the identity enables a more consistent identity encoding.
26

---

F Principal Component Basis
In this section, we analyze various properties of the Principal Component (PC) basis used define w2w Space. We investigate the distribution of PC coefficients and the effect of the number of PCs on identity editing and inversion.
Distribution of PC Coefficients. We plot the histogram of the coefficient values for the first three Principal Components Fig. 26. They appear roughly Gaussian. Next, we rescale the coefficients for these three components to unit variance for visualization purposes. We then plot the pairwise joint distributions for them in Fig. 27. The circular shapes indicates roughly diagonal covariances. Although the joint over other combinations of Principal Components may exhibit different properties, these results motivate strategy from Sec. 3.2.
model the PCs as independent Gaussians, leading to the w2w sampling
Number of Principal Components for Identity Editing We empirically observe that training classifiers based on the 1000 dimensional PC representations (first 1000 PCs) of the model weights results in the most semantically aligned and disentangled edits directions. We visualize comparison for the "goatee" direction in Fig. 28. After finding the direction, we calculate the maximum projection component onto the edit direction among the training set of model weights. This determines the edit strength. As seen in the figure, restricting to the first 100 Principal Components may be too coarse to achieve the fine-grained edit, instead relying on spurious cues such skin color. Training with the first 10,000 Principal Components suffers from the curse of dimensionality and the discovered direction may edit other concepts such as eye color or clothes. Finding the direction using the first 1000 Principal Components achieves the desired edit with minimal entanglement with other concepts.
Number of Principal Components for Identity Inversion We qualitatively observe that inversion using the first 10,000 Principal Components balances identity preservation while not overfitting to the source image. We visualize comparison in Fig. 29, where each column has fixed seed and prompt. Optimizing with the first 1000 PCs underfits the identity and does not generate a consistent identity. Inversion with the first 20,000 Principal Components overfits to the source image of face shot, which results in artifacted face images despite different generation seeds and prompts. Optimizing with the first 10,000 Principal Components enjoys the benefits of lower dimensional representation than the original LoRA parameter space (~100,000 trainable parameters), while still preserving identity and compositionality. This supported quantitatively by Fig. 30, which shows the average ID score for 100 inverted FFHQ identities optimized over a varying number of principal components.
1st Principal Component
2nd Principa Component
3rd Principa Component
Sn Count
Bn
Figure 26: Histogram of principal component coefficients. The first three principal component coefficients appear approximately Gaussian.
2
2nd PC
1st PC
3rd
1st
2nd PC
3rd
Figure 27: Pairwise joint histogram of principal component coefficients. We rescale the first three principal component coefficients and plot the pairwise joint distributions for visualization purposes. Given that the marginals are roughly Gaussian, the circular appearance of the joint suggests pairwise independence for the first three components.
27

---

+ Goatee
Original
100 PCs
1000 PCs 10000 PCs
Figure 28: Edit results with varying number of Principal Components. Training classifiers to find semantic weight space directions with the first 1000 Principal Components achieves the most semantically aligned and disentangled results.
Input
Inversion with 1000 PCs
Inversion with 10000 PCs
Inversion with 20000 PCs
Figure Identity inversion results with varying number principal components. We optimize the coefficients for the first 1000, 10, 000, and 20, 000 Principal Component. Each column indicates a fixed generation seed and prompt. Inversion with the first 10, 000 components balances parameter efficiency, realism, and identity preservation without overfitting to the single image.
0.6
0.5
ID Score
0.4
0.3
0.2
0.1
0.0
102
103
104
Principal Components
Figure 30: Identity preservation vs. number of principal components used for w2w inversion. We optimize the coefficients for the first N principal components to 20,000 and measure the average ID score for 100 inverted FFHQ identities.
28

---

Visualizing Principal Components. We provide visualization of traversals along set of principal components in Fig. 31. The principal components change attributes of the identity, although various semantic attributes are entangled. For instance, the first PC appears to change age, hair color, and hair style. The second PC appears to change gender and skin complexion. The third PC seems to change age, skin complexion, and facial hair. This motivates our use of linear classifiers to find separating hyperplanes in weight space and disentangle these attributes.
st PC +
2nd PC +
-rd PC +
Figure 31: Traversal along the first three principal components in w2w space. The directions encode various entangled identity attributes such as age, gender, and facial hair.
29

---

G weights2weighhts for Other Visual Concepts
We find that similar subspaces can be created for other visual concepts beyond human identites. For instance, we apply the weights2weights framework to create two subspaces for models encoding different dog breeds and models encoding car types. We present examples of editing these models in Fig. 32. This suggest the generality of weights2weights and linear subspaces within diffusion model weights.
Large
Wavy Fur
SUV
Light Color
Figure 32: Applying w edits on dog-encoding models and car-encoding models.
We create two datasets of model weights and apply PCA to define two separate weight subspaces. We then train linear classifiers to find semantic edit directions.
30

---

H Multi-Conceptt Merging
Multiple models living in w2w space cannot be merged since they live in the same weight subspace. So, merging will lead to interpolation of the identities. However, w2w models can be merged with models lying approximately orthogonal to the subspace. This merging can be done adding the weights. We present an example of merging a model living in w2w space with model fine-tuned to encode "Pixar" style Fig. 33.
A photo of V, person
A photo of V, person in V style
Figure 33: Merging w2w models with non-identity models. Here, another model is fine-tuned to map 1/2 to "Pixar" style. The two models are merged with simple addition.
I Timestep Analysis
Edits in w2w space correspond to identity edits with minimal interference with other visual concepts. Although not focus, image editing is achieved as byproduct. For further context preservation, edits w2w Space can be integrated with delayed injection [7, 17, 35, 71], where after T timesteps, the edited weights used instead of the original ones. We visualize this in Fig. 34. Larger T in the range [700, 1000) are helpful for more global attribute changes, while smaller [400, 700) can be used for more fine-grained edits. However, by decreasing the timestep T, the strength of the edit is lost in favor of better context preservation. For instance, the dog's 's face is better preserved in the second row T 600, although the man is not as chubby compared to other T.
+Chubby
Original
7 10100
7-800
7=600
Figure 34: Injecting edited weights at varying timesteps. Using the edited weights smaller timestep T better preserves context at the expense of edit strength and fidelity.
31

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
Experiments in the paper back up the claims made in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: There is dedicated limitations section in Sec. 6 detailing failure cases. Computational efficiency addressed in Sec. and scale is addressed in Sec. 4.6. Privacy and fairness is addressed in Sec. 7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
32

---

Answer:
[NA]
Justification:
There are no theoretical results in the submission.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Secs. 3, 4 and the Appendix detail all necessary steps reproduce the results. We plan to release the model weights we trained.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Code, data, and weights can be found through the project page at: https: // seaannneeeeeee github. htw.eiigttttwwihhss//
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
ceabbbmisssoooooooo wwwsssololiooooooooooooooooollol icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/
wttp//ggueeeooooooooo Suoee//mmdmSeiiicaaa Saneeuuumisssss wesssssssoosoooooooo innoolccy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present data splits, hyperparameters, etc. in Sees. 3, 4, and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Confidence intervals are reported for the results in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The appendix details the compute resources used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics atttp::/neeurie http:///eeeurppscccpoieieaee petpulicc/thissuuiideeeoo
Answer: [Yes]
Justification: The authors have read the NeurIPS Code of Ethics and confirm that the paper conforms with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts has dedicated section in Sec. 7.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work utilizes pretrained models and well-established datasets that already have proper safeguards in place.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The original creators of assets are either cited or linked.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No assets being submitted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Screenshots and instructions are provided for both identity editing and identity inversion user studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Potential risks are not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "Re3BBQzE3G", "reviewer": "Reviewer_aBHC", "summary": "This paper explores the latent space of weights in customized diffusion models, introducing the weights2weights (w2w) space, a subspace encoding different human identities. By fine-tuning over 65,000 models, each representing a distinct human identity, the authors model this weight space using low-rank adaptation (LoRA) and principal component analysis (PCA). The key contributions include demonstrating three applications of the w2w space: Sampling: Generating novel identities by sampling from the weight space; Editing: Performing semantic edits, such as adding a beard, by traversing linear directions in the weight space; Inversion: Reconstructing a realistic identity from a single image, even if it is out-of-distribution. The paper validates the w2w space’s expressiveness through quantitative evaluations and qualitative observations.", "questions": "1) I'm curious if style information can be decoupled via w2w space in addition to attributes. I would suggest that the authors could add the results of style editing to their paper.\n\n2) How does w2w perform on multi-concept generation?", "limitations": "The paper discussed the limitations of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The w2w space is interesting because it shifts the focus from traditional pixel or feature spaces to the model’s weight space, allowing for new forms of manipulation and analysis.\n\n2) The paper is well-organized and clearly written, with detailed explanations of the proposed methods and their theoretical underpinnings.", "weaknesses": "1) The overall idea of this paper is more similar to [1], the significant difference I see so far is that [1] does PCA on token embedding and w2w does PCA on Lora in Unet. I suggest the authors could discuss more about the differences between this paper and [1]. Also, I have some methodological doubts:\n\nA. does w2w have a similar phenomenon to mean person in [1]? (see Figure 2 in [1])\n\nB. Have the authors studied the effect of PCA on the generation of results? (see section 4.3 in [1])\n\n[1] Inserting Anybody in Diffusion Models via Celeb Basis\n\n2) While this paper provides new ideas and insights into personalized generation models, I am concerned about his real-world application, what are the advantages of w2w compared to existing personalized generation efforts such as IP-adapter, InstabntID, custom diffusion, Mix-of-show, etc.? Faster generation or smaller training parameters? The experiments also don't give a comparison with existing personalization generation efforts, however, I think this is important to note", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**“The overall idea of this paper is more similar to [1]...I suggest the authors could discuss more about the differences between this paper and [1].”**\n\nAlthough these two papers share the use of PCA and the application to personalization, there are fundamental differences. Many works have found subspaces in the *input space*, such as text embeddings [1], to produce meaningful linear interpolations. But as you point out, we use PCA to discover a subspace within the *weights themselves*, suggesting that weights can also be interpretable spaces. Our analysis of the model weights shows that we can get valid identity-encoding diffusion models via linear interpolation among the weights of existing diffusion models spanning the subspace. For instance, our *w2w* inversion essentially finds the best linear combination of existing diffusion model weights to define a **new customized diffusion model**, which, in turn, can generate an unlimited number of identity-consistent images of that novel person. \n\nFurthermore, beyond personalization, which is the main focus of [1], we demonstrate two additional applications of this subspace. We also develop methods for sampling and editing identities, which “expands the scope of manifold applications” (rzPz). We show how to **sample new models** each of which **encodes a novel identity** (Sections 3.2, 4.2), and **edit the models** (Sections 3.3, 4.3), which as a result, **edits the identities** encoded in them. As such, *weights2weights* space can be thought of as a **meta**-latent space as its applications are “analogous to those of a generative latent space – inversion, editing, and sampling – but producing model weights rather than images” (L316-318 in paper). We will make sure to clearly highlight these differences with Celeb-Basis in the related works section of the paper, where we briefly introduced Celeb-Basis. \n\n\n**“...similar phenomenon to mean person in [1]?”**\n\n*w2w* has a mean person, as shown in rebuttal Figure 5. These images are generated by a “mean model” using the averaged weights of models spanning *w2w* space.\n\n**“Have the authors studied the effect of PCA on the generation of results?”**\n\nThank you for the suggestion. We present an ablation on the number of principal components used during *w2w* inversion in rebuttal PDF Figure 1. We vary the number of principal components used during identity inversion, and measure the average ID score (higher is better) over 100 inverted FFHQ evaluation identities. The ID score monotonically increases until 10,000 principal components, after which it starts to decrease. We qualitatively visualize this phenomenon in rebuttal Figure 2, where each column represents a fixed seed. 20,000 PCs overfits to the input image and pose, with varying seeds producing only face shots with artifacts which degrade the identity. 1000 PCs underfits the original identity, but shows diversity across different generation seeds. 10,000 PCs fits the identity without overfitting. \n\n**...I am concerned about real-world applications… what are the advantages of w2w compared to existing personalization efforts…”**\n\nWe present comparisons against variants of Dreambooth in Table 2 in the main paper, showing how given only a single image, *w2w* can achieve competitive performance with a method which uses multiple images. Following your suggestion, we compare quantitatively against other personalization efforts, Celeb-Basis [1] and IP-Adapter FaceID [2], in rebuttal Table 1. We follow the same evaluation protocol used for Table 2 in the main paper. We further conduct a user study presented in Table 2 of the rebuttal PDF. We show qualitative comparisons against these two methods in rebuttal PDF Figure 6. Details are provided in Global Response 2 and 3. \n\nOverall, across all these metrics, *w2w* performs stronger than Celeb-Basis and IP-Adapter FaceID. Our results indicate that operating in our weight subspace is highly expressive and flexible compared to text embedding space as it is able to faithfully capture nuanced identity without overfitting to the input image. Based on the user study, users found *w2w* generations capture identity better while also generating more diverse images that better align with the prompts. Compared to other learning based approaches like Celeb-Basis, *w2w* is more lightweight as training takes less than a minute with 8GB on a single NVIDIA A100 GPU, while Celeb-Basis takes 21GB and trains a costly MLP in three minutes on a single A100. We present details on the efficiency of *w2w* in Appendix C.\n\nWe would like to note that previous personalization methods such as IP-adapter [2] or Celeb-Basis [1] only demonstrate identity inversion. As we mentioned earlier, we create a unified space for identity-encoding weights with applications of inversion, editing, and sampling identity-encoding models.\n\n**“I’m curious if style information…” and “How does w2w perform on multi-concept generation?”**\n\nThank you for suggesting these creative applications! Since the models used to create *w2w* space are fine-tuned on realistic identities, style information, such as cartoon or painting, would not be encoded in any principal components. However, models from *w2w* space can be merged with other personalized models for multi-concept generation. In Figure 7 of the rebuttal PDF, we train a LoRA with Dreambooth for Pixar style, and merge it with a *w2w* model to showcase multi-concept generation ability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kRNpWW6Wnv", "reviewer": "Reviewer_rzPz", "summary": "This paper constructs a dataset that contains LoRA models trained on images of 60,000 different people. A weight manifold is determined based on the parameters of these 60,000 models using PCA. Sampling, editing, and inversion can be performed on this manifold, and the rationality of this manifold is demonstrated.", "questions": "Refer to the Weaknesses 1 and 2.", "limitations": "Yes, limitations are discussed in this paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper constructs a dataset that contains LoRA models trained on images of 60,000 different people.\n2. This paper demonstrates the existence of a LoRA model manifold that encapsulates identity information and enables the generation of new identities by sampling from this manifold.\n3. The paper also presents methods of editing and inversion, expanding the scope of manifold applications.", "weaknesses": "1. There are some recent works present the way of generating identity preserved images given only one reference image. It is better to provide the comparison between this paper and these works.\n2. LoRA training often affects the performance of the base model to some extent, such as small generation variance. More results to examine this effect will be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**“...some recent works present the way of generating identity…given only one reference image…provide the comparison between this paper and these works.”**\n\nThank you for your suggestion. In the rebuttal PDF Table 1, we compare against Celeb-Basis [1] and IP-Adapter FaceID [2], following the same evaluation protocol used for Table 2 in the main paper. We further conducted a user study comparing with [1] and [2], which is presented in Table 2 of the rebuttal PDF. We show qualitative comparisons against these two methods in rebuttal PDF Figure 6. Details are provided in Global Responses 2 and 3. Across all these metrics, *w2w* performs stronger than Celeb-Basis and IP-Adapter FaceID. Our results indicate that operating in the *weights2weights* weight subspace is highly expressive as it is able to faithfully capture identity without overfitting to the input image. This enables diverse generation of the identity. \n\n\n**“LoRA training often affects the performance of the base model to some extent…”**\n\nWe are aware of this phenomenon known as “drift” as introduced in Dreambooth [4]. We agree with your concern of LoRA training affecting the base model, but we believe this is not necessarily an issue of *weights2weights* itself, but fine-tuning in general. Section 4.3 in Dreambooth [4] and Section 4.4 in Custom Diffusion [5] already run experiments indicating that the use of prior-preservation regularization limits the undesirable change of the base model. That is why we use the prior preservation loss introduced by Dreambooth to avoid losing the priors of the base model. Furthermore, the *w2w* models inherit the rich priors of the base model as shown by composition of the identities in novel contexts such as posing with a famous figure in Figures 6, 7 of the paper. These identities can also be converted into new styles with prompts like “statue” or “painting”  as seen in Figures 7, 8 in thr paper. More examples demonstrate this compositionality throughout the qualitative examples in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RooEBzmL6U", "reviewer": "Reviewer_wMSF", "summary": "The paper investigates the weight space spanned by a large collection of customized diffusion models, proposing a novel subspace termed weights2weights (w2w). The study populates this space with a dataset of over 60,000 models, each fine-tuned to encode a different person's visual identity. The paper demonstrates three main applications of this space: sampling novel identities, editing existing identities, and inverting a single image to reconstruct realistic identities, showing that these weight spaces can serve as a highly interpretable latent identity space.", "questions": "Please refer to the weaknesses above.", "limitations": "The authors have discussed the limitations of this work.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The concept of modeling a manifold over customized diffusion model weights is novel and shows potential for broad applications in generative modeling.\n\n2. The PCA and LoRA are used to construct the w2w space. Furthermore, the PCA can be used to edit the image attributes. The method is novel and effective.\n\n3. The paper includes extensive experiments that demonstrate the effectiveness of the proposed w2w space in generating, editing, and inverting identities.", "weaknesses": "1. The construction of w2w space needs over 6000 weights and the number of the model can further improve the ID score.  The computing resources should be provided. And is there a contained solution to balance computing resources and performance?\n\n2. The paper uses several evaluation metrics (e.g., identity preservation, LPIPS, CLIP score), some metrics like CLIP score can describe the quality of image generation, but might not fully evaluate the nuanced attributes related to human faces. \n\n3. In the weights manifold, new models can be sampled to find directions that separate identity properties. However, are the first m principal components all identity attributes, and do they correspond one-to-one?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**“The construction of w2w spaces needs over [60,000 models]...The compute resources should be provided…”**\n\nTraining a single identity LoRA with rank 1 for our dataset of models requires ~8GB VRAM, and takes 220 seconds on a single A100 GPU. Please refer to Appendix C for details. \n\n**“And is there a contained solution to balance computing resources and performance.”**\n\nThanks for the great question. We ablate the number of models used to construct *w2w* space in Figure 9 of the paper, and find that a subspace spanned by 10,000 models retains most of the performance in ID preservation with a score of 0.53 versus 0.65. The score of 0.65 is achieved by using more than six times the number of models. We highlight that although creating this space requires a notable amount of compute, using the space for the variety of applications is lightweight. For instance, inverting an identity only requires one image and uses 1/10 the number of parameters (just 10000) and 1/4 the time (under 55 seconds) for typical Dreambooth LoRA finetuning (see Appendix C). \n\n**“The paper uses several evaluation metrics…nuanced attributes related to human faces”**\n\nWe agree with your point on the limitations of the reported metrics, acknowledging it in L226-227: “...score can be noisy...too coarse to describe attributes as nuanced as those related to the human face.” We utilize these metrics since they are commonly adopted for evaluating edits and identity preservation. Following your point, we present user studies for identity inversion in Table 2 and identity editing in Table 3 of the rebuttal PDF. Overall, *w2w* is largely preferred over baselines. We discuss this further in Global Responses 3 and 4. \n\n**“...are the first m principal components identity attributes…?”**\n\nThat is an interesting question! Thank you for bringing it up. We provide a visualization of traversals along a set of principal components in Figure 4 of the rebuttal. The principal components change attributes of the identity, although various semantic attributes are entangled. For instance, the first PC seems to be changing age, hair color, and hair style. The second PC appears to change gender and skin complexion. The third PC seems to change age, skin complexion, and facial hair. This motivates our use of linear classifiers to find separating hyperplanes in weight space and disentangle these attributes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "saqjrFEabs", "reviewer": "Reviewer_215A", "summary": "The paper proposes to learn a manifold over customized diffusion model weights as a subspace for interpretable downstream applications such as sampling, editing, and inversion.  At the technological level, the authors collect the weights of over 60,000 models as the dataset and fine-tune those weights via LoRA to constitute the desired editing space. Experiments are conducted on CelebA dataset.", "questions": "Please see my comments in the Weaknesses section. \n\nI have mixed feelings about this paper. On the one hand, the paper risks over-claiming on some high-level motivations as pointed out in the Weaknesses section, lacking a clear positioning concerning several important related works. The methodological level design is also not entirely new and intriguing. On the other hand, I acknowledge that using model weights as datasets is interesting, and the efforts of “scaling up” to over 60.000 models seem to be one of the biggest contributions to me.", "limitations": "The limitations and broader societal impact are discussed in the main paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well organized and easy to follow, with reasonable motivation and intuitions. \n\n- The idea of using weights as the dataset to constitute an editing space is interesting.\n\n- Among the experimented application scenarios, while editing and inverting have been widely studied in the field (see Weaknesses), the exploration of the OOD projection seems more interesting to me.", "weaknesses": "- The paper lacks a comprehensive comparison with prior literature, resulting in inaccurate claims on “Alas, in multi-step generative models, like diffusion models, such a linear latent space is yet to be found”. In [a], it has been explicitly and theoretically proven such interpretable and linear latent space can be found in frozen DMs, without the need to fine-tune the model parameters, neither the DreamBooth tuning nor LoRA-based. Recent studies [2,3] have extended these findings for customization tasks and latent diffusion models. These methods have covered the scenarios including latent sampling, editing, and inverting, and thus should be discussed.\n\n- From the technical point of view, the authors put much more emphasis on the empirical investigations on the interpretability of latent spaces in DMs, but neglect the dimension of diffusion time steps from a more fundamental perspective. It would strengthen the paper if the authors could elaborate on the impact of time steps on the construction of your w2w space because multiple existing works [a,b,d] suggest the time step of latent spaces in DMs could carry important information on the distribution learning process and thus influence the downstream tasks. \n\n- From the experiments and evaluation perspective, it would be helpful to perform some user studies as subjective evaluations as many other generative application works do. In addition, these attribute directions and their properties have been demonstrated other than identity context, it may strengthen the paper if the authors could extend their application scenarios to a wider context. \n\n---\n\n[a] Boundary Guided Learning Free Semantic Control with Diffusion Models, NeurIPS 2023.\n\n[b] Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation, ICLR 2024.\n\n[c]  Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions. 2024\n\n[d] Diffusionclip: Text-guided diffusion models for robust image manipulation. CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**“...editing and inverting have been widely studied in the field (see Weaknesses)”** and **“These methods have covered the scenarios including latent sampling, editing, and inverting and thus should be discussed”** and **“The methodological level design is also not entirely new and intriguing.”**\n\nWe agree that latent sampling, editing, and inverting has been studied extensively. However, we believe there might be a misunderstanding. We explore these applications on **model weights**, not input noise or latents. *weights2weights* space can be thought of as a **meta**-latent space, with  “applications analogous to those of a generative latent space – inversion, editing, and sampling – but **producing model weights rather than images**” (L316-318 in paper). We agree that the works you bring up are relevant, and we will include them in the related works. However, they focus on image manipulation, operating on the **image manifold**. As pointed out by reviewers rzPz and wMSF, we model and operate on the **“model weight manifold,”** manipulating the models themselves in order to produce new models. To the best of our knowledge, we believe this idea of a **meta**-latent space over model weights is novel. Furthermore, we add to this contribution by finding semantic weight subspaces with PCA and linear classifiers. Our analysis of the model weights shows that we can get valid identity-encoding diffusion models via linear interpolation among the weights of existing diffusion models spanning the subspace. \n\nWe contrast sampling, editing, and inverting in the traditional sense with our idea of operating on model weights in Figure 2 of our paper. **1)** Sampling in the traditional sense will produce a latent that results in a single image. Sampling from *w2w* space produces a **new diffusion model** that can generate an infinite number of different images of the same identity. **2)** Editing an input such as a latent results in one edited image. In contrast, edits in *w2w* space result in a **new model** with edited weights such that attributes of the identity that this model generates are changed. Once the edit is conducted, we can generate new images from the model and the edited identity is consistent (e.g., now always has a beard). Throughout the paper, we show results of generated images before and after the model weight edits and observe that the generated images are largely unchanged except for the identities. This is to show that after editing the weights, the model is largely unaffected except its identity. **3)** Inversion into the input space in generative models reconstructs a single image. However, inversion into *w2w* space produces a **new model** based on the identity in the input. We can then indefinitely sample new images from the model which consistently generates that identity. Furthermore, with the OOD projection, we are distilling a realistic identity from the input image into a new model.  \n\n**“... resulting in inaccurate claims on ‘Alas, in multi-step generative models, like diffusion models, such a linear latent space is yet to be found.’”**\n\nThank you for pointing this out. We agree that this sentence is poorly phrased. The point we meant to get across is nicely summarized in paper [a] that you brought up: “In contrast to a single unified latent space in GANs [the multistep process of diffusion models] imposes extra difficulties for studying the latent spaces of DDMs.” Still, our motivation for studying the weight space and modeling its manifold remains. The fact that personalization approaches can insert concepts into the model weights without disrupting the prior of the model suggests that we can find such subspaces. This inspires us to develop the *weights2weights* as a *meta-latent space* controlling model weights. We will clarify this in the manuscript. \n\n\n**“The authors put much more emphasis on the empirical investigations on the interpretability of latent spaces in DMs, but neglect the dimension of diffusion time…”**\n\nWe agree that analysis of diffusion timesteps is interesting with regards to the latent or noise spaces in DMs. However, we are analyzing the weights, which are fixed across timesteps. The identity and attribute subspaces we find are in the weights, and the applications such as inversion, sampling, and editing produce new weights parameterizing a new model which is fixed across timesteps. \n\n\n**“... it would be helpful to perform some user studies as subjective evaluations…”**\n\nThank you for your suggestion! We conducted user studies for identity inversion in rebuttal PDF Table 2 and identity editing in rebuttal Table 3. We discuss the details of these in global responses 3 and 4. Overall, users have a strong preference towards *weights2weights* for identity inversion and editing compared to baselines. \n\n\n\n**“...other than identity context…it may strengthen the paper if the authors could extend their application scenarios to a wider context.”**\n\nWe agree with your suggestion. Following your comment, we create a *weights2weights* weight subspace for dogs and  present the results in rebuttal PDF Figure 3. We conduct Dreambooth fine-tuning for 120 different dog breeds and train linear classifiers to define separating hyperplanes in weight space for different semantic attributes, such as “large” or “wavy fur.” In rebuttal Figure 3, we sample two models from this space and edit the weights to change attributes of the encoded breeds. \n\n\n**\"...I acknowledge that using model weights as datasets is interesting…‘scaling up’... one of the biggest contributions to me.”**\n\nThank you for acknowledging the scale of our weight space analysis! We will open source our code and release the weights for the community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response: 

We sincerely thank the reviewers for their feedback. We are glad that the reviewers found our creation and analysis of a large dataset of model weights “interesting” (abHC, 215A), and the concept of modeling the manifold of diffusion model weights “novel” with “broad applications in generative modeling” (wMSF). Furthermore, we are happy that reviewer wMSF found that our “extensive experiments...demonstrate the effectiveness of the proposed w2w space in generating, editing, and inverting identities”, which further expands the “scope of manifold applications” according to reviewer rzPz. We have uploaded a PDF of visual results to supplement our response to individual reviewers. Below, we address points that may be of interest to all reviewers. 

## 1) Summary of Contributions
As pointed out by reviewers 215A and abHC, previous works have explored inversion, editing, or sampling in the context of the **input space** of a model (e.g., latents, text conditioning, noise, images, etc.).  However, *weights2weights* space enables these three applications on the **network weights themselves, producing new models** as illustrated in Figure 2 of the main paper. As such, *weights2weights* space can be thought of as a **meta**-latent space, with “applications analogous to those of a generative latent space – inversion, editing, and sampling – but producing model weights rather than images” (L316-318 in the paper).  We demonstrate how new models can be created simply via linear interpolation within a linear subspace defined by PCA. In our submitted paper, these different models are each encoding a different instance of a visual identity (i.e., a person). We further train linear classifiers using model weights as data to find separating hyperplanes in weight space to edit semantic identity attributes encoded in the model weights. Each edit results in a new model with its original identity changed for some attribute (e.g., adding a beard). In Figure 3 of the rebuttal PDF, we demonstrate that the *weights2weights* idea is not limited to just faces and identities. We hope our work inspires further efforts to discover interpretable and controllable subspaces of model weights. 

## 2) Comparison to More Baselines in Identity Inversion
Following the suggestions of the reviewers, we have added quantitative results of *w2w* inversion against more baselines. Our current results in Table 2 of the paper contain comparisons against variants of Dreambooth [4], and we now add a comparison in the rebuttal PDF Table 1 to single-shot personalization methods: Celeb-Basis [1] and IP-Adapter FaceID [2]. We follow the same evaluation protocol as Section 4.4 of the paper, using the same base model. Our results indicate that inversion into *w2w* space achieves better identity preservation. We plan to update Table 2 of the paper with these additional results. 

## 3) Identity Inversion User Study
In rebuttal PDF Table 2, we present a two-alternative forced choice (2AFC) user study on the overall quality of generated identities from *w2w* inversion. Twenty users were given ten sets of images. Each set contained a randomly sampled original image of an identity, and then three random images generated using Celeb-Basis [1], IP-Adapter FaceID [2], and *w2w* with the same random prompt. Users were then asked to choose between alternate pairs based on three criteria: identity preservation, prompt alignment, and diversity of generated images. Our results show that users have a strong preference toward *w2w* identities. In Figure 6 of the rebuttal PDF, we present a qualitative comparison of the three methods. We plan to include more such qualitative comparisons in an updated version of the manuscript. We will also include this user study in this update. 

## 4) Identity Editing User Study
In rebuttal PDF Table 3, we present a two-alternative forced choice (2AFC) user study to evaluate the quality of identity edits. Twenty-five users were given ten sets of images. Each set contained a randomly sampled original image of an identity, and then an image of that identity edited for an attribute using Concept Sliders [3], *w2w*, and text prompting. Users were then asked to choose between alternate pairs based on three criteria: identity preservation, alignment with the desired edit, and disentanglement. These three criteria are similar to what we measure quantitatively in Table 1 in the paper. Our results show that users have a strong preference toward *w2w* edits. We plan to add this user study in an updated version of the manuscript. 



### References:

[1] ”Inserting Anybody in Diffusion Models via Celeb Basis.” Yuan et al. 2023. 

[2] “IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models.” Ye et al. 2023

[3] “Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models.” Gandikota et al. 2023

[4] “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.” Ruiz et al. 2023. 

[5] “Multi-Concept Customization of Text-to-Image Diffusion.” Kumari et al. 2023.

Meta-review: This paper has received consensus from all reviewers tending to accept the paper with the views ranging from Borderline Accept to Accept. All the reviewers have agreed on the contributions made by the paper and the clarifications by the authors has helped in addressing most of the concerns of the work. In view of the consensus achieved, AC agrees with an acceptance recommendation.
Decision: Accept (poster)