Interpreting the Weight Space of Customized Diffusion Models

Authors: amil  dravid
yossi  gandelsman
kuan-chieh  wang
rameen  abdal
gordon  wetzstein
alexei a efros
kfir  aberman
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:43:23

Abstract: We investigate the space of weights spanned by a large collection of customized diffusion models. We populate this space by creating a dataset of over 60,000 models, each of which is a base model fine-tuned to insert a different person's visual identity. We model the underlying manifold of these weights as a subspace, which we term $\textit{weights2weights}$. We demonstrate three immediate applications of this space that result in new diffusion models -- sampling, editing, and inversion. First, sampling a set of weights from this space results in a new model encoding a novel identity. Next, we find linear directions in this space corresponding to semantic edits of the identity (e.g., adding a beard), resulting in a new model with the original identity edited. Finally, we show that inverting a single image into this space encodes a realistic identity into a model, even if the input image is out of distribution (e.g., a painting). We further find that these linear properties of the diffusion model weight space extend to other visual concepts. Our results indicate that the weight space of fine-tuned diffusion models can behave as an interpretable $\textit{meta}$-latent space producing new models.

Main paper: 
 introduction: 
 Generative models have emerged as a powerful tool to model our rich visual world. In particular, the latent space of single-step generative models, such as generative adversarial networks (GANs) [18, 28] , has been shown to linearly encode meaningful concepts in the output images. For instance, datasets of latent vectors were used to discover linear directions in the GAN latent space encoding different attributes (e.g., gender or age of faces) [20, 60] . Even earlier, datasets of images and keypoints were leveraged to discover subspaces of facial shape and appearance [6, 53] .We aim to extend this even further, using datasets of model weights instead datasets of images or latents. Can we discover such interpretable subspaces in the model weights themselves? Recently introduced personalization approaches, such as Dreambooth [54] or Custom Diffusion [34] , may hint that this is the case. These methods aim to learn an instance of a subject, such as a person's visual identity. Rather than searching for a latent code that represents an identity in the input noise space, these approaches customize diffusion models by fine-tuning on subject-specific images, which results in identity-specific model weights. We therefore hypothesize that a latent space can exist in the weights themselves.To test our hypothesis, we fine-tune over 60,000 personalized models on individual identities to obtain points that lie on a manifold of customized diffusion model weights. To reduce the dimensionality of each data point, we use low-rank approximation (LoRA) [23] during fine-tuning and further apply Principal Components Analysis (PCA) to the set of data points. This forms our final space: weights2weights (w2w). Unlike traditional generative models like GANs, which model the pixel space of images, we model the weight space of these personalized models. Thus, each sample in our space corresponds to an identity-specific model which can consistently generate that subject. We provide a schematic in Fig. 2 that contrasts a typical latent space with our proposed w2w space, demonstrating the differences and analogies between these two representations. w2w space can be thought of as a meta-latent space, enabling controllable creation of new models instead of just images like a traditional latent space.Creating this space unlocks a variety of applications that involve traversal in w2w (Fig. 1 ). First, we demonstrate that sampling model weights from w2w space corresponds to a new model encoding a novel subject. Second, we find linear directions in this space corresponding to semantic edits of the identity. Finally, we show that enforcing weights to live in this space enables a diffusion model to learn a subject given a single image, even if it is out of distribution.We find that w2w space is highly expressive through quantitative evaluation on editing customized models and encoding new identities given a single image. Qualitatively, we observe this space supports sampling models that encode diverse and realistic identities, while also capturing the key characteristics of out-of-distribution identities. We finally demonstrate that similar weight subspaces exist for other visual concepts such as dog breeds and car types.Figure 3 : Building weights2weights (w2w) space. We create a dataset of model weights where each model is personalized to a specific identity using low-rank updates (LoRA). These model weights lie on a weights manifold that we further project into a lower-dimensional subspace spanned by its principal components. We train linear classifiers to find disentangled edit directions in this space. 
related work: 
 Image-based generative models. Various models have been proposed for image generation, including variational autoencoders (VAEs) [31] , flow-based models [12, 32, 49] , generative adversarial networks (GANs) [18] , and diffusion models [22, 43, 62] . Within the realm of high-quality photorealistic image generation, GANs [25, 28, 29] and diffusion models [22, 43, 52, 63] have garnered significant attention due to their controllability and ability to produce high-quality images. Leveraging the compositionality of these models, methods for personalization and customization have been developed which aim to insert user-defined concepts via fine-tuning [16, 34, 40, 54] . Various works try to reduce the dimensionality of the optimized parameters for personalization either by operating in specific model layers [34] or in text-embedding space [16] , by training hypernetworks [55] , and by constructing a linear basis in text embedding space [73] .Latent space of generative models. Linear latent space models of facial shape and appearance were studied extensively in the 1990s, using PCA-based representations (e.g. Active Appearance Models [9] , 3D Morphable Models [6] ) as well as operating directly in pixel and keypoint space [53] . However, these techniques were restricted to aligned and cropped frontal faces. More recently, generative adversarial networks (GANs), particularly the StyleGAN series [26, 27, 28, 29] , have showcased editing capabilities facilitated by their interpretable latent space. Furthermore, linear directions can be found in their latent space to conduct semantic edits by training linear classifiers or applying PCA [20, 60] , among other methods for discovering semantic directions [8, 66] . Several methods aim to project real images into the GAN latent space in order to conduct this editing [1, 3, 51, 64, 77] . Beyond the latent space, works such as [4] found that directions could be discovered in the neuron activation space, suggesting the interpretability of weights.Although diffusion models architecturally lack a GAN-like latent space, some works aim to discover similar spaces in these models. This has been explored in the UNet bottleneck layer [35, 41] , noise space [10, 78] , and text-embedding space [5] . Concept Sliders [17] explores the weight space for semantic image editing by conducting low-rank training with contrasting image or text pairs.Weights as data. Past works have exploited the structure within weight space of deep networks for various applications. In particular, some have found linear properties of weights, enabling simple model ensembling and editing via arithmetic operations [24, 56, 59, 69] . Other works create datasets of neural network parameters for training hypernetworks [14, 19, 42, 55, 68] , predicting properties of networks [58] , and creating design spaces for models [46, 47] . 
method: 
 We start by demonstrating how we create a manifold of model weights as illustrated in Fig. 3 . We explain how we obtain low-dimensional data points for this space, each of which represents an individual subject from a broad class (i.e., identity). We then use these points to model a weights manifold. Next, we find linear directions in this manifold that correspond to semantic attributes and use them for editing the identities. Finally, we demonstrate how this manifold can be utilized for constraining an ill-posed inversion task with a single image to reconstruct its identity.In this section, we first introduce latent diffusion models (LDM) [52] , which we will use to create a dataset of weights. Then, we explain the approach for obtaining identity-specific models from LDM via Dreambooth [54] fine-tuning. We finally present a version of fine-tuning that uses lowdimensional weight updates (LoRA [23] ). We will use the fine-tuned low-dimensional per-identity weights as data points to construct the weights manifold in Sec. 3.2.Latent diffusion models [52] . We will extract weights from latent diffusion models to create w2w space. These models follow the standard diffusion objective [22] while operating on latents extracted from a pre-trained Variational Autoencoder [15, 31, 50] . With text, the conditioning signal is encoded by a text encoder (such as CLIP [44] ), and the resulting embeddings are provided to the denoising UNet model. The loss of latent diffusion models is:EQUATIONwhere ϵ θ is the denoising UNet, x t is the noised version of the latent for an image, c is the conditioning signal, t is the diffusion timestep, and w t is a time-dependent weight on the loss.To sample from the model , a random Gaussian latent x T is deterministically denoised conditioned on a prompt for a fixed set of timesteps with a DDIM sampler [63] . The denoised latent is then fed through the VAE decoder to generate the final image.Dreambooth [54] . To obtain an identity-specific model, we use the Dreambooth personalization method. Dreambooth fine-tuning introduces a novel subject into a pre-trained diffusion model given only a few images of it. During training, Dreambooth follows a two-part objective:EQUATIONwhere the first term corresponds to the standard diffusion denoising objective using the subjectspecific data x conditioned on the text prompt "[identifier] [class noun]" (e.g., "[v] person"), denoted c. The second term, weighted by λ, corresponds to a prior preservation loss, which involves the standard denoising objective using the model's own generated samples x ′ for the broader class c ′ (e.g., "person"). This prevents the model from associating the class name with the specific instance, while also leveraging the semantic prior on the class.Low Rank Adaptation (LoRA) [23] . Dreambooth requires fine-tuning all the weights of a model, which is a high-dimensional space. We turn to a more efficient fine-tuning scheme, LoRA, that modifies only a low-rank version of the weights. LoRA uses weight updates ∆W with a low intrinsic rank. For a base model layer W ∈ R m×n , the LoRA update for that layer ∆W can be decomposed into ∆W = BA, where B ∈ R m×r and A ∈ R r×n are low-rank matrices with r ≪ min(m, n).During training, for each model layer, only the A and B are updated. This significantly reduces the number of trainable parameters. During inference, the low-rank weights are added residually to the weights of each layer in the base model and scaled by a coefficient α ∈ R: W + α∆W .Creating a dataset of model weights. To construct the weights2weights (w2w) space, we begin by creating a dataset of model weights θ i . We conduct Dreambooth fine-tuning on latent diffusion models in order to insert new subjects with the ability to control image instances using text prompts. This training is done with LoRA in order to reduce the space of model parameters. Each model is fine-tuned on a set of images corresponding to one human subject. After training, we flatten and concatenate all of the LoRA matrices, resulting in a data point θ i ∈ R d which represents one identity. After training over N different instances, we have our final dataset of model weights D = {θ 1 , θ 2 , ..., θ N }, representing a diverse array of subjects.Modeling the weights manifold. We posit that our data D ⊆ R d lies on a lower-dimensional manifold of weights that encode identities. A randomly sampled set of weights in R d , would not be guaranteed to produce a valid model encoding identity as the d degrees of freedom can be fine-tuned for any purpose. Therefore, we hypothesize that this manifold is a subset of the weight space. Inspired by findings that high-level concepts can be encoded as linear subspaces of representations [13, 37, 45, 48] , we model this subset as a linear subspace R m where m < d, and call it weights2weights (w2w) space. We represent points in this subspace as a linear combination of basis vectors w = {w 1 , ..., w m }, w i ∈ R d . In practice, we apply Principal Component Analysis (PCA) on the N models and keep the first m principal components for dimensional reduction and forming our basis of m vectors.Sampling from the weights manifold. After modeling this weights manifold, we can sample a new model that lies on it, resulting in a new model that generates a novel identity. We sample a model represented with basis coefficients {β 1 , ..., β m }, where each coefficient β k is sampled from a normal distribution with mean µ k and standard deviation σ k . The mean and standard deviation are calculated for each principal component k from the coefficients among all the training models.We seek a direction n ∈ R d defining a hyperplane that separates between binary identity properties embedded in the model weights (e.g., male/female), similarly to hyperplanes observed in the latent space of GANs [60] . We assume binary labels are given for attributes present in the identities encoded by the models. We then train linear classifiers using weights of the models as data based on these labels, imposing separating hyperplanes in weight space. Given an identity parameterized by weights θ, we can manipulate a single attribute by traversing in a direction n, orthogonal to the separating hyperplane: θ edit = θ + αn. An edit operation in w2w spaee produces a new model with the original subject edited, allowing the model to generate infinitely many new images of the edited subject.Traditionally, inversion of a generative model involves finding an input such as a latent code that best reconstructs a given image [38, 70] . This corresponds to finding a projection of the input onto the learned data manifold [77] . With w2w space, we model a manifold of model weights rather than images. Inspired by latent optimization methods [1, 77] , we propose a gradient-based method of inverting a single identity from an image into our discovered space.Given a single image x, we follow a constrained denoising objective:EQUATIONSpecifically, we constrain the model weights to lie in w2w space by optimizing a set of basis coefficients {β 1 , ..., β m } rather than the original parameters. Unlike Dreambooth, we do not employ a prior preservation loss, since the optimized model lies in the subspace defined by our dataset of weights, and inherits their priors. 
experiments: 
 We demonstrate w2w space on the visual concept of human identities for a variety of applications. We begin by detailing implementation details. Next, we use w2w space for 1) sampling new models encoding novel identities, 2) editing identity attributes in a consistent manner via linear traversal in w2w space, 3) embedding a new identity given a single image, and 4) projecting out-of-distribution identities into w2w space. Finally, we analyze how scaling the number of models in our dataset of model weights affects the disentanglement of attribute directions and preservation of identity.Creating an identity dataset. We generate a synthetic dataset of ∼65,000 identities using [67] ,where each identity is associated with multiple images of that person. Each identity is based on an image with labeled binary attributes (e.g., male/female) from CelebA [36] . Each set of images corresponding to an identity is then used as data to fine-tune a latent diffusion model with Dreambooth. Further details on this dataset and train/test splits are provided in Appendix E.Encoding identities into model weights. We conduct Dreambooth fine-tuning using LoRA with rank 1 on the identities. Following [56] , we only fine-tune the query and value projection matrices in the cross-attention layers. We utilize the RealisticVision-v51 checkpoint 2 based on Stable Diffusion 1.5. Conducting Dreambooth fine-tuning on each identity training set results in a dataset of ∼65,000 weights θ where θ ∈ R 100,000 .Finding semantic attribute directions. We utilize binary attribute labels from CelebA to train linear classifiers on the dataset of model weights we curated. We run Principal Component Analysis (PCA) on the ∼65,000 training models and project to the first 1000 principal components in order to reduce the dimensionality. The orthogonal edit directions are calculated via the analytic least squares solution on the matrix of projected training models D ∈ R 65,000×1000 , and then unprojected to the original dimensionality of the model weights: θ ∈ R 100,000 .Figure 4 : Identity samples from w2w space. We show the samples from w2w space do not overfit to nearest-neighbor identities, although they incorporate facial attributes from them. The identities are diverse and consistent across generations.We present images generated from models that were sampled from the weights manifold (i.e., w2w Space) in Fig. 4 . We follow the sampling procedure from Sec. Comparing with the nearest neighbors shows that these samples are not just copies from the dataset, but rather encode diverse identities with different attributes. Yet, the samples still demonstrate some similar features to the nearest neighbors. These include jawline and eye shape (top row), facial hair (middle row), and nose and eye shape (bottom row). Appendix A includes more such examples.We demonstrate how directions found by the linear classifiers can be used to edit subjects encoded in the models. It is desired that these edits are 1) disentangled (i.e., do not interfere with other attributes of the embedded subject and preserve all other concepts such as context) 2) identity preserving (i.e., the person is still recognizable) 3) and semantically aligned with the intended edit.Baselines. We compare against a naïve baseline of prompting with the desired attribute (e.g., "[v] person with small eyes"), and then Concept Sliders [17] , an instance-specific editing method which we adapt to subject editing. In particular, we train their most accessible method, the text-based slider, which trains LoRAs to modulate attributes in a pretrained diffusion model based on contrasting text prompts. We then apply these sliders to the personalized identity models. Evaluation protocol. We evaluate these three methods for identity preservation, disentanglement, and edit coherence. To measure identity preservation, we first detect faces in the original generated images and the result of the edits using MTCNN [74] . We then calculate the similarity of the FaceNet [57] embeddings. We also use LPIPS [76] computed between the images before and after the edit to measure the degree of disentanglement with other visual elements, and CLIP score [21] , to measure if the desired edit matches the text caption for the edit.Prompting Sliders w2w Bald Beard Age Pale Chubby Black HairTo generate samples, we fix a set of prompts and random seeds which are used as input to the held-out identity models. Then, we choose a set of identity-specific manipulations. For prompt-based editing, we augment the attribute description to the set of fixed prompts (e.g., "chubby [v] person"). For Concept Sliders and w2w, we apply the weight space edit directions to the personalized model with a fixed norm which determines the edit strength. The norm is calculated using the maximum projection component onto the edit direction among the training set of model weights.w2w edits are identity preserving and disentangled. We evaluate over a range of identity-specific attributes and present three (gender, chubby, narrow eyes) in Tab. 1. Edits in w2w preserve the identity of the original subject as measured by the ID score. These edits are semantically aligned with the desired effect as indicated by the CLIP score while minimally interfering with other visual concepts, as measured by LPIPS. We note that the CLIP score can be noisy in this setting as text captions can be too coarse to describe attributes as nuanced as those related to the human face. We supplement this with a user study presented in Appendix B.Qualitatively, w2w edits make the minimal amount of changes to achieve semantic and identitypreserving edits (Fig. 5 ). For instance, changing the gender of the man does not significantly change the facial structure or hair, unlike Concept Sliders or prompting with text descriptions. Prompting has inconsistent results, either creating no effect or making drastic changes. Concept Sliders tends to make caricaturized effects, such as making the man cartoonishly chubby and baby-like.Composing edits. Edit directions in w2w space can be composed linearly as shown in Fig. 6 . The first column represents samples from the original model, and each subsequent column represents samples from the edited models. Each row shares the same fixed random generation seed. The composed edits persist in appearance across different generations, binding to the identity. Furthermore, the edited weights result in a new model, where the subject has different attributes while still maintaining as much of the prior identity. This is in contrast to editing in a traditional latent space, where an edit only corresponds to a single image. Additionally, as we operate on an personalized identity-specific weight manifold, minimal changes are made to other concepts, such as scene layout or other people. For instance, in Fig. 6 , adding edits to the woman does not interfere with the person standing by her. We present generated samples from the inverted models. These inverted identities can be composed in novel contexts and edited using our discovered semantic directions in weight space. These edits persist in appearance across generation seeds and prompts. Evaluation protocol. We measure w2w space's ability to represent novel identities by inverting a set of 100 random FFHQ [28] face images. We follow our inversion objective from eq. 3. We then provide a set of diverse prompts to generate multiple images and follow the identity preservation metric from Sec. 4.3 to measure subject fidelity. Implementation details are provided in Appendix C. We compare our results to two approaches that use Dreambooth with rank-1 LoRA. The first is trained on a single image. The second is trained on multiple images of each identity. We generate such images by following our identity dataset construction from Sec. 4.1. This approach can be viewed as a pseudo-upper bound on modeling identity as it uses multiple images.w2w space provides a strong identity prior. Inverting a single image into w2w space improves on the single image Dreambooth baseline and closes the gap with the Dreambooth baseline that uses multiple identity images (Tab. 2). Conducting Dreambooth fine-tuning with a single image in the original weight space leads to image overfitting and poor subject reconstruction as indicated by a lower ID score. In contrast, by constraining the optimized weights to lie on a manifold of identity weights, w2w inversion inherits the rich priors of the models used to discover the space. As such, it can extract a high-fidelity identity that is consistent and compositional across generations. We present qualitative comparisons against Dreambooth and single-image Dreambooth in Appendix C. We additionally compare against other personalization methods in that section. Inverted models are editable. Fig. 7 demonstrates that a diverse set of identities can be faithfully represented in w2w space. After inversion, the encoded identity can be composed in novel contexts and poses. For instance, the inverted man (rightmost example) can be seen posing with a celebrity or rendered as a statue. Moreover, semantic edits can be applied to the inverted models while maintaining appearance across generations.w2w space captures out-of-distribution identities. We follow the w2w inversion method from Sec. 4.4 to project images of unrealistic identities (e.g., paintings, cartoons, etc.) onto the weights manifold, and present these qualitative results in Fig. 8 . By constraining the optimized model to live in w2w space, the inverted identities are converted into realistic renditions of the stylized identities, capturing prominent facial features. In Fig. 8 , notice how the inverted identities generate a similar blonde hairstyle and nose structure in the first example, defined jawline and lip shape in the second example, and head shape and big nose in the last example. As also shown in the figure, the inverted identities can also be translated to other artistic domains using text prompts. We present a variety of domains projected into w2w space in Appendix D.We ablate the number of models used to create w2w space and investigate the expressiveness of the resulting space. In particular, we measure the degree of entanglement among the edit direction and how well this space can capture identity.Disentanglement vs. the number of models. We find that scaling the number of models in our dataset of weights leads to less entangled edit directions in w2w space (Fig. 9 ). We vary the number of models in our dataset of weights and reapply PCA to establish a basis. We then measure the absolute value of cosine similarity (lower is better) between all pairs of linear classifier directions found for CelebA labels. We repeat this as we scale the number of model weights used to train the classifiers. We report the mean and standard deviation for these scores, along with three notable semantic direction pairs. We observe a trend in decreasing cosine similarity. Notably, pairs such as "Black Hair -Pale Skin," "Young -Bald," and "Male -Beard" which may correlate in the distribution of identities, become less correlated as we scale our dataset of model weights.Identity preservation vs. the number of models. We observe that as we scale the number of models in our dataset of weights, identities are more faithfully represented in w2w space (Fig. 10 ). We follow the same procedure as the disentanglement ablation, reapplying PCA to establish a basis based on the dataset of model weights. Next, following Sec. 4.4, we optimize coefficients for this basis and measure the average ID score over the 100 inverted FFHQ evaluation identities. As each model in our dataset encodes a different instance of an identity, growing this dataset increases the span of w2w space and its ability to capture more diverse identities. We plot the average multi-image Dreambooth LoRA (DB-LoRA) ID score from Sec. 4.4, which is agnostic to our dataset of models. This establishes a pseudo-upper bound on identity preservation. Scaling enables w2w to represent identities given a single image with performance approaching that of traditional Dreambooth with LoRA, which uses multiple images and trains in a higher dimensional space. 
extending to other domains: 
 We extend our hypothesis of interpretable linear weight subspaces in diffusion models to other visual concepts beyond human identities. We apply the weights2weights framework to form a subspace for models encoding different dog breeds. To create a dataset for fine-tuning, we generate images with Stable Diffusion based on each of the 120 dog classes from ImageNet [11] . We then conduct Dreambooth fine-tuning on each set of dog breed images to create a dataset of 120 dog-encoding models, subsequently applying PCA. To find edit directions, we use GPT-4 [2] to create labels for Small Large Van Sports CarFigure 11 : weights2weights linear subspaces can be created for other visual concepts. We follow the same procedure of applying PCA and finding edit directions with linear classifiers on datasets of models encoding dog breeds and models encoding car types.each dog breed (e.g., wavy hair or not) and then train linear classifiers on the model weight principal component projections like Sec. 3.3. We apply the same framework to different car categories using models fine-tuned on images from a dataset of 197 different car types [33] . We present results for traversing edit directions in these two subspaces in Fig. 11 . Each column represents samples from an edited model. Each row shares the same fixed random generation seed.Our results provide further evidence that diffusion models can encode visual concepts linearly. This enables the creation of new models in a controlled manner via simple interpolation. For instance, in Fig. 11 , we rewrite the model's learned concept of a small golden retriever to make it bigger, or the model's encoding of a red van to make it a sports car. Additionally, unlike older PCA-based methods [6, 53, 61, 65] which rely on aligned pixels or keypoints of human faces, weights2weights can extend to other domains beyond human identities. We refer the reader to Appendix G for more results of applying w2w space to other visual concepts.6 Limitations As with any data-driven method, w2w space inherits the biases of the data used to discover it. For instance, co-occurring attributes in the identity-encoding models would cause linear classifier directions to entangle them (e.g. gender and facial hair). However, as we scale the number of models, spurious correlations will drop as evidenced by Fig. 9 . These semantic directions are also limited by the labels present in CelebA. Additionally, the span of the w2w space is dictated by the models used to create it. Thus, w2w space can struggle to represent more complex identities as seen in Fig. 12 . Inversion in these cases amounts to projecting onto the closest identity on the weights manifold. Despite this, our analysis on the size of the model dataset reveals that forming a space using a larger and more diverse set of identity-encoding models can mitigate this limitation. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We present data splits, hyperparameters, etc. in Secs. 3, 4, and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion and broader impact: 
 We presented a paradigm for representing diffusion model weights as a point in a subspace defined by other customized models -weights2weights (w2w) space. This enabled applications analogous to those of a generative latent space -inversion, editing, and sampling -but producing model weights rather than images, resulting in what we term a meta-latent space. We demonstrated these applications on model weights encoding human identities and extended this space to other visual concepts. Although these applications could enable malicious manipulation of real human identities and model weights, we hope the community uses the framework to explore visual creativity as well as utilize this interpretable space for controlling models for safety.Qualitative Results. We display additional examples of applying edits in w2w space based on the directions discovered using linear classifiers and CelebA labels. In Fig. 14 , we demonstrate how the strength of these edits can be modulated and combined with minimal interference. These edits are apparent even in more complex scenes beyond face images. Also, the edits do not degrade other present concepts, such as the dog near the man (top left example).In Figs. 15 and 16 , we demonstrate how multiple edits can be progressively added in a disentangled fashion with minimal degradation to the identity. Additionally, since we operate in a subspace of weight space, these edits persist with a consistent appearance across different generations. For instance, even the man exhibits the edits as a painting in Fig. 15 . User Study. We present a two-alternative forced choice (2AFC) user study to evaluate the quality of identity edits in Tab. 3. Twenty-five users were given ten sets of images. Each set contained a randomly sampled original image of an identity, and then an image of that identity edited for an attribute using Concept Sliders [17] , w2w, and text prompting. Users were then asked to choose between alternate pairs based on three criteria: identity preservation, alignment with the desired edit, and disentanglement. Our results in Tab. 3 show that users have a strong preference towards w2w edits. User instructions and an example question from the study are provided in Figs. 17, 18 . Age Pale Chubby Black Hair Pointy Nose CurlsWe present additional details on w2w inversion and comparisons against training Dreambooth LoRA on a single image vs. multiple images.Implementation Details: To conduct w2w inversion, we train on a single image following the objective from eq. 3. We qualitatively find that optimizing 10,000 principal component coefficients balances identity preservation with editability. This is discussed in Appendix F. We optimize for 400 epochs, using Adam [30] with learning rate 0.1, β 1 = 0.9, β 2 = 0.999 and with weight decay factor 1e-10. For conducting Dreambooth fine-tuning, we follow the implementation from Hugging Face3 using LoRA with rank 1. To create a dataset of multiple images for an identity, we follow the procedure from Sec. 4.4. Comparison against Single Image Personalization Methods. We compare w2w inversion to single shot personalization methods Celeb-Basis [73] and IP-Adapter FaceID4 [72] , following the same evaluation protocol from Sec. 4.4. These quantitative results are presented in Tab. 5. While Celeb-Basis optimizes in the input text embedding space for a given face image, IP-Adapter trains light-weight adapters [39, 75] to condition generation on any input face image.We further conducted a two-alternative forced choice (2AFC) user study on the perceptual quality of w2w identity preservation. Twenty users were given ten sets of images. Each set contained a randomly sampled original image of the identity, and then three random images generated using IP-Adapter FaceID, Celeb-Basis, and w2w with the same random prompt. Users were then asked to choose between alternate pairs based on three criteria: identity preservation, prompt alignment, and diversity of generated images. Our results in Tab. 6 show that users found generations from w2w models capture identity better while also generating more diverse images that better align with the prompts.Across both these metrics, w2w performs stronger than Celeb-Basis and IP-Adapter FaceID. Our results indicate that operating in our weight subspace is highly expressive and flexible as it is able to faithfully capture nuanced identity without overfitting to the input image. For instance, in Fig. 21 , w2w inversion enables diverse generation with various poses, facial expressions, and clothing while maintaining identity. Original "A photo of V 1 person"Celeb-Basis w2wAdditional examples of out-of-distribution projections are displayed in Fig. 24 . A diverse array of styles and subjects (e.g. paintings, sketches, non-humans) can be distilled into a model in w2w space.After embedding an identity into this space, the model still retains the compositionality and rich priors of a standard personalized model. For instance, we can generate images using prompts such as "[v] person writing at a desk" (top example), "[v] person with a dog" (middle example), or "a painting of [v] person painting on a canvas" (bottom example).Figure 24 : Projection into w2w space generalizes to a variety of inputs. A range of styles and entities can be inverted into a realistic identity in this space. Once a model is obtained, it can be prompted to generate the identity in a variety of contexts.In Fig. 25 , we present examples of synthetic identity datasets used to conduct our Dreambooth fine-tuning as discussed in Sec 4. Each dataset is a set of ten images generated with [67] conditioned on single CelebA [36] images associated with binary attribute labels. Note that we only display a subset of images per identity in the figure. The same identity can occur multiple times in different images in CelebA but have different appearances. So, creating these synthetic datasets reduces intra-dataset diversity and creates a more consistent appearance for each subject. For instance, the first two rows in the figure are the same identity, but look drastically different. So we instead treat them as different identities associated with a different set of images.For evaluating identity edits from Sec. 4.3, we hold out 100 identities, which results in leaving out ∼1000 models since multiple models may encode different instances of the same identity. For instance, if we left out the model encoding the identity in the first row of Fig. 25 for evaluation, the model encoding the second row identity would also be left out since it encodes the same identity but a different instance. The left column shows a CelebA image used to condition generation of a set of identity-consistent images in the right column associated with that identity using [67] . The consistent appearance of the identity enables a more consistent identity encoding.In this section, we analyze various properties of the Principal Component (PC) basis used to define w2w Space. We investigate the distribution of PC coefficients and the effect of the number of PCs on identity editing and inversion.Distribution of PC Coefficients. We plot the histogram of the coefficient values for the first three Principal Components in Fig. 26 . They appear roughly Gaussian. Next, we rescale the coefficients for these three components to unit variance for visualization purposes. We then plot the pairwise joint distributions for them in Fig. 27 . The circular shapes indicates roughly diagonal covariances. Although the joint over other combinations of Principal Components may exhibit different properties, these results motivate us to model the PCs as independent Gaussians, leading to the w2w sampling strategy from Sec. 3.2.We empirically observe that training classifiers based on the 1000 dimensional PC representations (first 1000 PCs) of the model weights results in the most semantically aligned and disentangled edits directions. We visualize a comparison for the "goatee" direction in Fig. 28 . After finding the direction, we calculate the maximum projection component onto the edit direction among the training set of model weights. This determines the edit strength. As seen in the figure, restricting to the first 100 Principal Components may be too coarse to achieve the fine-grained edit, instead relying on spurious cues such as skin color. Training with the first 10,000 Principal Components suffers from the curse of dimensionality and the discovered direction may edit other concepts such as eye color or clothes. Finding the direction using the first 1000 Principal Components achieves the desired edit with minimal entanglement with other concepts.We qualitatively observe that inversion using the first 10,000 Principal Components balances identity preservation while not overfitting to the source image. We visualize a comparison in Fig. 29 , where each column has a fixed seed and prompt.Optimizing with the first 1000 PCs underfits the identity and does not generate a consistent identity. Inversion with the first 20,000 Principal Components overfits to the source image of a face shot, which results in artifacted face images despite different generation seeds and prompts. Optimizing with the first 10,000 Principal Components enjoys the benefits of a lower dimensional representation than the original LoRA parameter space (∼100,000 trainable parameters), while still preserving identity and compositionality. This is supported quantitatively by Fig. 30 , which shows the average ID score for 100 inverted FFHQ identities optimized over a varying number of principal components. We optimize the coefficients for the first N principal components up to 20,000 and measure the average ID score for 100 inverted FFHQ identities.Visualizing Principal Components. We provide a visualization of traversals along a set of principal components in Fig. 31 . The principal components change attributes of the identity, although various semantic attributes are entangled. For instance, the first PC appears to change age, hair color, and hair style. The second PC appears to change gender and skin complexion. The third PC seems to change age, skin complexion, and facial hair. This motivates our use of linear classifiers to find separating hyperplanes in weight space and disentangle these attributes. We find that similar subspaces can be created for other visual concepts beyond human identites. For instance, we apply the weights2weights framework to create two subspaces for models encoding different dog breeds and models encoding car types. We present examples of editing these models in Fig. 32 . This suggest the generality of weights2weights and linear subspaces within diffusion model weights.Figure 32 : Applying weights2weights edits on dog-encoding models and car-encoding models.We create two datasets of model weights and apply PCA to define two separate weight subspaces.We then train linear classifiers to find semantic edit directions.Multiple models living in w2w space cannot be merged since they live in the same weight subspace. So, merging will lead to interpolation of the identities. However, w2w models can be merged with models lying approximately orthogonal to the subspace. This merging can be done adding the weights.We present an example of merging a model living in w2w space with a model fine-tuned to encode "Pixar" style in Fig. 33 .Figure 33 : Merging w2w models with non-identity models. Here, another model is fine-tuned to map V 2 to "Pixar" style. The two models are merged with simple addition.Edits in w2w space correspond to identity edits with minimal interference with other visual concepts.Although not a focus, image editing is achieved as a byproduct. For further context preservation, edits in w2w Space can be integrated with delayed injection [7, 17, 35, 71] , where after T timesteps, the edited weights are used instead of the original ones. We visualize this in Fig. 34 . Larger T in the range [700, 1000] are helpful for more global attribute changes, while smaller [400, 700] can be used for more fine-grained edits. However, by decreasing the timestep T , the strength of the edit is lost in favor of better context preservation. For instance, the dog's face is better preserved in the second row at T = 600, although the man is not as chubby compared to other T .Original T=1000 T=800 T =600 • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.+ChubbyQuestion: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Confidence intervals are reported for the results in the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The appendix details the compute resources used.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The authors have read the NeurIPS Code of Ethics and confirm that the paper conforms with it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Broader impacts has a dedicated section in Sec. 7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work utilizes pretrained models and well-established datasets that already have proper safeguards in place.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original creators of assets are either cited or linked.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://huggingface.co/stablediffusionapi/realistic-vision-v51https://github.com/huggingface/pefthttps://huggingface.co/h94/IP-Adapter-FaceID 
latent space: 
 Figure 2 : The weights2weights space operates as a meta-latent space. Unlike a traditional generative latent space, w2w space controls the model itself rather than single image instances. New identity-encoding models can be sampled from the space and edited by linearly traversing along semantic directions in weight space. Additionally, a single image can be inverted into the space to produce a model that consistently generates that identity. 
 
 Appendix:  


Decision: This paper has received consensus from all reviewers tending to accept the paper with the views ranging from Borderline Accept to Accept. All the reviewers have agreed on the contributions made by the paper and the clarifications by the authors has helped in addressing most of the concerns of the work. In view of the consensus achieved, AC agrees with an acceptance recommendation.

Reviews: review: paper_summary: paper_summary: This paper explores the latent space of weights in customized diffusion models, introducing the weights2weights (w2w) space, a subspace encoding different human identities. By fine-tuning over 65,000 models, each representing a distinct human identity, the authors model this weight space using low-rank adaptation (LoRA) and principal component analysis (PCA). The key contributions include demonstrating three applications of the w2w space: Sampling: Generating novel identities by sampling from the weight space; Editing: Performing semantic edits, such as adding a beard, by traversing linear directions in the weight space; Inversion: Reconstructing a realistic identity from a single image, even if it is out-of-distribution. The paper validates the w2w space’s expressiveness through quantitative evaluations and qualitative observations.
strength_weakness: strength_weakness: 1) The overall idea of this paper is more similar to [1], the significant difference I see so far is that [1] does PCA on token embedding and w2w does PCA on Lora in Unet. I suggest the authors could discuss more about the differences between this paper and [1]. Also, I have some methodological doubts:

A. does w2w have a similar phenomenon to mean person in [1]? (see Figure 2 in [1])

B. Have the authors studied the effect of PCA on the generation of results? (see section 4.3 in [1])

[1] Inserting Anybody in Diffusion Models via Celeb Basis

2) While this paper provides new ideas and insights into personalized generation models, I am concerned about his real-world application, what are the advantages of w2w compared to existing personalized generation efforts such as IP-adapter, InstabntID, custom diffusion, Mix-of-show, etc.? Faster generation or smaller training parameters? The experiments also don't give a comparison with existing personalization generation efforts, however, I think this is important to note
questions: questions: 1) I'm curious if style information can be decoupled via w2w space in addition to attributes. I would suggest that the authors could add the results of style editing to their paper.

2) How does w2w perform on multi-concept generation?
limitations: limitations: The paper discussed the limitations of this work.
paper_summary: paper_summary: This paper constructs a dataset that contains LoRA models trained on images of 60,000 different people. A weight manifold is determined based on the parameters of these 60,000 models using PCA. Sampling, editing, and inversion can be performed on this manifold, and the rationality of this manifold is demonstrated.
strength_weakness: strength_weakness: 1. There are some recent works present the way of generating identity preserved images given only one reference image. It is better to provide the comparison between this paper and these works.
2. LoRA training often affects the performance of the base model to some extent, such as small generation variance. More results to examine this effect will be better.
questions: questions: Refer to the Weaknesses 1 and 2.
limitations: limitations: Yes, limitations are discussed in this paper.
paper_summary: paper_summary: The paper investigates the weight space spanned by a large collection of customized diffusion models, proposing a novel subspace termed weights2weights (w2w). The study populates this space with a dataset of over 60,000 models, each fine-tuned to encode a different person's visual identity. The paper demonstrates three main applications of this space: sampling novel identities, editing existing identities, and inverting a single image to reconstruct realistic identities, showing that these weight spaces can serve as a highly interpretable latent identity space.
strength_weakness: strength_weakness: 1. The construction of w2w space needs over 6000 weights and the number of the model can further improve the ID score.  The computing resources should be provided. And is there a contained solution to balance computing resources and performance?

2. The paper uses several evaluation metrics (e.g., identity preservation, LPIPS, CLIP score), some metrics like CLIP score can describe the quality of image generation, but might not fully evaluate the nuanced attributes related to human faces. 

3. In the weights manifold, new models can be sampled to find directions that separate identity properties. However, are the first m principal components all identity attributes, and do they correspond one-to-one?
questions: questions: Please refer to the weaknesses above.
limitations: limitations: The authors have discussed the limitations of this work.
paper_summary: paper_summary: The paper proposes to learn a manifold over customized diffusion model weights as a subspace for interpretable downstream applications such as sampling, editing, and inversion.  At the technological level, the authors collect the weights of over 60,000 models as the dataset and fine-tune those weights via LoRA to constitute the desired editing space. Experiments are conducted on CelebA dataset.
strength_weakness: strength_weakness: - The paper lacks a comprehensive comparison with prior literature, resulting in inaccurate claims on “Alas, in multi-step generative models, like diffusion models, such a linear latent space is yet to be found”. In [a], it has been explicitly and theoretically proven such interpretable and linear latent space can be found in frozen DMs, without the need to fine-tune the model parameters, neither the DreamBooth tuning nor LoRA-based. Recent studies [2,3] have extended these findings for customization tasks and latent diffusion models. These methods have covered the scenarios including latent sampling, editing, and inverting, and thus should be discussed.

- From the technical point of view, the authors put much more emphasis on the empirical investigations on the interpretability of latent spaces in DMs, but neglect the dimension of diffusion time steps from a more fundamental perspective. It would strengthen the paper if the authors could elaborate on the impact of time steps on the construction of your w2w space because multiple existing works [a,b,d] suggest the time step of latent spaces in DMs could carry important information on the distribution learning process and thus influence the downstream tasks. 

- From the experiments and evaluation perspective, it would be helpful to perform some user studies as subjective evaluations as many other generative application works do. In addition, these attribute directions and their properties have been demonstrated other than identity context, it may strengthen the paper if the authors could extend their application scenarios to a wider context. 

---

[a] Boundary Guided Learning Free Semantic Control with Diffusion Models, NeurIPS 2023.

[b] Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation, ICLR 2024.

[c]  Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions. 2024

[d] Diffusionclip: Text-guided diffusion models for robust image manipulation. CVPR 2022.
questions: questions: Please see my comments in the Weaknesses section. 

I have mixed feelings about this paper. On the one hand, the paper risks over-claiming on some high-level motivations as pointed out in the Weaknesses section, lacking a clear positioning concerning several important related works. The methodological level design is also not entirely new and intriguing. On the other hand, I acknowledge that using model weights as datasets is interesting, and the efforts of “scaling up” to over 60.000 models seem to be one of the biggest contributions to me.
limitations: limitations: The limitations and broader societal impact are discussed in the main paper.

Comments: title: post rebuttal
post rebuttal
comment: Thank you for clarifying the doubts. Most of my concerns have been addressed. I will update my rating. However, I still think the use of weights does introduce a large amount of difficulty in preprocessing. Also how to merge multiple LoRa is a big challenge now, and the token-based approach can easily generate multiple roles.
We appreciate your response and are happy to clarify further. 

**“Biggest concern is the collecting a large number of weights…”**

We agree that collecting a large number of model weights is more complex than a large number of images. However, the bigger picture of this work is the “idea of using weights as the dataset” (215A) in order to demonstrate "the existence of a LoRA model manifold” (rzPz). As mentioned in the global response, we introduce this idea of a **meta**-latent space **creating new models**, with “broad applications in generative modeling” (wMSF), which we believe is beyond just the personalization application.  We would like to note that although creating this space requires a notable amount of compute, using the space for the variety of applications is lightweight. We discuss these practical details below. 

**“...in terms of the number of parameters, Celeb-Basis only has 1024 learning parameters.”**

While Celeb-Basis uses 1024 coefficients to define an identity in token space, it requires optimizing an MLP at test-time with 525,312 parameters and backpropagation through the text-encoder, leading to \~3 minutes and \~21GB VRAM on a single NVIDIA A100. In contrast, we directly optimize 10,000 coefficients (\~2% of Celeb-Basis's parameters) without text-encoder backpropagation, reducing optimization time to \~1 minute and VRAM usage to \~8GB. Furthermore, our studies (Table 1 and 2 in the rebuttal PDF) show *w2w* offers better identity preservation without overfitting.

We would also like to note that Celeb-Basis works in token embedding space, which is already fairly low dimensional. In their case, they use 1024 dimensions to span the original 1536 dimensional space (768 dimensions for first name + 768 for last name), resulting in ~66% of the original dimensionality. In contrast, we compress a much higher dimensional LoRA space (100,000 dimensions) to 10,000 dimensions (10% original dimensionality). The fact that even LoRAs, which are by design low rank, have significantly lower rank semantic subspaces is quite intriguing, potentially inspiring further exploration of interpretable and controllable subspaces in model weights.


**“...what guarantees that information like cartoon drawing is not encoded in the principal components?”**

When you say, “the weight information represents a Buzz Lightyear,” the weights of the base Stable Diffusion model encodes prior concepts such as “Buzz Lightyear,” “Pixar,” etc, but not the LoRA weights added on top. During Dreambooth fine-tuning, we train LoRAs on realistic human identities, so the only new inserted concept is the identity. We apply PCA to the LoRAs, not the base model, and since there’s no style variation among the encoded identities, no principal components encode style. However, if “style-editing” means generating images of the identity in different styles with prompts, that’s possible because the base model retains prior concepts. For example, in Fig. 7 and Fig. 8, we generate a person as a statue or painting using prompts. Furthermore, in rebuttal PDF Fig. 8, we show that a separately trained style LoRA can be merged with an identity model from *w2w* space. 

**“... whether the identity information of each person can be maintained when generating multiple characters at the same time…”**

Thank you for the clarification. Identities from *w2w* space can be generated alongside existing characters or concepts in the base diffusion model (e.g., the celebrity examples in Figures 6 and 7 of the paper). However, two *w2w* models can't be naively merged by adding weights, as they share the same subspace and will interfere, leading to identity interpolation. Separate LoRAs can be trained to encode identities and then be merged with *w2w*, but they cannot live in the same subspace. This can be enforced using something like [1]. On the other hand, a LoRA encoding another concept such as style can be seamlessly merged as seen in rebuttal PDF Fig. 8. We will discuss this as a limitation in the paper. 

**Summary**

We hope we addressed your main concerns. Overall, we have shown from an efficiency and performance standpoint, that *w2w* performs stronger than Celeb-Basis as well as other baselines. More importantly, as you mention in your original review, our work provides “new ideas and insights into personalized generation models.” We believe that the merits of our work should not be subsumed by Celeb-Basis just because of the personalization application. If that were the case, then works like Dreambooth can be thought of as just Textual Inversion but optimizing model weights instead of token embeddings. But works like Dreambooth provide further insights beyond the application. 

[1] “Orthogonal Adaptation for Modular Customization of Diffusion Models.” Po et al. 2024
Thanks to the authors for their efforts during the rebuttal. The author's response solved part of my problem. But I still have a lot of concerns that haven't been clarified. 

1. The biggest concern is that collecting a large number of weights to build the w2wk space is far more complex than collecting a large number of images. In addition, in terms of the number of parameters, Celeb-Basis only has 1024 learning parameters. 

2. And for question 1, what guarantees that information like cartoon drawing is not encoded in the principal components? For example, the weight information represents a Buzz Lightyear. Wouldn't Pixar style and Buzz Lightyear images appear at the same time? Then, why is the new image created based on the weight information not affected by the Pixar style?

3. For question 2, perhaps my question is ambiguous and I am sorry for this, I am actually concerned with the generation of multi-subject, whether the identity information of each person can be maintained when generating multiple characters at the same time (I only saw two case shows in Figure 6 and 7 in the paper).
Thank you for your further comments. We are happy to hear that we sufficiently answered some of your concerns, and we appreciate you raising your score. We agree with you that the weight space and noise space are not completely independent. This can make for some interesting future work.
I appreciate the author's efforts in preparing the rebuttal. 

While I understand that this work focuses on **parameter weights**, I remain unconvinced (with a high level of confidence) that this parameter space and image space with noises should be completely independent, as the optimization of those parameters is driven by objectives that are intrinsically conditioned on the images. 

Nevertheless, as I noted in my initial review, the work has its merits, and the authors have addressed at least some of my concerns in the rebuttal. In light of this, I am raising the score to 6.
We thank all reviewers for their time and valuable comments on our work. We are grateful that this paper has overall been well-received by the reviewers. As the end of the discussion period is approaching, we kindly ask all reviewers to acknowledge if our rebuttal has addressed their concerns and give us an opportunity to address any further follow-up. Thank you again for your participation.