PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction
Authors: Shangyu Chen
Zizheng Pan
Jianfei Cai
Dinh Phung
Venue: ICLR 2025 Spotlight (2025)
Keywords: Text-to-Image diffusion model
Diffusion model fine-tuning
Primary area: generative models
TL;DR: Achieve lightweight and reliable personalized models through the subspace of the pre-trained Stable Diffusion model.

Abstract: Personalizing a large-scale pretrained Text-to-Image (T2I) diffusion model is chal-
lenging as it typically struggles to make an appropriate trade-off between its training
data distribution and the target distribution, i.e., learning a novel concept with only a
few target images to achieve personalization (aligning with the personalized target)
while preserving text editability (aligning with diverse text prompts). In this paper,
we propose PaRa, an effective and efficient Parameter Rank Reduction approach
for T2I model personalization by explicitly controlling the rank of the diffusion
model parameters to restrict its initial diverse generation space into a small and
well-balanced target space. Our design is motivated by the fact that taming a T2I
model toward a novel concept such as a specific art style implies a small generation
space. To this end, by reducing the rank of model parameters during finetuning, we
can effectively constrain the space of the denoising sampling trajectories towards
the target. With comprehensive experiments, we show that PaRa achieves great
advantages over existing finetuning approaches on single/multi-subject generation
as well as single-image editing. Notably, compared to the prevailing fine-tuning
technique LoRA, PaRa achieves better parameter efficiency (2× fewer learnable
parameters) and much better target image alignment.

Published as a conference paper at ICLR 2025
PARA: PERSONALIZING TEXT-TO-IMAGE DIFFUSION VIA PARAMETER RANK REDUCTION
Shangyu Chen & Zizheng Pan & Jianfei Cai & Dinh Phung
Monash University
Melboume, Australia (shangyu. chen, zizheng. pan, jianfei.com caii winc.aiiiiniiiiii Baahhhhphhgggmmmmmn edu
ABSTRACT
Personalizing a large-scale pretrained Text-to-Imae Iomae (T2I) diffusion model is chal- lenging as typically struggles to make an appropriate trade-off between train- ing data distribution and the target distribution, i.e., learning a novel concept with only few target images to achieve personalization (aligning with the personal- ized target) while preserving text editability (aligning with diverse text prompts). In this paper, we propose PaRa, an effective and efficient Parameter Rank Reduc- tion approach for T2I model personalization by explicitly controlling the rank of the diffusion model parameters to restrict its initial diverse generation space into a small and well-balanced target space. Our design is motivated by the fact that tam- ing a T21 model toward a novel concept such as a specific art style implies a small generation space. To this end, by reducing the rank of model parameters during finetuning, we can effectively constrain the space of the denoising sampling tra- jectories towards the target. With comprehensive experiments, we show that PaRa achieves great advantages over existing finetuning approaches single/multi- subject generation as well as single-image editing. Notably, compared to the pre- vailing fine-tuning technique LoRA, PaRa achieves better parameter efficiency (2x fewer learnable parameters) and much better target image alignment.
INTRODUCTION
Recent text-to-image (T2I) diffusion models (Rombach al., 2022; Podell al. 2023 OpenAl 2023 MidJourney) have achieved unprecedented success. However, despite being trained on large- scale datasets, most T2I models struggled to generate novel concepts as they were limited within their training data distribution. For example, pretrained Stable Diffusion (SD) models (Rombach jtt al. 2022) cannot generate unseen objects like a novel anime character. Thus, it has drawn in- creasing attention in the community to teach off- aff-the-shelf T2I models with few target images to learn a novel concept (e.g., a specific personal pet) for aligning with user preferences, which is known T21 model personalization.
Much effort has been made in this direction. Some works (Tia et al. 2023 Wei et al., 2023) seek for a general encoder to learn a novel concept without test-time finetuning. However, training such an encoder usually requires building a large collection of text-image pairs and expensive computing resources, e.g., 1.4 million pairs for the category of person in InstantBooth (Shi et al. 128 TPUv4 chips in SuTI (Chen et al., 2024). Another prevalent line of work directly finetunes T2I models based on the target images, which can be roughly categorized into using text embed- ding (Gal et al. 2022), cross-attention layers (Kumari et al. 2023), full model finetuning (Ruiz et al.) 2023). low-rank update (Hu et al., |021 cloneofsimo (022) or adjusting singular values of model parameters (Han et al. 2023). However, all these fine-tuning methods directly change the initial full generation space, which naturally results trade-off between generation diversity and the align- ment with the target concept. Consequently, they usually suffer from either poor alignment the new concept (Gal et al. 2022), or overfitting the few target text-image pairs (Ruiz et al. 2023).
2023) and
In this paper, introduce PaRa, ization via Parameter Rank Reduction. Our motivation comes from two folds: First, diffusion models trained capture their training data distribution, indicating diverse image generation
efffmeter-efficient framework for T2I model personal-

---

Published as a conference paper at ICLR 2025
Generate images with different prompis
"A small bear
LoRA
sitting on rockery in park"
PaRa
"A small
bear [V] sitting the grass"
"A small bear
"A small bear [V] sitting on boat"
"A holding a small bear [VT'
"A small bear [V] swimming in the swimming pool"
"A golden sculpture of small
[V] sitting on rockery in the park"
Input images bear plushic and their captions
[VT'
Decreasing relevance between prompts and training captions
Figure 1: Comparison between LoRA (cloneofsimo) 2022 and our propsoed PaRa on T21 person- alization for learning a new concept of bear, i... "(VT". For a fair comparison, we set the rank as and adopt the same latent noise for both methods. LoRA scale is set to 1.0.
space due to large-scale pretraining. However, taming T2I model for a novel concept may in- stead suggest a small restricted generation space. For example, finetuning SDs into specific anime style like Ghibli Studio does not requires strong photorealistic knowledge learned from LAION-5B dataset (Schuhmann et al., 2022]. Second, broad literature, the rank matrix has been shown to be effective to control the degree of certain restrictions such as in image compression (Andrews) & Patterson 1976), network pruning (Idelbayev & Carreira- Periiraa 2020) and parameter-effiiint finetuning (Hu al. 2021).
Based on the discussions above, our key idea is to explicitly control the rank of a layer output in diffusion model during the denoising sampling, thus effectively steering the image generation into a well-aligned and restricted space for learning a new concept. In practice, this is achieved by introducing low-rank learnable parameter B Raxr for pretrained linear projection Wo R where r is the reduced rank and is the number of hidden dimensions of this layer. Next, with simple QR decomposition of B QR, we can obtain a of orthonormal bases Q which can be adopted to reduce the rank of the initial layer outputs by Wa. QQ" Wa, where x denotes the layer input. We theoretically prove this result in Appendix [ Note that this is fundamentally different from LoRA (cloneofsimo) 2022 Hu et al. (2021), which adds low-rank updates to a layer output to achieve parameter-efficienct finetuning.
With extensive experiments, we demonstrate several advantages of PaRa. First, compared to the typ- ical LoRA fine-tuning (cloneofsimo) 2022), PaRa can generate images beyond its initial training prompts while maintaining well the consistency of the specific target concept, shown in Figure Furthermore, PaRa only requires half of the storage cost of LoRA since only needs one matrix B for each layer. Second, benefiting from the explicit rank control, further propose approach to combine multiple individually fine-tuned PaRa weights, which enables the blending of multiple per- sonalized concepts for multi-subject T2I generation. without the need for additional training with augmented multi-subject images (Han et al.) 2023). Third, PaRa is fully compatible with LoRA weights (Hu et al. 202T), supporting the combination of PaRa with a wide range of existing LoRA finetuned SD models in the public model zoo. Finally, thanks to the restricted small generation space, single image editing under PaRa quite stable the initial noise has negligible effect on the final image generation. For example, as demonstrated in the experimental section 4.3 under the same prompts, PaRa achieves a significantly higher average SSIM, compared to the baseline.
The major contributions of this work can be summarized as follows.
We propose Parameter Rank Reduction (PaRa), framework for personalizing T2I diffusion models by explicitly reducing the rank of the diffusion model parameters.
2

---

Published as a conference paper at ICLR 2025
We propose simple yet effective approach to combine multiple individually fine-tuned PaRa models, enabling the blending of multiple personalized concepts. Our model also facilitates single-image editing without the need to do the noise inversion (Song et al. 2020a; Mokady et al. 2023) in the diffusion process.
We conduct comprehensive experiments show that PaRa achieves tute-of-the-aa perfor- mance in personalized single/multi-suhjeet generation.
2
RELATED WORK
2.1
TEXT-TO-IMAGE DIFFUSION MODELS
Generative models exemplified by Diffusion Models ModelssChang Chang et 2023 Gu al. 2022; Ho al. 2020 Nichol & Dhariwal [202]] Song et al.) 2020al Sihl-tickstein et al. 2015 Song et al. 2020b) have achieved significant advancements in the past few years. Ho et al. (2020) proposed the Denois- ing Diffusion Probabilistic Models (DDPM), which enable diffusion models to achieve excellent performance in image generation tasks. Song et al. (2020a) further improved this approach by com- bining score function with diffusion probabilistic models, significantly enhancing the efficiency of image generation. DALL-E (Ramesh et al. 2022). Imagen (Saharia et al. 2022). ) and Stable Diffu- sion (Rombach et al. 2022) iteratively denoise data within latent space, trained on large image-text datasets, significantly enhancing the practicality and effectiveness of diffusion models.
Among the existing T2I diffusion models, Stable Diffusion (SD) (Rombach al. (022) is the most widely used one, which is a variant of Latent Diffusion Models (LDMs) (Rombach et al.) 2022). LDMs encode input images into latent code with encoder E, and then execute the denoising process on 2. The training objective is to minimize
E- [|c- ca (21, T(V))
where y is the input condition such as text or semantic maps, T is a domain domain-specific encoder of y. t is the sampling step, 6g (z,, f, 7 (v)) is the conditional denoising model with parameters 6. The denoising process is modeled as a reverse process of a fixed Markov Chain length T, and is uniformly sampled from (T).
SD uses a frozen CLIP text encoder (i.e. T) to encode prompt words (i.e. y) and a UNet Model (i.e. Eg) for the denoising process (Rombach et al.) (022). Our experiments primarily rely Stable Diffusion. The forward linear units in its UNet Model are fundamental to our proposed rank reduction.
2.2 FINE-TUNING GENERATIVE MODELS FOR PERSONALIZATION
Customizing and personalizing pre-trained text-to-image diffusion models has garnered significant research interest recently. Many methods have been proposed including enriching text embed-
dings (Gal et al. 2022). fine-tuning the UNet (Ruiz et al. 2023
Kumari et al. 2023 Hu et al.
2021 Gandikota et al. 2023) Han et al.) 2023; cloneolsimo 2022
Qiu et al. [2033] Yeh al. [023]]
Marjit et al.) 2024), and providing adapters (Mou et al. 2024 Zhang et al. 2023a) to control the
generated outcomes, as well as some training-free free approaches (Chen et al. 2024 Gal et al. 2023
Jia et al. 2023 Shi et al. 2023 Wei et al., 2023).
Among the fine-tuning based personalized T2I methods, one common and effective idea is via matrix decomposition and adjustment of its components, which have been introduced in early GAN-based generative models (Zhu et al., 2021 Feng et al., (002)). The widely-used LoRA (Hu et al. 2021 cloncofsimo) 2022] Gandikota et al. 2023) in diffusion models also follows the idea of matrix de- composition. Another method SVDiff (Han et al. 2023) uses SVD (singular value decomposition) to decompose matrices. However, these methods only focus on the scale of the obtained vector com- ponents. For example, LoRA (cloneotsimo 2022 Hu et al. 2021) adds the scale a to the original formula Wo + BA as
Wo + a&W = Wo + aBA, where Wo dxck Rdxr B
where Wo is matrix of a pretrained SD model. Note that since the column vectors of AW are not normalized, B and A inherently include the learning of scales for the components. Similarly,
3

---

Published as a conference paper at ICLR 2025
SVDiff (Han al. 2023) optimizes the scales of the diagonal matrix obtained from the SVD de- composition Wo.
Our PaRa method, in contrast, chooses to directly climinate certain components during personal- ization rather than adjusting their scales. Because neural networks often tend to overfit, climinating some components can be more stable and robust than adjusting their scales (Liu al.) 2013 Guo
et al. 2016 Han et al. 2013). Furthermore, eliminating components is an idempotent operation (meaning can be applied multiple times without changing the result go beyond the initial appli- cation), making model mixing or combination more stable, compared to scale adjustments.
2.3
VERSONALIZAAIONNBBAAN IMAGE EDITING IN DIFFUSION MODELS
Existing fine-tuning models Ruuz et al. 2023 Kumari et al., 2023; Hu et al. 2021 Gandikota et al. 2023 Han et al., 2023 Zhang al. 2023b) that attempt to perform image editing directly by "one-shot training and adjusting the text of the original training image" are prone to overfitting avoid overfitting, the diversity of images generated from the same text and Gaussian noise makes nearly impossible to reproduce the exact training images during generation. Therefore, personalized diffusion models require the inversion process (Song et al., 2020a, Mokady et al., (023) to lock in the noise during image editing. SVDiff (Han al., (023) demonstrates that
to the single training image. Even if they
treat the inversion process as optional component. However, omitting the inversion process still significantly impacts its faithfulness to the target image. In contrast, our model PaRa can maintain editability after only one-shot learning, and eliminate the need for the inversion process to achieve image editing.
3 METHOD
This section introduces the details of our PaRa model, including three sections. Firstly, we explain the fundamental principles and implementation of PaRa, including its formulation, training process, and application to convolutional layers. Secondly, we discuss how to combine two trained PaRa models, what the relationship is between PaRa and LoRA, and how PaRa can be effectively used in conjunction with pre-trained LoRA models. Lastly, we delve into the application of PaRa in single-image editing.
3.1 PARAMETER RANK REDUCTION
In this section, we describe our approach PaRa with a simple demonstration based on a linear pro- jection. Let Wo [d be the weight matrix of one linear projection in diffusion model, where d and & denote the number of input and output hidden dimensions, respectively. Given the input of this layer æ, the output be written as h Woz. Note that omit the bias term for simplicity. To reduce the output space (i.e., column space) of Wo, we introduce a learnable parameter B E Rddr where is the hyperparameter that controls the matrix rank. We first decompose B using QR de- composition = RR, where Q is an orthogonal matrix and is the corresponding upper triangular matrix. Based on Q, we then compute W reduce as
Wreduer Wo - QQ
Next, the rank-reduced layer outputs can be formulated
Wreduce x
Wor - QQ Wox.
(4)
With theoretical proof provided in Appendix.1 Eq. ensures that the column space W Wreduee is a subset of the column space of Wo. effectively reducing the dimension of the output while main- taining the key features learned by the model.
In other words, given Wo = - W2 wk-laxx= where nे denotes the column vector of Wo. we define the column space of Wo as So Span ( ... By adjusting Wo W ensure that the column space of Wreduce a subset of the column space of Wg. In practice, we initialize B to and finetune with few text-image pairs as the common practice in (Ruiz et al.
2023). Our goal is to evolve B into a set of orthonormal bases, and then adjust W, by subtracting its components these bases. This process ensures the column space effectively reduced.
4

---

Published as a conference paper at ICLR 2025
Moreover, note that QQ"W: now represents the components of Wo projected onto the column space of B. capturing the influence of the orthonormal basis derived from B on Wo. When has linearly independent columns, the reduced dimension will be the same as r, i.c. the column number of B. Since r is typically set to a small number e.g., 2 or 4, compared to hundreds/thousands of hidden dimensions in a layer output), B is likely have linearly independent columns after training.
Remark convolutional kernels. For the weights of convolutional layers, need employ a reshaping method similar to FSGAN (Robb et al.] 2020) before reducing the rank. Specifically, we reshape the convolution kernel weight Wo.come Reout X xhxw to the matrix as second- order tensor Wo Cout x (Cix (cxx)xxx) After this, we proceed with the steps of PaRa, setting B E Reou xhxw), B = QR, calculate and reshape QQ'W, from Cout X (Cin h x w) to Cout X Cin X W as AW. Finally, the rank-reduced kernel weight becomes
W Wreduce.cone Wo.com
AW,
(5)
where AW Rooo XCin Xhxw
3.2 COMBINING PARA
Our framework also supports the combinations of different personalized PaRa models well as combining with LoRA-base based personalized T2I models. Let's first consider two individually trained PaRa models
W1 = Wo - QQQ Wo, W/2 = Wo - Q2 Wo.
(6)
where Q1 = [Qi Qī2 ... lax,, and Q2 [921 922 .2.22aalaxas' We can merge Q1 and Q2 into Qm = [ qí2 Q..QQi, 921 922 ... Pr22 and reduce new orthonormal matrix Qnn by QR decomposition Qm Qmn P' Then the combined PaRa model has new weights W., = Wo Q' 2 Wo. In practice, a more convenient approach is via sequential addition to a diffusion model, where the first PaRa diffusion model is used as the new base diffusion model for the second one. This can be expressed as:
Wm W1 Q2Q = (Wo Q.QT Q2Q -Wo
We provide a proof in the Appendix A that this is equivalent to W,, Wo S.T.WW
Rank boundary. For a matrix Wo with rank k the reduced rank is too large compared k it can cause the original model to collapse and fail to generate images properly. Especially in the PaRa combination, we are uncertain about the exact value of the combined rank Teombines for which we only know that it is greater than r1 and r2, but less than V1 + 12. Thus, there is possibility that T Feombine could large enough to cause the original model to collapse. Therefore, for an PaRa, even though set the same r for the entire pre-trained model, for each layer's 's weight Wo in the pre-trained diffusion model, we impose an upper limit on the rank r that can be reduced, denoted as Tadjust "rank((o)) where y is a factor less than 1. For each Wo, the reduced rank is
Tadjust (r
if S yrank(Wo) [[rramk(WW) if > yrank(Wo)
(8)
Comparison and combination with LoRA. PaRa has a corresponding relationship with the formulation of LoRA (cloneofsimo) 2022). Wo + aBA, A, i.e., we can consider -Q as B and Q as A. This means that we can combine individually trained LoRA and PaRa as AW + a2.)) where a₁ and a2 parameters controlling the strength of PaRa and LoRA. However, this combination method treats PaRa entirely as LoRA, failing to pre- serve control over the diversity of the generated images.
(-aiQ (a
According to the rank-reduction property of PaRa, does not require the scaling parameter. In other words, PaRa effectively determines the optimal scale a, under which it reduces the rank. Therefore, we propose to combine the models as W Woombine = Wo - QQ W, + as.oA B.A. OLARA is the seale parameter of LoRA. In this way, we first reduce the rank and then add BA.
5

---

Published as a conference paper at ICLR 2025
A question naturally arises that whether it is feasible first apply LoRA and then reduce the rank of the new weights, i.e., Weombine = Wy + A feasible but does not work well in practice since the variations introduced by LoRA extend the acti- vation space of the initial layer, e-g., learning a new concept on top of the existing data distribution. Placing PaRa afterwards would lead PaRa to attempt to eliminate these variations. The larger the intersection between the column vector spaces of PaRa's Q and LoRA's B, the more the effect of LoRA will be diminished. An extreme example is that if Q the same as the orthogonal basis of B, then regardless of how LoRRA is adjusted, the LoRA model will not have any effect. Comparative experiments can be found in Section 4.4
(Wo a BA). This is technically
3.3 PERFORMING SINGLE IMAGE EDITING LIKE TEXT-TO-IMAGE GENERATION
PaRa can perform single image editing directly through one-shot training and adjusting the text of the original training image. PaRa offers solution by stabilizing the output, such that different Gaussian noises tend to yield the same result. This stability means that the model can generate images that closely resemble the training image, even when using various text prompts. This enables direct modification of the text prompt to facilitate image editing on the single training image.
By controlling the rank in PaRa, we can balance between faithful reconstruction and editability. When large rank is selected, the model produces images that very similar to the training image, enhancing reconstruction fidelity. Conversely, selecting a smaller rank increases the diversity of the generated images, improving editability. We provide detailed mathematical discussion of how PaRa achieves this balance and the role of linear algebra concepts in Appendix
EXPERIMENTS
The experiments evaluate PaRa on various tasks including single/multi-subjer generation and single image editing, together with ablation studies. The SDXL1.0 (Podell et al. (023) and the DDIM sampler with 1) = O are used for image generation. All experiments are conducted on a single A100 GPU with 40GB of VRAM.
4.1 SINGLE-SUBJECT GENERATION
Implementation details. We evaluated the effects of PaRa on customized single-subject generation, based on the Dreambooth dataset, where each label consists of five to six images. First, we verified that PaRa can indeed reduce the output space by confirming that the output diversity of PaRa has indeed decreased, compared Vanilla SDXL1.0. Then we compared the effects of PaRa different ranks with baseline methods on customized single-subject generation, including Dreambooth (Ruiz iet al. 2023). Textual Inversion (Gal et al. 2022), OFT (Qiu et al. 2023), SVDiff (Han al.) 2023), LyCoris (Yeh et al., 2023) and LoRA (Hu et al. 2021 cloneofsimo 2022). All baselines were trained for 1000 steps with a batch size of 1, and LoRA chose a rank of r 16 and scale of = 2.2 as the best model for fair comparison. In our experiments, we found that PaRa already achieved ideal results at around 200 steps. Therefore, we compare the results of PaRa at 200 steps with the baselines 1000 steps. Also, employ rank boundary 7 1/40 for PaRa.
Evaluation metrics. We quantify the generation diversity using the metric of average SSIM. SSIM (Wang et al., 2004) is commonly used to measure image similarity. For generated image set, calculate the SSIM for each pair of images and then compute the average value. We use the CLIP score cos( x c) measure the text alignment between the generated image and the text (Radford et al. 2021 ). and we compute 1 1 LLPPPS (x) to measure the image alignment between the generated image and the training image (Zhang et al. 2018). following the experimental design of SVDiff' (Han) et al. 2023) to create a comparison plot. Another benchmark we calculate for image alignment is the average CLIP image similarity scores cos(X. x) (Radford et al., (2021) between the training data and the generated image set (500 images). Here, the generated and training images are denoted as x and x, respectively, and the prompt denoted as c.
Comparison of output diversity. To verify that the diversity of the output has indeed decreased in PaRa, we compared the average SSIM of SDXI. 1.0 (Podell et al. 2023) and its finetuned version

---

Published as a conference paper at ICLR 2025
Table I: Average SSIM results of two different prompts (top and bottom) with different subjects (each column) to indicate the diversity of the generated images. We compared Vanilla SDXL (Podell jtt al., 2023) with our PaRa using ranks and 8. Higher average SSIM values suggest lower diversity and better al alignment with the train images. We bold the results of the highest average SSIM values.
[S] SCCLITURD MADE or GOLD"
DEAR FEAR-FLUSSI
D0C8
DUCSTOY
CREY SLOTE PLUSIE
MONSTER ToV
KID CARTIOO
WOLT PLUSIIC
VANELA SEAL
0.185210115
0.31225.015 001
0.30M83. 022
0,391 25 021
0,375224442
0.341=00017
0.25% 522
0.162230013
PARA
0.203210088
0.300:0-003
2465899.011
0.31635.010
0.366-20013
0.33755.01 014
0.178
TA [M AAAAt sisss SQUARR"
DO08
DUCLTOV
GKBV SEVIN JLLISIN
MONATAR J014
WOLT FUUSHI
Vo sExI.
0.294 130
0.356
10551..000 0..04
0:526 1d 67.076
0.267 0.010
0.31510.000
6.201 0.014
PARA
0-127 18:85
00003 16 088
2.20010 1.221.00000 203
333 1.0.0000 2202
0.276 0.017
B 01200 0. 811
0:705 (.B1
PARA-=5
0.399
0.0
0.313600.016
0.359e50.00 0.0
0.20
0.372=5:014
0.251=0.044
0.332=0.0.3
using PaRa. Based on 8 categories in Dreambooth, we generated 500 images for various prompts un- der different random seeds. Error bar is calculated by 5 runs. As shown Table on single-subject generation, PaRa demonstrates higher SSIM scores, indicating less diverse generations, compared to SDXL.. Furthermore, as shown in Fig- 5 on single-image editing, this advantage becomes more apparent since it achieves even better average SSIM scores. Overall, it supports our assumption that the image space of the linear transformation Wo + AW becomes smaller than that of Wo-
Comparison of generation quality. In Fig. 2 we compare PaRa with representative personal- ization techniques for T21 models, i.e., LoRA (Hu et al. 202T) and SVDiff (Han et al., 2023) Overall, PaRa consistently achieves better image alignment performance across different prompts under different ranks. Moreover, in Fig. 3 we benchmark PaRa across all subjects and report the text alignment and image alignment scores under different ranks. It can be seen that PaRa results are positioned in the lower right part, indicating that PaRa achieves much better image alignment. Note that the relatively lower rext-alignment ability of PaRa can be improved by reducing the rank In general, a larger value of usually helps to align the generated image with the learned concept, while a smaller value of r allows the text prompt to more effectively control the generated image.
To quantitatively compare the fidelity of the generated images the training images, we compared the average CLIP image similarity scores between PaRa and other personalization models, as shown in Table 2] Based on 8 categories in Dreambooth, we generated 500 images for the most basic prompt under different random seeds. The error bar is calculated from 5 runs. We compared LoRA with ranks 4 and 8 against PaRa with ranks 4 and 8. As shown in these cases, PaRa achieved the highest scores in most instances, specifically in 7 out of 8 cases. mostly occurring at rank 8.
Table 2: CLIP image similarity scores between the training data and the generated image set (500 images). Bold values indicate the best scores.
"A PHOTO or [NT
BEAR FLUSIIE
CAT
DUGS
DUCKTOY
CREY SLOTH FLUSHIE
MONSTEALTTY
REDECCRTTONN
WOLP PLUSHIE
PARA
0 +1 9156
0.9515=00000)
0.8700 0.0755
0.8915 00117
D.8102 4 0005
0/703 + +0.0252
0:3772
0756
0.8516=00166
PARA
80551 4 0004
0.9467 + 0.66
8.8955 0225
0.8971 0 0..114
BJ838 + 0136
07219-000 0.015
0.7955 + 07663
LORA
0:7741 + 0178
0. 3.3057 07744
7371 -8.166
0.7915 +0.01
07437 +0.0
0:5955
0.7194-0000
LORA
0200
0.8583 2 0:0246
0.8295 200115
0.3361 0.0243
0.7672 0.0012
0.2488 20.0251
0.7452 0.0024
0.7873
SYDur
0.39115
0.0145
DREAMBOUTIE
0.7921
2..9136
0.5893 z 0.0145
0.8302
200249
0.3520 0.0123
0.7736 0.0225
02:032000291
0.7818 c 0.0263
0,5067 00067=000 0.0169
TESTURS INVERsSION
(17431 4 31
0.80 0.8015 à 0.122
07433 0.022
8.7788 1 0.010
8.7525 0.0045
0.625 ...2274
0.7016 0.00273
0.6876 0.01
OFT
1181111 1
0.9024 a 07788
0.821 0.0771
0.3565 0..01
8.2913 0.0144
07708 10.0211
0.7972 0 01
0.8366 0.0158
4.2 MULTI- GENERATION
We demonstrate the effectiveness of combining PaRas which are individually trained on different subjects. Mixing two T2I model weights in previous works (Wu al., 2023] cloneotsimo 2022) generally result in problematic generation results, where they tend to over-emphasize one subject or mix two subjects to a 'hybrid' entity that does not exist in reality. Moreover, these methods require auxiliary techniques such as Cut-Mix-Unmin data augmentation (Han et al. (023). spatial condition (Gu et al. 2024). where the additional data annotation and training bring non-negligible financial or computational costs. In contrast, under our proposed PaRa, combining differently trained weights effectively eliminates the need for any auxiliary techniques while it can generate multiple subjects in single image with little unrealistic blending.
The results shown in Fig. were obtained using two challenging prompts. It can be observed that LoRA (Hu et al. 2021 cloneofsimo) 2022) only manages generate the primary concept,
7

---

Published as a conference paper at ICLR 2025
Train Image
PaRa
LoRA
SVDHn
1
r
r 4=
r p= 8
r 16
"A bear [V]
*A bear
[V] holding Lollipop"
A monster [V]"
"A monster [V] on skateboard in times square"
"A duck [V]
duck [V] sculpture made of gold"
"A wolf [V]"
"A cartoon style wolf [V] indoors"
Figure 2: Comparing the proposed PaRa with LoRA Hu et al. 2021 and SVDiff (Han et al. 2023) on single-subject generation. Each subject has 5 training images. PaRa includes results with ranks r ranging from I to 16. For LoRA, we adopt rank 8. We provide more generation results on Dreambooth, Textual Inversion, OFT and LyCoris in Appendix
Conceptl [VI], into the generated images, while the second concept, Concept2 [V2], significantly deviates from the original image. In contrast, PaRa successfully captures both concepts well. As Cut-Mix-Unmix introduces unfair advantages due to the extra training data, we did not include it in
our comparisons. More results can be found in Fig. 10 Appendix
Text digement neterd
Model
PaRa rank-1
LoRA
0.30
PaRa rarkm2
Texture Inversion
PaRa rank=44
SVDIff
PoRa rank--
0.28
PaRa rank=11
LyCoris
Dreambooth
0.22
0.24
0.26 Image Alignment (1 Better)
0.28
0.30
0.32
Figure 3: Text and image alignments for single subject generations. Text Alignment is measured by the CLIP score cos(x, c), and Image Alignmnet measured by LLPPP (x). The further to right indicates higher fidelity the personalized target, while higher along the vertical axis indicates improved text editability.
4.3 SINGLE IMAGE EDITING
In this section, we present the results of direct editing of a single image based on PaRa with = 8. The experiment in Fig. aiims to demonstrate that with properly chosen r. PaRa does not generate overly creative results and avoids the language drift issue (Ruiz et al. 2023). As comparison, we also present the results of SVDiff (Han et al., (203)) on the same task without DDIM inversion. SVDiff can also partially achieve image editing effects without noise tracking (e.g. DDIM inver- sion). Here, we can see that PaRa has significantly less deviation from the original image compared to SVDiff. Numerically, for each target image and prompt pair, generated 100 images and calcu- lated their average SSIM, where the result PaRa was significantly higher, reflecting the stability of the generated images.
8

---

Published as a conference paper at ICLR 2025
4.4 COMBINATION WITH LORA
In Fig. 6 we show the effects of combining PaRa and LoRA (Hu et al. 2021| cloneofsimo 2022
We adopt public pre-trained LoRA models (mec 2024 fig 2024 dia 2024) examples demon- strate the compatibility with PaRa weights. In general, PaRa complements existing LoRA weights as naturally combines the strength of both models, i.e., learned novel concept and image generation style. Here, we demonstrate our discussion in Section 3.2; during this combination, it is important first apply PaRa and then LoRA. Conversely, applying LoRA first and then PaRa will, as shown in the figure, severely affect the performance of LoRA.
4.5 ABLATION AND OTHER ANALYSIS
Parameter subsets and model sizes. We evaluated using different subset parameters UNet to update weights in Appendix B which includes the model sizes of PaRa within different parameter subsets for finetuning and the corresponding generation performance.
discuss the need for the rank boundary hyperparameter y to pre- vent the reduced rank from becoming too large, which could cause the generative model to collapse. In Section 4.2 we mention that our chosen rank boundary of y 1/40 based on empirical results. In Appendix we provide such empirical results by comparinging the effects of different
Rank bound 7. In Section 3.2
Comparing different rank reductions r. The effect of using different rank reductions in PaRa is one of the central topics of this paper. As r increases, the diversity of the generated images de- creases, leading to more faithful reconstructions in image editing; as r decreases, the diversity the generated images approaches that of the original models, improving text editability in image editing; r is too large, generative model collapses. Appendix [D] we compare the performance of different r values in image editing, multi-subject generation, and LoRA combination.
5 CONCLUSION
In conclusion, we have proposed PaRa, an effective and efficient framework for personalizing T21 diffusion models via parameter rank reduction. Our PaRa that finetunes with fewer parameters based on rank reduction can achieve better results than the prevailing LoRA fine-tuning. PaRa can well balance the diversity of generated images and the faithfulness to the customization objectives by selecting different ranks. PaRa is flexible framework and performs well in different applications including single-subject generation, multi-subject generation, and single-iman image editing.
Limitations. PaRa focuses only reducing the original output space. However, acknowledge that because current diffusion models are not yet universal models, sometimes customization might assume that the generative model
still need to expand the original output space. Although grows larger, the need for expansion will decrease, a model that can simultancously address both space extension and space reduction may be stronger. There are algebraic problems concerning space extension hidden here that might be worth exploring.

---

Published as a conference paper at ICLR 2025
Tuln aaga
-1 n- 12-22
MI
Concopts
locened
Trosept
Conerppt (15) wear
Sp.0e Togging Concepti PRs
atonding 2ing Weceee BEcc
handkreft teal Sath
Figure 4: Examples of multi- generation results. The PaRa results are generated with the reduced rank rı for Concept and r2 being 8 and for Concept 2. LoRA results are generated with the best rank of 16 for both concepts and the scale value of 1.
Inpul Images
PaRa Edited Images
SVDIIf Ediled Images
Ave. SSIM: O.0.0000000 0.0422 Ave. M9MM-0.177 Ave. S9IM: 122
Ave. SSIM O 3706 Ave. SSM--00465 Avo. SSIIE
Aun
pase purple
purple
bege"
transparent
photo of green diatue flwwrty by holcin gullien torch in hanz?
"shoto of green statue
photo of statue
photo of green statue of
ppoto of green stasue of
flbwrty hokcing gusidees lorel-fine torch hand
Eourly toiding sodeieo
Raerly halding a gaokien--aee hunele fire torch in hand
Eberly halldnng godeio
loeain pineapple in hane"
toran pineapple in hand"
Aun.
yed
black
coller
Figure 5: Single-image editing results. PaRa allows for image editing through one-shot learning of the original image and performs generation by directly modifying the prompt. We can see that PaRa achieves the expected modifications and preserves the personalized target subject well. In addition, PaRa achieves high consistency with untargeted elements of the initial image under different Gaus- sian noises.
LORA
PaRa train doy [V] [V]] [V] [V2] A dog [V] [V3] A [V] [V]] tay [V] [V2] [V] (V
dog [V]
A TOY
SKAN
Figure 6: Generation results of combining PaRa with LoRA. Top: the results first applying PaRa and then LoRA; Bottom: the results of first applying LoRA and then PaRa.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
diablo-sliders.
diablo-s_iidr33nttcaiyxy 2024.
Figurine-sliders.
http:///eivtaail.commoodti/
2024.
mecha-slider.
2024.
Andrews and CLIII Patterson. Singular value decomposition (svd) image coding. JEEE transac- fions Communications, 24(4):425 432, 1976.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301 100004 2023.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems, 36, 2024.
cloneofsimo. Low-rank adaptation for fast tex--to-image diffusion fine-tuning. https://
github. com www.cooooonooooooccoooooo
2022.
Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. Advances Neural Information Processing Systems, 35: 33054-33065, 2022.
Rinon Gal, Yuval Alaluf. Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Trans. Graph., 2023.
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint (XXx:2311.12099) 2023.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models, Advances in Neural Information Processing Systems, 36, 2024.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 22344,33,,2 2023.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6800-8551. 2020.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 106.000009955 2021.
11

---

Published as a conference paper at ICLR 2025
Yerlan Idelbayev and Miguel Á. Carreira-Perpinánl Low-rank compression of neural nets: Learning the rank of each layer. In CVPR, pp. 8046-80555 2020.
Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.0260 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convo- lutional neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 606-814,20 2015.
Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, and Pin-Yu Chen. Dif- fusekrona: A parameter efficient fine-tuning method for personalized diffusion model. arXiv preprint arXiv:2402.17412 2024.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Im- age synthesis and editing with stochastic differential equations. arXiv preprint 1080022008.0103,, 2021.
MidJourney. Midjourney.
http:///www.iddjourree com
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296- 4304, 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162 8171. PMLR, 2021.
OpenAl. Dall 3: Language models for image generation. papers/dall-e-3 pdf 2023.
https::/cdn. com/
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36.79320-79362. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763, PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Esther Robb, Wen Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang. Few-shot adaptation of gener- ative adversarial networks. arXiv preprint arXiv:2010. 11943, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuming text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Imagen: Text-to-image diffusion models. arXiv preprint arXiv:2205.11449 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jit- sev. LAION-5B: open large-scale dataset for training next generation image-text models. In NewrIPS, 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.034/2 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011..33455 2020b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15943-15953, 2023.
Xun Wu, Shaohan Huang, and Furn Wei. Mole: Mixture of lora experts. In The Twelfth International Conference on Learning Representations, 2023.
Shih- Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pullern recognition, pp. 586-595, 2018
Zhixing Zhang, Ligong Han, Amnb Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6027-6037, 2023b.
Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zheng-Jun Zha, Jingren Zhou, and Qifeng Chen. Low-rank subspaces in gans. Advances in Neural Information Processing Systems, 14:16648- 16658, 2021.
13

---

Published as a conference paper at ICLR 2025
Appendix
We organize our supplementary material as follows.
In Section we provide the theoretical basis for PaRa, including the proofs mentioned in
Section 3.1
In Section A we provide the proof mentioned in Section 3.2. explaining why PaRa Com- bination can be achieved through PaRa Sequential Addition.
In Section B
we provide a comparison of the model sizes and generated results when different subsets SDXL parameters are updated using PaRa.
In Section C compare the generating results of selecting different Rank Boundaries 7.
In Section D we compare the performance of different r values on the tasks of image
editing and PaRa model combination.
present additional generating results of PaRa, including comparisons with DreamBooth (Ruiz et al., 2023), Textual Inversion (Gal et al. 2022) and OFT (Qiu et al.) )023 on single-subject generation. We also provide more examples of effective PaRa combinations for multi-subject generation.
In Section B
In Section we provide mathematical argument demonstrating why directly generating with PaRa one-shot learning an effective method for image editing, supplement
to Section
In Section G we present the results of a survey conducted to evaluate the quality of the generated images, with 209 participants.
In Section H
we provide sufficient comparisons to demonstrate the improvement in training time achieved by our model.
In Section we provide overview of PaRa assist readers who may not be familiar with the structural details of Stable Diffusion and similar models like LoRA and DreamBooth in understanding work.
* In Section we present additional results of Single-image editing.
THEOREM AND PROOF OF PARA
Theorem 1. For matrix W, the image space of W is d-dimension vector space S4. If we have matrix Q [qi ... Te] with q Sa and vectors qs are mutually orthonormal, then W QQTW W has (d-r)-dimension image space.
This is actually an intuitive result, and minimize any potential misleading, we provide proof here.
Proof
Assume W is a b x k matrix. Denote W - QQ" W as [ī Tg... Jz). Denote W as [W] Wg... wxk. Present W QQ" W in vector form is
[7) W2... ūz = [1ū] W2... wk]
17i To...
[ W2
Whk
g
o Wi
Wi
o Wi
= [ Wo... Tike
l ..... grl
02 W1
W2
92 Wk
9 W1
T
of W
[1 W2... Wik]
E Te W1
E
42
Ei क. wfn
14

---

Published as a conference paper at ICLR 2025
For each column, it is ti =tū; Ei=1 q9 wi
The image space of W is a d-dimension vector space Sa, then we have d < b and d < k.
(W (Q"QQ"W) has rank (d r) means need to show 3 independent (Ø1 G2. ......r.rrrr gs E Sa. such that Vūū {u1, u2... ūgg J. 3 scalars 01, E R. s.t. Ui Σx= a.g.
Construct new set P containing all column vectors of the matrices Q and W. P = (qi. T2,.9,,... .... Gr. 11, W2 "V,,,...00 w. Apply the Gram-Schmidt process P, will get orthogonal ...........................
basis vectors (T1, U2,,.., (a)
Apply the Gram- Schmidt process P:
Stepl: 0j = 7i
Step2: T2 = 02-
Th
02
01 91
Step3: du = Ts
a 4 Q1
72 42
43
Step(r): U, = Gr
Step(r+1), 1): U/+1 = w111 - Ei=1
Qi
Step(r+2): Ur42 = 11²2 Σ
Go:ui G.
Ti
T 17 02-01
01
As we only have basis vectors, we will have the orthogonal basis {h, f2..4..... Tr, Ur+1,..., (id) finally.
Assume w. Ei=ı aiqi + Ei brū
J = 1 Ei=1 q: jjj
Ei aiqi + >1 -1rrt1 by Ei-1 प: (E-, u: + + būr)
El aiq 2i 1m biū Ei=1 aiqi (as प: प = l =
Eie+1 b
As we want proof 3(g F2. ... ..rrr s.t. 9r Sa, Vū, {ūn t2... ú =a. s.t.
11 E a.f.a ag E R.
so we have (01, 12........0 -2.... } = (ūr41...., s.t. g. E Sa, Vū, E (ūz, T.... Ueh. adarh..arrh
s.t. Ei=l d,gi, ai R
Therefore, W QQ"TQQ"""" W has (dmre-dimnnnoon image space.
The case not covered by this theorem is when there exists Q's column vector qे & S.. When & Sa, have q uig 0 for any column vector W, of W. That is, this component q will not update W in W QQQ" W. because Vē कंद Wi 0. In PaRa, if the PaRa training has yet converged, then this के will be updated. it has already converged, it indicates that our choice of r might be large, leading to over-parameeterizatio of PaRa, but the other & Sa can still achieve the objective of personalization.
Note:
A PROOF OF PARA SEQUENTIAL ADDITION IN PARA COMBINATION
Here we will prove that in PaRa combination, Wo Yn WmQ.W W, Q2QTW, Wo Qi of Wo Q2Q2 Wo + Q2Q2 Q Qf W.
For two trained PaRa models, we have parameters:
Q1 = 1@11 T12 -irnaaxx
Q2 I021 722 ...araa
15

---

Published as a conference paper at ICLR 2025
For the standard combination of Q1 and Q2: Qm
(Ti1 Q12 ...Qir, 421 922 ...02r -.02ralax(rr (x +ra))
With Qm R' Qm, same as what we dicussed Appendix A, we can find out a orthonormal basis
(uu, Q12...Qir.................... Pi, Piii.,....------- ) of rm vectors from the column vectors of Qi and Q2.
Wm Wo- 1 Wo means for each column vector wi, of W, it is transferred to
Vo ΣSS T
Σ (pr W
For the practical combination Wm Q2Q Q Wo, Wi is transferred to
W1 Q2 W1
Wo QiQF Q2 Wo +
22
82
r1
wi Σ
(. T Σ
1g Wi )2
ag ()))))))
920 12.
(10)
We have 02.
Σ3 Asj + Σl
batpi, and asj
ワij 925s dat = pi T2s then expression 10 is
F1
12
Pim
r2
vi;- Σ
Th uT,) Σ
Asjlis +
Σ
barp w2) Σe
( E (ni ub mb))
T2s
(11)
P'm
Wi
Σ
(ai . Th - - D
Anjlis +
Σ
ba(pt) W2 + >
(hi w1 (1)) F2e 72s
(12)
12 Pm -2
agy fij w 112
)22
Σ
(b.1pi 1.)
(13)
When substituting aaj and bat, the intermediate terms are eliminated, which is
Σ Thy
( TeaT w.)72.
(14)
T'm P1 12
wi Σ
th )
Σ
> (pt T2. -0,
(15)
The term E (pi (2) represents the total component of pi in the column space of Q2. As pi is orthogonal with (Ø11, //1 }. we have E (pt (2- p Hence, it is proven that Expression
g and Expression 15 are equal.
B COMPARISONS OF THE GENERATION PERFORMANCE FOR DIFFERENT SUBSET PARAMETERS
FxCA, as parameter subset, is choosed in our experiments. Here, we provide additional exam- ples generated using other subsets of parameters. We experimented with eight different parameter subsets.
Exclude Cross-Attention (ExCA)
This method trains all layers except cross-attention and time embedding layers.
Exclude Sell-Attention (ExSA)
This method trains all layers except self-attention layers.
Self-Attention Only (SAO)
16

---

Published as a conference paper at ICLR 2025
This method trains only the self-attention layers.
Cross-Attention Only (CAO)
This method trains only the cross-attention layers.
Full Model Training (FMT)
This method trains all layers of the model.
Strict Cross-Attention (SCA)
-hiis method trains only the queries and keys within the cross-attention mechanisms.
Exclude Cross-Attention High-Level (ExCA-HL)
This method trains all layers except cross cross-attention layers, with an emphasis on high- level feature representations.
Exclude Cross-Attention High-Level Last (ExCA-HI--Last)
This method trains all layers except cross-attention layers, focusing on the final stages of the high-level feature space.
As shown in Table 3 regardless of the subset used, the parameter count of PaRa significantly lower than that of LoRA. In Fig. / we use the example of bear plushie, with = 16, to illustrate the effects of different parameter subsets. counts, such as FMT, CAO, and ExSA, tend to align better with the training images. Conversely, models with smaller parameter counts may not align well with the target subject but match the text more closely. The model with the largest parameter count, FMT, even produced 'hybrid' result in the multi-subject example "A girl is holding a small bear [V]". Models like SAO and SCA strike better balance. EXCA, being well-performing subset, has numerous examples listed in other sections and will not be repeated here.
can be observed that models with larger parameter
SUBSET
PARA
LORA
PARA
LORA
PARA
LORA
EXCLUDE ATTSS-ATTENNION (ExCA)
L88 MB
4.8 MB
13 MB
33 MB
87
190 MB
EXCLUDE SELF-AFTENTION (EXSA)
1.9 MB
4.8 MB
13 MR
33 MR
87 MB
190 MB
SELP-AITENTION
21M
162M
ONLY (CAD)
1.6M
3.5M
11M
25M
82M
193M
FULL MODEL TRAINING (FMT)
3,4M
8.2M
23M
58M
169M
382M
STRICT
(SCA)
2.8M
7.9M
20M
62M
152M
EXCLUDE CROSS-AITENTES HIGH-LEVEL (EXCA-HL) HL)
276K
700K
1.9M
5.0M
14M
30M
EXCLUDE CROSS-ATTENTINN
HEX-A-HL-LAST)
82K
803K
Table 3:
Fine-tuning subsets of parameters in UNet, comparing them with LoRA at various ranks, along with their corresponding model sizes.
C COMPARISON OF DIFFERENT RANK BOUNDARIES
In Section 3.2 we discussed that setting the r too high can lead to model instability. To mitigate this issue, necessary to establish rank boundary 7 We empirically chose 7 across various experimental scenarios. The Fig. g illustrates the results generated with different values. Red markers indicate errors the generated images. The figure demonstrates that when is small, the setting of r has minimal impact. However, as r increases, it becomes crucial to select appropriate y, with values around 20 40 yielding the best performance.
D DIFFERENT r VALUES IN IMAGE EDITING AND MODEL COMBINATION
In Figure we compare the performance PaRa in one-shot learning for image editing across different rank values. We also calculate the average SSIM for each prompt generated with different ranks. It can be observed that the results for r 2 to r show minimal differences. In such cases, it is natural prefer complexity while maintaining comparable results. However, when is larger, such as r = 16 to 32, the generated results for more challenging images tend to degrade significantly.
2, it requires fewer parameters and reduces computational
17

---

Published as a conference paper at ICLR 2025
'A small bear [V] sitting boat'
'A small bear [V] sculpture made of gold'
'A girl is holding small bear [V]
'A small bear [V] swimming in the swimming pool'
'A small bear [V] is smoking
'A small bear [V] holding
Lollipop'
FMT
CAO
ExSA
SAO
SCA
EXCA-HL-- Last
ExCA-
HL
Figure 7: PaRa Single subject generation different parameter subsets
18

---

Published as a conference paper at ICLR 2025
boundary
1 100
ratio
7
7
7
7
7
small
bear [V]
sitting on
P= 64
boat
small
bear [V]
sculpture
made
gold'
'A small
bear [V]
sitting on
T 7= 24
boat
small
bear [V]
sculpture
made
gold'
Figure& 8: Comparison of Different Rank Boundaries for single subject generation. Red boxes
indicate emrors in generating "boat", and red circles indicate errors generating "gold".
Impui Imagees
PaRa Edited Images
T'
4
16
T
32
phots of grean statee
Ave. SSIM: 0.4396
Ave. SSIM: 0.4672
Ave. SSIM: 0.4267
Ave. SSIM: 0.3012
Ave. SSIM: 0.5210
tonch hand
"photo of green statue of liberty holding fire torch in hand"
block eanud worring bike coller ling bage ground
SSIM: 0.3221
SSIM: *A black-nased yellow dag wearing blue scarf is lying beige sandy
SSIM: 0.3255
SSIM: 0.3352
SSIM: 6.6882
Mack lnge
Ave. SSIM: 0.6629
Ave. SSIM: 0.6228
Ave. SSIM: 0.6640
Ave. SSIM: 0.6239
Ave. SSIM: 0.6527
"photo of purple chair with transporent legs"
Figure 9: Comparison Different Rank Image Editing.r = 2 to r = show minimal differences. When is larger, such as 16 to 32, the generated results for more challenging images tend to degrade significantly.
we compare the performance of PaRa with different ranks model combination. The results indicate that ranks between 2 and 8 are more suitable. When the rank is too large, the interaction between the two PaRa models becomes significant, potentially causing one subject to be ignored in the generated results. However, if there is primary and secondary subject, the rank of the PaRa corresponding to the primary subject can set higher. For example, in Figure [0] the dog and bear considered primary subjects, setting the other hand, for secondary subjects such as the can or vase, it is preferable to keep r as low as 2. This setting be applied to the general combination of primary and secondary objects.
In Figure TO
8 is appropriate capture their details. On
19

---

Published as a conference paper at ICLR 2025
AromplA: Concepti [VI]is trying To open Concept2 [V2] with pows, located nostalgic kitchen filled with vintage furniture and scattered biscuit....
Prompt B: Concepti [Vi] wearing blue apron topping Concept2 [V2] with miniature hammer standing on a wooden workbench with a red hondcraft tool cloth
PaRa
LoRA
PaRa
LoRA
Y2 16
F1 72 16
[V)
0=33
[V2) Can
liaciiee SIXXL
0=
0=
Images
72
72
72
72
[V]]
(V2) Vane
Buscine SDXL
0=
0
Images
64
7=
f2 8
f2
64
==1
Joy
[VS Can
Buscine SDXL
a=3
[V])
(Z2) lesport
Boscire SDXL
0=
0=3
Figure 10: Multi-Subjer Generation. In the PaRa comparison, we analyzed results for Concept with reduced ranks r set at 2, 8, 16; and for Concept 2, the reduced ranks 72 were 1, 2, 8. 64. LoRA, based on experimental optimization, used the best rank of for both concepts and compared different scales with alpha values of 3.
20

---

Published as a conference paper at ICLR 2025
Train Image
"A bear [V]""A monster [VT"
duck [V]
"A toy [V]"
"A wolf [V]"
"A bear [V]
"A monster
"a duck [V]
"A toy [V] in
cartoon
Generation prompts
holding
[V] on skateboard in times square"
sculpture
swimming pool"
style wolf [V] indoors"
Lollipop" "
made of gold"
PaRa
7
Textual Inversion
DreamBooth
OFT
LYCORIS
Figure 11: Comparison of Single-Subject Generation with DreamBooth, Textual Inversion, OFT and LyCoris
E MORE GENERATION RESULTS
In Figure 11 compare the performance of PaRa in single-subject generation with DreamBooth, Textual Inversion and OFT. The results clearly demonstrate the significant advantages of PaRa.
present more examples of PaRa combinations. In these examples, the primary subject rank is 2, and the secondary subject rank is 32, which we have found through experience to be relatively suitable pairing.
In Figure 12
21

---

Published as a conference paper at ICLR 2025
ULUIAES KAB
CARIION
MOLF PLOOUAE
BEAR PLUSEERAIII
DOGRWWAE
SLOTH FLUSHISTTS
HUMAN PROFERENCE PARA
96.175
07.13%
91.18%
91 1,0009
96.62%
$2.055
10.04%
78.459
55 75.66
Table 4: Human evaluation results comparing PaRa and LoRA. The first five columns compare single-subject generation, while the last four columns compare multi- combination. Each of the last four columns does not sum to 100% because there is a "Neither" option in the survey indicating dissatisfaction with both results.
F IMAGE EDITING ALGEBRACC DISCUSSION
The stability of PaRa outputs is reflected in that different Gaussian noises tend to yield the same result, represented as:
Wx = W(x + Az)
(16)
the equation is true when WAr 0, which is denoted by Ar kernel(W).
According to the rank-nullity theorem, for the linear transformation W X H. rank(W)+ + nullity(W) dim (The nullity of W is the dimension of W kemnel.) In PaRa, our reduced rank r = rank(W,) - rank(Wredune (eeduce) rank(W,) > rank(Wrreduc reduce) which implies nallity reduce) > nullity(Wo), more of Ar in PaRa will not produce different outputs. This reduced rank r demonstrates the problem (Meng al. (02T) of the trade-off between faithful reconstruction and editability in image editing, As increases, the modifiable features decrease, making the reconstruction more faithful. As r decreases, the modifiable features increase, bringing the diversity of the generated images closer to that of the underlying pre-trained generation model, and improving the editability. When large r is selected for training the PaRa on a single image, the model generates images that closely resemble the train- ing image, even when using various text prompts. This enables direct modification of the text prompt to facilitate image editing on the single train image.
= nullity(Wrenn (WWdeeee))))... nullity(Wo). we have
G SURVEY: USER ASSESSMENT OF SINGLE-SUBJECT AND MULTI-SUBJECT IMAGE GENERATION
To evaluate the quality of the generated images, we conducted survey where 209 participants provided their feedback. In order to streamline and ensure effectiveness, we designed a survey form consisting of nine questions. These questions allowed users to compare the image generation performance of PaRa and LoRA, covering both single-subject and multi-subject generation, as well as comparisons across different ranks.
The survey form shown in this link:
If you click on the survey link, you will find that participants are blind to which model corresponds to each image, which helps reduce subjective bias. The survey may have regional bias because 181 participants completed the Google Form, while 28 participants completed the Tencent Form. The survey consists of 9 questions. The first five questions are actually about PaRa and LoRA's Single-Subject generation. Options (a)-(e) correspond to PaRa ranks 1, 2, 4, 8, and 16, respectively. Options (f) and (g) correspond to LoRA rank 8 with scales 1.0 and 2.2, respectively. The last four questions about multi-subject generation, where options (a) and (b) correspond the results of PaRa and LoRA, respectively.
The overall results summarized in Table
22

---

Published as a conference paper at ICLR 2025
"A berry bowl [V2]'
"A can [V2]'
"A vase [V2]*
"A clock [V2]*
"A can [V2]*
Train Images
r 32
Train Images
p=
= 2
cat [VT
"A robot [VI
"A wolf (va)
sloth [VT
dog [V2]"
dog [VIT
"A dog [Ver"
"A dog Ivi
Figure 12: More Multi-Subject Generation Examples: The left column shows the training images of the primary subject, and the top row shows the training images of the secondary subject, each represented by a single image but actually based on five images for few-shot training. The prompts used here are "[V]] in the kitchen with [V2] next it."
23

---

Published as a conference paper at ICLR 2025
r=2
r=
r=32
r=
MODEL
PARA
LORA
PARA
LORA
PARA
LORA
PARA
LORA
MODEL SIZE
1.8 MB
4.8 MB
13 MB
33 MB
22 MB
56 MB
87 MB
190 MB
TRAIN. TIME
5.8MIN
6,2MIN
6.9MIN
10.4MIN
9.4MIN
15.1MIN
11.2MIN
M0.I
Table 5: Training time and model size comparison between LoRA and PaRa.
H How EFFICIENT IS PARA
In the Table 5
we compare the model size and training time of PaRa and LoRA when training for
1000 steps using the subset parameters of ExCA, as described in Appendix Section B It can be observed that as the selected rank increases, the training time advantage of PaRa becomes more pronounced.
I VISUAL OVERVIEW OF THE PARA FRAMEWORK
Figure 13 presents overview of the PaRa framework. It follows process similar to LoRA but focuses on the subspace of the entire model's output. For the linear weights W the U-net of the diffusion model, such the QKV units in the attention module of the Unet, originally W serves as a linear transformation, and its image space is Sh. If each column vector W is subtracted by its component base vector, resulting in the reduction of W rank to W reduces the dimension of S₁ will decrease. This ultimately reduces the diversity of the final output of the generative model.
Latent
h = Wr E S
Gonerated images
Freaeso
Sh.
Podd Space
reduosd bese vector
Sh.
Denoising U-Nett O KV
S smb
Suub
h W Wiluwnt Suub
Conditioning
Latent diffusion model
photo
Figure 13: Overview of the PaRa framework
ADDITIONAL EXAMPLES OF PARA IMAGE EDITING PERFORMANCE
Figure 14 provides additional examples showcasing PaRa's performance on different one-shot pairs derived from subjects in the DreamBooth dataset. These examples highlight the model's ''s robustness across diverse subjects and varying input pairs.
24

---

Published as a conference paper at ICLR 2025
A small bear sitting on rockary in the park
A small bear sitting by the leke the park
A small bear sitting front of wall
small bear sitting on the road
small bear sitting on the grass
Inpul Images
A small bear weering hat sitting the lake in the park
A amall bear wearing hat sitting in front of a red brick wall
A small beer wearing hat sitting rockery in the park
wearing while
wearrigg white
hat sitting on the road
hat sitting on the grass
PaRa Edited Images
A smmall bear frog head sitting rockery in the park
A smail bear with a frog head sitting by the lake in the park
small bear frag on head sitting in front of red brick wall
with
with
head sitting
head sitting
the road
the grass
A bowl placed pure white cloth
bow placed on pure
bowl piaced on a pure while cloth
bowi placed on a pure white cloth
A bowl placed pure white cloth
Input Images
cloth
A bawi filled with cucumbers placed on white clath
bowl filled with cucumbers
A bowl filled with placed pure white cloth
A bowl filled with placed pure white cloth
A bowl filled with placed white cloth
placed white cloth
PaRa Edited Images
A pink bowl placed on cloth
A pink bowl placed on a pure cloth
pink bowl placed pure white cloth
pink bowl while cloth
pink bowl white cloth
pure
pure
More single-image editing results: The top and bottom parts represent two different subjects, with input images sourced from the DreamBooth dataset. Each column shows the training input image and its corresponding edited result. The prompts used for PaRa-edited images displayed above each edited image.
Figure 14:
25


Reviews:
[{"review_id": "FaotQdOFxC", "reviewer": "Reviewer_gJ5u", "summary": "This paper proposes a framework for T2I model personalization called PaRa (Parameter Rank Reduction). It introduces an innovative approach to control the rank of diffusion model parameters, thereby constraining the initially diverse generation space to a smaller, more balanced target space. This enables the generation of personalized concepts and single-image editing. Furthermore, multiple individually fine-tuned PaRa modules can be combined to achieve the fusion of multiple personalized concepts. The framework also demonstrates higher parameter efficiency and better alignment with target images, with experimental results validating its effectiveness.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper is well-structured and easy to understand.\n2. The idea of achieving T2I model personalization by controlling the rank of diffusion model parameters is highly innovative. Additionally, detailed explanations are provided for the introduced learnable low-rank parameters.\n3. The experimental section is well-designed with sufficient data, demonstrating the effectiveness of PaRa in single/multi-subject generation and single-image editing, as well as its compatibility with other modules.", "weaknesses": "1. The methodology section includes extensive explanations of the mathematical principles behind PaRa but lacks an organized overview of the model’s framework. From the subsequent experimental section, it is evident that the approach also utilizes text embeddings, among other elements. While these are not the main focus of the methodology, they should be appropriately explained. Additionally, adding some visualizations in the methodology section would make the concepts more intuitive.\n2. In the last part of the introduction (lines 110–111), it states that \"PaRa achieves state-of-the-art performance in personalized single/multi-subject generation.\" However, the experiments do not provide sufficient comparisons to support this claim. Firstly, there is no comparison with encoder-based personalization methods mentioned in the introduction. Secondly, the fine-tuning-based personalization methods used in the experiments are not the latest approaches in this field, which weakens the persuasiveness of the results.\n3. In section 3.1, it is mentioned that “B is initialized to zero and fine-tuned with a few text-image pairs.” The experimental section should clarify the scale of the data used, the fine-tuning time, and other relevant information. Additionally, to support the claim of higher parameter efficiency, a computational efficiency analysis should be provided.\n4. From the image editing results, it appears that introducing PaRa may result in some loss of texture information. The analysis and interpretation of the experimental results are insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cewCMlxSxp", "reviewer": "Reviewer_TfPo", "summary": "A diffusion based image personalization method is presented via a parameter rank reduction technique. Particularly, the proposed method controls the rank of the diffusion model parameters to restrict the generation space into a small and well-balanced target space, achieving trade-off between the training data distribution and the target distribution. The motivation of the method lies in two main aspects. Firstly, this paper assumes that taming a T2I model toward a novel concept implies a small generation space. Secondly, the rank of matrix is important for parameter efficient fine-tuning. In this case, by reducing the rank of model parameters during finetuning, the proposed method is proven to achieve advantages over existing finetuning approaches on single/multi-subject generation and single-image editing.", "questions": "1. One important motivation of the paper is that taming a T2I model toward a novel concept implies a small generation space, making it possible to perform one-shot training for the low-rank parameter B. I'm curious about the relationship between edibility and the optimal rank of B. Is it always possible to learn a good low-rank B to perform reasonable editing in one-shot manner? It's just an open question. Please share your experiences.\n3. As a one-shot training techniques for image editing, how the model perform with different one-shot pairs? Please explain robustness of the model.\n4. Both numbers and visualisation are good in explaining the superiority of Para compared with existing techniques. Then, what are the limitations of Para? Please provide a rough failure case analysis.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Solving personalisation from matrix decomposition is a reasonable and good idea.\n2. Combing rank with edibility, especially diversity of the generated image is proven effective in this paper.\n3. The experiments are convincing in explaining the superiority of the proposed solution.", "weaknesses": "1. Matrix decomposition is proven to be effective in parameter efficient fine-tuning tasks, e.g. image editing. Although the difference between the proposed solution to Lora is clear. As both methods are based on matrix decomposition, it's not clear what are the foundational differences, and how the authors come up with the current solution.\n2. SSIM is used in this paper for image diversity evaluation. I'm not quite sure whether SSIM is the best one, as SSIM focuses on pixel-level difference instead of semantic level difference. e.g. position difference of the same instance may contribute more to SSIM than than an extra accessory. \n3. Writing should be improved to highlight the contributions, especially how the solution is formed, and how it is different from the existing techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N4C8qSB5Rc", "reviewer": "Reviewer_h7W1", "summary": "This work aims to balance personalization and text editability by reducing the rank of model parameters, effectively narrowing the generation space to better align with target concepts. This method is more parameter-efficient and achieves better target image alignment than existing techniques like LoRA. It also supports combining multiple personalized models and facilitates stable single-image editing without additional noise inversion processes. The paper shows PaRa’s effectiveness through comprehensive experiments on single and multi-subject generation tasks.", "questions": "The only concern is insufficient comparisons to SOTA, adding which would allow a better comparison.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Parameter Rank Reduction (PaRa) method is a creative solution to the challenge of T2I model personalization, offering a new perspective on controlling the generation space.\n2. PaRa demonstrates significant parameter efficiency, requiring 2× fewer learnable parameters compared to existing methods like LoRA.\n3. The paper provides extensive experimental results showing PaRa’s advantages in single/multi-subject generation and single-image editing and shows better results than LoRA.\n4. The paper is very well written and presented.", "weaknesses": "1. The authors have not compared their method with SOTA such as LyCoris, DiffuseKronA, etc. Including these results would provide a better comparison of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6TIhogxpBn", "reviewer": "Reviewer_PSkP", "summary": "This paper proposes a new method to personalizing diffusion model with few samples, which is to reduce the rank of the original weights in diffusion models. This method allows the finetuned model to have smaller diversity and better alignment with the training data. This paper conducts extensive experiments on various generation task including single subject, multi-subject generation and editing. The paper is well written and easy to understand.", "questions": "Please refer to the concerns in the Weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper provides a very interesting ideas to personalize diffusion model by reducing the rank of the weights. \n2) The proposed finetuning method is simple, effective and show good flexibility.", "weaknesses": "1) Different hyper-parameters may affect the alignment of the previous methods, such as the Rank of LoRA and fine-tuning steps. Authors may need to provide more clarification on how these hyper-parameters are chosen for the compared methods to make sure they achieve good enough alignment in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)