PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction

## Appendix A Theorem and Proof of PRR

###### Theorem 1 .

For matrix $W$, the image space of $W$ is a d-dimension vector space $S_{d}$. If we have matrix $Q=\begin{bmatrix}\vec{q_{1}}\ \vec{q_{2}}\ ...\ \vec{q_{r}}\end{bmatrix}$ with $q_{i}\in S_{d}$ and vectors $q_{i}$ are mutually orthonormal, then $W-QQ^{T}W$ has a (d-r)-dimension image space.

This is actually an intuitive result, and to minimize any potential misleading, we provide a proof here.

Proof

The dimension of the image space of a matrix is same as the dimension of the column space of the matrix.

For all column vectors of $Q$ and $W$, $\{\vec{q}_{1},\vec{q}_{2}...\vec{q}_{r},\vec{w}_{i}\}\in S_{d}$, we perform the Gram-Schmidt process to obtain an orthonormal basis $\{\vec{q}_{1},\vec{q}_{2}...\vec{q}_{r},\vec{v}_{1},\vec{v}_{2}...\vec{v}_{d-r}\}$ of the image space of $W$.

$W-QQ^{T}W$ transforms each column vector $\vec{w_{i}}$ of $W$ into $\vec{w}_{i}-\sum_{j=1}^{r}(\vec{q}_{j}\cdot\vec{w}_{i})\vec{q}_{j}$, which is a linear combination of $\{\vec{v}_{1},\vec{v}_{2}...\vec{v}_{d-r}\}$. Then $W-QQ^{T}W$ has a (d-r)-dimension image space.

Detailed Proof

Assume $W$ is a $b\times k$ matrix.
Denote $W-QQ^{T}W$ as
$[\vec{u}_{1}\ \vec{u}_{2}...\ \vec{u}_{k}]$. Denote $W$ as $[\vec{w}_{1}\ \vec{w}_{2}...\ \vec{w}_{k}]$.
Present $W-QQ^{T}W$ in a vector form is

$[\vec{u}_{1}\ \vec{u}_{2}...\ \vec{u}_{k}]=[\vec{w}_{1}\ \vec{w}_{2}...\ \vec{
w}_{k}]-[\vec{q}_{1}\ \vec{q}_{2}...\ \vec{q}_{r}]\left[\begin{matrix}\vec{q}_
{1}^{T}\\
\vec{q}_{2}^{T}\\
...\\
\vec{q}_{r}^{T}\\
\end{matrix}\right][\vec{w}_{1}\ \vec{w}_{2}...\ \vec{w}_{k}]$

$=\left[\vec{w}_{1}\ \vec{w}_{2}...\ \vec{w}_{k}\right]-\left[\vec{q}_{1}\ \vec
{q}_{2}...\ \vec{q}_{r}\right]\left[\begin{matrix}\vec{q}_{1}^{T}\vec{w}_{1}&
\vec{q}_{1}^{T}\vec{w}_{2}&...&\vec{q}_{1}^{T}\vec{w}_{k}\\
\vec{q}_{2}^{T}\vec{w}_{1}&\vec{q}_{2}^{T}\vec{w}_{2}&...&\vec{q}_{2}^{T}\vec{
w}_{k}\\
...&...&...&...\\
\vec{q}_{r}^{T}\vec{w}_{1}&\vec{q}_{r}^{T}\vec{w}_{2}&...&\vec{q}_{r}^{T}\vec{
w}_{k}\end{matrix}\right]$

$=\left[\vec{w}_{1}\ \vec{w}_{2}...\ \vec{w}_{k}\right]-\left[\begin{matrix}
\sum_{i=1}^{r}\vec{q}_{i}\vec{q}_{i}^{T}\vec{w}_{1}&\sum_{i=1}^{r}\vec{q}_{i}
\vec{q}_{i}^{T}\vec{w}_{2}&...&\sum_{i=1}^{r}\vec{q}_{i}\vec{q}_{i}^{T}\vec{w}
_{k}\end{matrix}\right]$

For each column, it is $\vec{u}_{j}=\vec{w}_{j}-\sum_{i=1}^{r}\vec{q}_{i}\vec{q}_{i}^{T}\vec{w}_{j}$

The image space of $W$ is a d-dimension vector space $S_{d}$, then we have $d<b$ and $d<k$.

$W-QQ^{T}W$ has a (d-r)-dimension image space means:

$\exists\{\vec{g}_{1},\vec{g}_{2},...\vec{g}_{d-r}\}$, s.t. $\vec{g}_{i}\in S_{d}$, $\forall\vec{u}_{i}\in\{\vec{u}_{1},\ \vec{u}_{2}...\ \vec{u}_{k}\}$, $\exists\{a_{1},...a_{d-r}\}$, s.t. $\vec{u}_{i}=\sum_{i=1}^{d-r}\vec{a}_{i}\vec{g}_{i},a_{i}\in\mathbb{R}$

Construct a new set $P$ containing all column vectors of the matrices $Q$ and $W$, $P=\{\vec{q}_{1},\vec{q}_{2},...,\vec{q}_{r},\vec{w}_{1},\vec{w}_{2},...,\vec{w
}_{k}\}$. Apply the Gram-Schmidt process to $P$, we will get $d$ orthogonal basis vectors $\{\vec{v}_{1},\vec{v}_{2},...,\vec{v}_{d}\}$

Apply the Gram-Schmidt process to $P$:

Step1: $\vec{v}_{1}=\vec{q}_{1}$

Step2: $\vec{v}_{2}=\vec{q}_{2}-\frac{\vec{q}_{1}\cdot\vec{q}_{2}}{\vec{q}_{1}\cdot
\vec{q}_{1}}\vec{q}_{1}=\vec{q}_{2}$

Step3: $\vec{v}_{3}=\vec{q}_{3}-\frac{\vec{q}_{1}\cdot\vec{q}_{3}}{\vec{q}_{1}\cdot
\vec{q}_{1}}\vec{q}_{1}-\frac{\vec{q}_{2}\cdot\vec{q}_{3}}{\vec{q}_{2}\cdot
\vec{q}_{2}}=\vec{q}_{3}$

…

Step(r): $\vec{v}_{r}=\vec{q}_{r}$

Step(r+1): $\vec{v}_{r+1}=\vec{w}_{1}-\sum_{i=1}^{r}\frac{\vec{q}_{i}\cdot\vec{w}_{1}}{
\vec{q}_{i}\cdot\vec{q}_{i}}\vec{q}_{i}$

Step(r+2): $\vec{v}_{r+2}=\vec{w}_{2}-\sum_{i=1}^{r}\frac{\vec{q}_{i}\cdot\vec{w}_{2}}{
\vec{q}_{i}\cdot\vec{q}_{i}}\vec{q}_{i}-\frac{\vec{v}_{1}\cdot\vec{w}_{2}}{
\vec{v}_{1}\cdot\vec{v}_{1}}\vec{v}_{1}$

…

As we only have $d$ basis vectors, we will have the orthogonal basis $\{\vec{q}_{1},\vec{q}_{2},...,\vec{q}_{r},\vec{v}_{r+1},...,\vec{v}_{d}\}$ finally.

Assume $\vec{w}_{j}=\sum_{l=1}^{r}a_{l}\vec{q}_{l}+\sum_{l=r+1}^{d}b_{l}\vec{v}_{l}$

$\vec{u}_{j}=\vec{w}_{j}-\sum_{i=1}^{r}\vec{q}_{i}\vec{q}_{i}^{T}\vec{w}_{j}\\
=\sum_{l=1}^{r}a_{l}\vec{q}_{l}+\sum_{l=r+1}^{d}b_{l}\vec{v}_{l}-\sum_{i=1}^{r
}\vec{q}_{i}\vec{q}_{i}^{T}(\sum_{l=1}^{r}a_{l}\vec{q}_{l}+\sum_{l=r+1}^{d}b_{
l}\vec{v}_{l})\\
=\sum_{l=1}^{r}a_{l}\vec{q}_{l}+\sum_{l=r+1}^{d}b_{l}\vec{v}_{l}-\sum_{l=1}^{r
}a_{l}\vec{q}_{l}\quad\text{(as $\vec{q}_{i}^{T}\vec{q}_{l}=1$ if $i=l$, $\vec
{q}_{i}^{T}\vec{q}_{l}=0$ if $i\neq l$, $\vec{q}_{i}^{T}\vec{v}_{l}=0$ if $l
\geq r+1$)}\\
=\sum_{l=r+1}^{d}b_{l}\vec{v}_{l}$

As we want to proof $\exists\{\vec{g}_{1},\vec{g}_{2},...\vec{g}_{d-r}\}$, s.t. $\vec{g}_{i}\in S_{d}$, $\forall\vec{u}_{i}\in\{\vec{u}_{1},\ \vec{u}_{2}...\ \vec{u}_{k}\}$, $\exists\{a_{1},...a_{d-r}\}$, s.t. $\vec{u}_{i}=\sum_{i=1}^{d-r}\vec{a}_{i}\vec{g}_{i},a_{i}\in\mathbb{R}$,

so we have $\{\vec{g}_{1},\vec{g}_{2},...\vec{g}_{d-r}\}=\{\vec{v}_{r+1},...,\vec{v}_{d}\}$, s.t. $\vec{g}_{i}\in S_{d}$, $\forall\vec{u}_{i}\in\{\vec{u}_{1},\ \vec{u}_{2}...\ \vec{u}_{k}\}$, $\exists\{a_{1},...a_{d-r}\}$, s.t. $\vec{u}_{i}=\sum_{i=1}^{d-r}\vec{a}_{i}\vec{g}_{i},a_{i}\in\mathbb{R}$

Therefore, $W-QQ^{T}W$ has a (d-r)-dimension image space.

###### Theorem 1 .

## Appendix B Comparisons of the generation performance for different subset parameters

EXCA, as a parameter subset, is choosed in our experiments. Here, we provide additional examples generated using other subsets of parameters. We experimented with eight different parameter subsets.

- •
Exclude Cross-Attention (ExCA)
–
This method trains all layers except cross-attention and time embedding layers.
- •
Exclude Self-Attention (ExSA)
–
This method trains all layers except self-attention layers.
- •
Self-Attention Only (SAO)
–
This method trains only the self-attention layers.
- •
Cross-Attention Only (CAO)
–
This method trains only the cross-attention layers.
- •
Full Model Training (FMT)
–
This method trains all layers of the model.
- •
Strict Cross-Attention (SCA)
–
This method trains only the queries and keys within the cross-attention mechanisms.
- •
Exclude Cross-Attention High-Level (ExCA-HL)
–
This method trains all layers except cross-attention layers, with an emphasis on high-level feature representations.
- •
Exclude Cross-Attention High-Level Last (ExCA-HL-Last)
–
This method trains all layers except cross-attention layers, focusing on the final stages of the high-level feature space.

As shown in Table 1, regardless of the subset used, the parameter count of PRR is significantly lower than that of LoRA. In Fig. 1, we use the example of a bear plushie, with $r=16$, to illustrate the effects of different parameter subsets. It can be observed that models with larger parameter counts, such as FMT, CAO, and ExSA, tend to align better with the training images. Conversely, models with smaller parameter counts may not align well with the target subject but match the text more closely. The model with the largest parameter count, FMT, even produced a ’hybrid’ result in the multi-subject example "A girl is holding a small bear [V]". Models like SAO and SCA strike a better balance. EXCA, being a well-performing subset, has numerous examples listed in other sections and will not be repeated here.

**Table 1: Fine-tuning subsets of parameters in UNet, comparing them with LoRA at various ranks, along with their corresponding model sizes.**
|  | $r$=2 |  | $r$=16 |  | $r$=128 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Subset | PRR | LoRA |  | PRR | LoRA |  | PRR | LoRA |
| Exclude Cross-Attention (ExCA) | 1.8 MB | 4.8 MB |  | 13 MB | 33 MB |  | 87 MB | 190 MB |
| Exclude Self-Attention (ExSA) | 1.9 MB | 4.8 MB |  | 13 MB | 33 MB |  | 87 MB | 190 MB |
| Self-Attention Only (SAO) | 1.6M | 3.1M |  | 11M | 21M |  | 82M | 163M |
| Cross-Attention Only (CAO) | 1.6M | 3.5M |  | 11M | 25M |  | 82M | 193M |
| Full Model Training (FMT) | 3.4M | 8.2M |  | 23M | 58M |  | 169M | 382M |
| Strict Cross-Attention (SCA) | 1.2M | 2.8M |  | 7.9M | 20M |  | 62M | 152M |
| Exclude Cross-Attention High-Level (ExCA-HL) | 276K | 700K |  | 1.9M | 5.0M |  | 14M | 30M |
| Exclude Cross-Attention High-Level Last (ExCA-HL-Last) | 6.9K | 53K |  | 42K | 403K |  | 82K | 803K |

Figure: Figure 1: PRR Single subject generation on different parameter subsets
Refer to caption: figure/MoreSubset.pdf

## Appendix C Proof of PRR Sequential Addition in PRR Combination

Here we will prove that in PRR combination, $W_{0}-Q_{m}^{\prime}Q_{m}^{\prime T}W_{0}=W_{1}-Q_{2}Q_{2}^{T}W_{1}=W_{0}-Q_{1
}Q_{1}^{T}W_{0}-Q_{2}Q_{2}^{T}W_{0}+Q_{2}Q_{2}^{T}Q_{1}Q_{1}^{T}W_{0}$.

For two trained PRRs, we have parameters:

$Q1=\begin{bmatrix}\vec{q_{11}}\ \vec{q_{12}}\ ...\vec{q_{1r_{1}}}\end{bmatrix}
_{d\times r_{1}}$

$Q2=\begin{bmatrix}\vec{q_{21}}\ \vec{q_{22}}\ ...\vec{q_{2r_{2}}}\end{bmatrix}
_{d\times r_{2}}$.

For the standard combination of $Q_{1}$ and $Q_{2}$: $Qm=\begin{bmatrix}\vec{q_{11}}\ \vec{q_{12}}\ ...\vec{q_{1r_{1}}}\vec{q_{21}}
\ \vec{q_{22}}\ ...\vec{q_{2r_{2}}}\end{bmatrix}_{d\times(r_{1}+r_{2})}$

With $Q_{m}^{\prime}R_{m}^{\prime}=Q_{m}$, same as what we dicussed in Appendix A, we can find out a orthonormal basis $\{\vec{q}_{11},\vec{q}_{12}...\vec{q}_{1r_{1}},\vec{p}_{1},\vec{p}_{2}...\vec{
p}_{r_{m}-r_{1}}\}$ of $r_{m}$ vectors from the column vectors of $Q_{1}$ and $Q_{2}$.

$W_{m}=W_{0}-Q_{m}^{\prime}Q_{m}^{\prime T}W_{0}$ means for each column vector $\vec{w_{i}}$ of $W$, it is transferred to

$$ $\displaystyle\vec{w_{i}}-\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q} _{1j}-\sum_{l}^{r_{m}-r_{1}}(\vec{p}_{l}\cdot\vec{w}_{i})\vec{p}_{l}$ (1) $$

For the practical combination
$W_{m}=W_{1}-Q_{2}Q_{2}^{T}W_{1}=W_{0}-Q_{1}Q_{1}^{T}W_{0}-Q_{2}Q_{2}^{T}W_{0}+
Q_{2}Q_{2}^{T}Q_{1}Q_{1}^{T}W_{0}$, $\vec{w_{i}}$ is transferred to

$$ $\displaystyle\vec{w_{i}}-\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q} _{1j}-\sum_{s}^{r_{2}}(\vec{q}_{2s}\cdot\vec{w}_{i})\vec{q}_{2s}+\sum_{s}^{r_{ 2}}((\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q}_{1j})\cdot\vec{q}_{ 2s})\vec{q}_{2s}$ (2) $$

We have $\vec{q}_{2s}=\sum_{j}^{r_{1}}a_{sj}\vec{q}_{1j}+\sum_{l}^{r_{m}-r_{1}}b_{sl}
\vec{p}_{l}$, and $a_{sj}=\vec{q}_{1j}\cdot\vec{q}_{2s}$, $b_{sl}=\vec{p}_{l}\cdot\vec{q}_{2s}$ then expression 2 is

$$ $\displaystyle\vec{w_{i}}-\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q} _{1j}-\sum_{s}^{r_{2}}(\sum_{j}^{r_{1}}a_{sj}\vec{q}_{1j}+\sum_{l}^{r_{m}-r_{1 }}b_{sl}\vec{p}_{l}\cdot\vec{w}_{i})\vec{q}_{2s}+\sum_{s}^{r_{2}}((\sum_{j}^{r _{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q}_{1j})\cdot\vec{q}_{2s})\vec{q}_{2s}$ (3) $\displaystyle=\vec{w_{i}}-\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q }_{1j}-\sum_{s}^{r_{2}}((\sum_{j}^{r_{1}}a_{sj}\vec{q}_{1j}+\sum_{l}^{r_{m}-r_ {1}}b_{sl}\vec{p}_{l})\cdot\vec{w}_{i})\vec{q}_{2s}+\sum_{s}^{r_{2}}((\sum_{j} ^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q}_{1j})\cdot\vec{q}_{2s})\vec{q}_{ 2s}$ (4) $\displaystyle=...-\sum_{s}^{r_{2}}((\sum_{j}^{r_{1}}a_{sj}\vec{q}_{1j}\cdot \vec{w}_{i})-((\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q}_{1j}) \cdot\vec{q}_{2s}))\vec{q}_{2s}+\sum_{s}^{r_{2}}\sum_{l}^{r_{m}-r_{1}}(b_{sl} \vec{p}_{l}\cdot\vec{w}_{i})\vec{q}_{2s}$ (5) $$

When substituting $a_{sj}$ and $b_{sl}$, the intermediate terms are eliminated, which is

$$ $\displaystyle\vec{w_{i}}-\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q} _{1j}+\sum_{s}^{r_{2}}\sum_{l}^{r_{m}-r_{1}}(\vec{p}_{l}\cdot\vec{q}_{2s}\vec{ p}_{l}\cdot\vec{w}_{i})\vec{q}_{2s}$ (6) $\displaystyle=\vec{w_{i}}-\sum_{j}^{r_{1}}(\vec{q}_{1j}\cdot\vec{w}_{i})\vec{q }_{1j}+\sum_{l}^{r_{m}-r_{1}}\sum_{s}^{r_{2}}(\vec{p}_{l}\cdot\vec{q}_{2s}\vec {q}_{2s})\vec{p}_{l}\cdot\vec{w}_{i}$ (7) $$

The term $\sum_{s}^{r_{2}}(\vec{p}_{l}\cdot\vec{q}{2s}\vec{q}{2s})$ represents the total component of $\vec{p}_{l}$ in the column space of $Q_{2}$. As $\vec{p}_{l}$ is orthogonal with $\{\vec{q}_{11},\vec{q}_{12}...\vec{q}_{1r_{1}}\}$, we have $\sum_{s}^{r_{2}}(\vec{p}_{l}\cdot\vec{q}_{2s}\vec{q}_{2s})=\vec{p}_{l}$. Hence, it is proven that Expression 1 and Expression 7 are equal.

## Appendix D compare different rank boundary γ 𝛾 \gamma italic_γ

## Appendix E different r 𝑟 r italic_r values in image editing and LoRA combination

## Appendix F More generation results

comapre dreambooth and textual inversion

Figure: Figure 2:
Refer to caption: figure/singlePRRgenerate5Model.pdf

## Appendix G Image Editing Mathematical Discussion

The stability of PRR outputs is reflected in that different Gaussian noises tend to yield the same result, represented as:

$$ $\displaystyle h=Wx=W(x+\Delta x)$ (8) $$

the equation is true when $W\Delta x=\mathbf{0}$, which is denoted by $\Delta x\in kernel(W)$.

According to the rank-nullity theorem, for the linear transformation $W:X\to H$, ${rank(W)+nullity(W)=\dim X}$. (The nullity of $W$ is the dimension of $W$ kernel).) In PRR, our reduced rank $r=rank(W_{0})-rank(W_{reduce})=nullity(W_{reduce})-nullity(W_{0})$, we have $rank(W_{0})>rank(W_{reduce})$ which implies $nullity(W_{reduce})>nullity(W_{0})$, more of $\Delta x$ in PRR will not produce different outputs.
This reduced rank $r$ demonstrates the problem [meng2021sdedit] of the trade-off between faithful reconstruction and editability in image editing, As $r$ increases, the modifiable features decrease, making the reconstruction more faithful. As $r$ decreases, the modifiable features increase, bringing the diversity of the generated images closer to that of the underlying pre-trained generation model, and improving the editability. When a large $r$ is selected for training the PRR on a single image, the model generates images that closely resemble the training image, even when using various text prompts. This enables direct modification of the text prompt to facilitate image editing on the single train image.

## Appendix H Survey

Optionally include supplemental material
(complete proofs, additional experiments and plots) in appendix.
All such materials SHOULD be included in the main submission.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: 1. This paper is well-structured and easy to understand.
2. The idea of achieving T2I model personalization by controlling the rank of diffusion model parameters is highly innovative. Additionally, detailed explanations are provided for the introduced learnable low-rank parameters.
3. The experimental section is well-designed with sufficient data, demonstrating the effectiveness of PaRa in single/multi-subject generation and single-image editing, as well as its compatibility with other modules.
summary: This paper proposes a framework for T2I model personalization called PaRa (Parameter Rank Reduction). It introduces an innovative approach to control the rank of diffusion model parameters, thereby constraining the initially diverse generation space to a smaller, more balanced target space. This enables the generation of personalized concepts and single-image editing. Furthermore, multiple individually fine-tuned PaRa modules can be combined to achieve the fusion of multiple personalized concepts. The framework also demonstrates higher parameter efficiency and better alignment with target images, with experimental results validating its effectiveness.
weaknesses: 1. The methodology section includes extensive explanations of the mathematical principles behind PaRa but lacks an organized overview of the model’s framework. From the subsequent experimental section, it is evident that the approach also utilizes text embeddings, among other elements. While these are not the main focus of the methodology, they should be appropriately explained. Additionally, adding some visualizations in the methodology section would make the concepts more intuitive.
2. In the last part of the introduction (lines 110–111), it states that "PaRa achieves state-of-the-art performance in personalized single/multi-subject generation." However, the experiments do not provide sufficient comparisons to support this claim. Firstly, there is no comparison with encoder-based personalization methods mentioned in the introduction. Secondly, the fine-tuning-based personalization methods used in the experiments are not the latest approaches in this field, which weakens the persuasiveness of the results.
3. In section 3.1, it is mentioned that “B is initialized to zero and fine-tuned with a few text-image pairs.” The experimental section should clarify the scale of the data used, the fine-tuning time, and other relevant information. Additionally, to support the claim of higher parameter efficiency, a computational efficiency analysis should be provided.
4. From the image editing results, it appears that introducing PaRa may result in some loss of texture information. The analysis and interpretation of the experimental results are insufficient.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. One important motivation of the paper is that taming a T2I model toward a novel concept implies a small generation space, making it possible to perform one-shot training for the low-rank parameter B. I'm curious about the relationship between edibility and the optimal rank of B. Is it always possible to learn a good low-rank B to perform reasonable editing in one-shot manner? It's just an open question. Please share your experiences.
3. As a one-shot training techniques for image editing, how the model perform with different one-shot pairs? Please explain robustness of the model.
4. Both numbers and visualisation are good in explaining the superiority of Para compared with existing techniques. Then, what are the limitations of Para? Please provide a rough failure case analysis.
soundness: 3
strengths: 1. Solving personalisation from matrix decomposition is a reasonable and good idea.
2. Combing rank with edibility, especially diversity of the generated image is proven effective in this paper.
3. The experiments are convincing in explaining the superiority of the proposed solution.
summary: A diffusion based image personalization method is presented via a parameter rank reduction technique. Particularly, the proposed method controls the rank of the diffusion model parameters to restrict the generation space into a small and well-balanced target space, achieving trade-off between the training data distribution and the target distribution. The motivation of the method lies in two main aspects. Firstly, this paper assumes that taming a T2I model toward a novel concept implies a small generation space. Secondly, the rank of matrix is important for parameter efficient fine-tuning. In this case, by reducing the rank of model parameters during finetuning, the proposed method is proven to achieve advantages over existing finetuning approaches on single/multi-subject generation and single-image editing.
weaknesses: 1. Matrix decomposition is proven to be effective in parameter efficient fine-tuning tasks, e.g. image editing. Although the difference between the proposed solution to Lora is clear. As both methods are based on matrix decomposition, it's not clear what are the foundational differences, and how the authors come up with the current solution.
2. SSIM is used in this paper for image diversity evaluation. I'm not quite sure whether SSIM is the best one, as SSIM focuses on pixel-level difference instead of semantic level difference. e.g. position difference of the same instance may contribute more to SSIM than than an extra accessory. 
3. Writing should be improved to highlight the contributions, especially how the solution is formed, and how it is different from the existing techniques.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: The only concern is insufficient comparisons to SOTA, adding which would allow a better comparison.
soundness: 3
strengths: 1. The proposed Parameter Rank Reduction (PaRa) method is a creative solution to the challenge of T2I model personalization, offering a new perspective on controlling the generation space.
2. PaRa demonstrates significant parameter efficiency, requiring 2× fewer learnable parameters compared to existing methods like LoRA.
3. The paper provides extensive experimental results showing PaRa’s advantages in single/multi-subject generation and single-image editing and shows better results than LoRA.
4. The paper is very well written and presented.
summary: This work aims to balance personalization and text editability by reducing the rank of model parameters, effectively narrowing the generation space to better align with target concepts. This method is more parameter-efficient and achieves better target image alignment than existing techniques like LoRA. It also supports combining multiple personalized models and facilitates stable single-image editing without additional noise inversion processes. The paper shows PaRa’s effectiveness through comprehensive experiments on single and multi-subject generation tasks.
weaknesses: 1. The authors have not compared their method with SOTA such as LyCoris, DiffuseKronA, etc. Including these results would provide a better comparison of the method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please refer to the concerns in the Weakness.
soundness: 3
strengths: 1) This paper provides a very interesting ideas to personalize diffusion model by reducing the rank of the weights. 
2) The proposed finetuning method is simple, effective and show good flexibility.
summary: This paper proposes a new method to personalizing diffusion model with few samples, which is to reduce the rank of the original weights in diffusion models. This method allows the finetuned model to have smaller diversity and better alignment with the training data. This paper conducts extensive experiments on various generation task including single subject, multi-subject generation and editing. The paper is well written and easy to understand.
weaknesses: 1) Different hyper-parameters may affect the alignment of the previous methods, such as the Rank of LoRA and fine-tuning steps. Authors may need to provide more clarification on how these hyper-parameters are chosen for the compared methods to make sure they achieve good enough alignment in the experiments.

Meta-review: All reviewers consider the proposed approach to text-to-image (T2I) model personalization using learnable low-rank parameters (PaRa) as  innovative. They find the idea of controlling the rank of diffusion model parameters for personalization novel and well-explained. The experimental evaluation is deemed comprehensive, showcasing PaRa's effectiveness in single/multi-subject generation, single-image editing, and compatibility with other modules. The novel approach, strong empirical results, and detailed responses to reviewer concerns during the discussion phase make it deserved to be accepted.

Decision: Accept (Spotlight)