Exploring Diffusion Models' Corruption Stage in Few-Shot Fine-tuning and Mitigating with Bayesian Neural Networks

## Abstract

Abstract Few-shot fine-tuning of Diffusion Models (DMs) is a key advancement, significantly reducing training costs and enabling personalized AI applications. However,
we explore the training dynamics of DMs and observe an unanticipated phenomenon: during the training process, image fidelity initially improves, then unexpectedly deteriorates with the emergence of noisy patterns, only to recover later with severe overfitting.
We term the stage with generated noisy patterns as corruption stage .
To understand this corruption stage, we begin by theoretically modeling the one-shot fine-tuning scenario, and then extend this modeling to more general cases.
Through this modeling, we identify the primary cause of this corruption stage: a narrowed learning distribution inherent in the nature of few-shot fine-tuning.
To tackle this, we apply Bayesian Neural Networks (BNNs) on DMs with variational inference to implicitly broaden the learned distribution,
and present that the learning target of the BNNs can be naturally regarded as an expectation of the diffusion loss and a further regularization with the pretrained DMs.
This approach is highly compatible with current few-shot fine-tuning methods in DMs and does not introduce any extra inference costs.
Experimental results demonstrate that our method significantly mitigates corruption, and improves the fidelity, quality and diversity of the generated images in both object-driven and subject-driven generation tasks.

## 1 Introduction

Recent years have witnessed a surge in the development of Diffusion Models (DMs). These models have showcased extraordinary capabilities in various applications, such as image editing [] and video editing [], among others. Particularly noteworthy is the advent of few-shot fine-tuning methods [, , ], in which a pretrained model is fine-tuned to personalize generation based on a small set of training images. These approaches have significantly reduced both memory and time costs in training. Moreover, these techniques offer powerful tools for adaptively generating images based on specific subjects or objects, embodying personalized AI and making AI accessible to everyone. In recent years, this innovation has even fostered the emergence of several communities, such as Civitai [], which boasts tens of thousands of checkpoints and millions of downloads.

Despite the importance and widespread usage of few-shot fine-tuning methods in DMs, these methods often struggle or even fail when transferring from a large distribution (i.e., pretrained DMs’ learned distribution) to a much smaller one (i.e., fine-tuned DMs’ learned distribution) using limited data [, ].
We are the first to identify when and how these failures occur, and find that they are related to an unusual phenomenon:
As shown in Fig. 1, the similarity between the generated images and the training images initially increases during fine-tuning, but then unexpectedly decreases, before increasing once more.
Ultimately, the DMs are only capable to generate images identical to the training images.
Notably, on the stage with decreasing similarity, we observe there appear some unexpected noisy patterns on the generated images, therefore we name it as corruption stage.

To understand this corruption stage, we carry out further theoretical analysis on the few-shot fine-tuning process.
Specifically, we start from modeling a one-shot case, i.e.,
using only one image during fine-tuning, and then extend it to more general cases.
The modeling provides an estimation of the error scale remaining in the generated images.
We further show how corruption stages emerge due to
the limited learned distribution inherent in few-shot tasks with this modeling.

Based on the analysis above,
the solution to corruption should concentrate on expanding the learned distribution.
However, this expansion remains challenging in few-shot fine-tuning.
Traditional data augmentation methods, when applied to generative models, often face significant problems such as leakage [] and a reduction in generation quality []. Inspired by recent advancements in Bayesian Neural Networks (BNNs) [], we propose to incorporate BNNs as a straightforward yet potent strategy to implicitly broaden the learned distribution.
We further present that its learning target can be regarded as an expectation of the diffusion loss and an extra regularization loss related to the pretrained model.
These two losses can be adjusted to reach a trade-off between image fidelity and diversity.
Our method does not introduce any extra inference costs, and has good compatibility with existing few-shot fine-tuning methods in DMs, including DreamBooth [], LoRA [], and OFT [].
Experimental results demonstrate that our method significantly alleviates the corruption issues and substantially enhances the performance across various few-shot fine-tuning methods on diverse datasets under different metrics.

Figure: (a) Few-shot fine-tuning process without BNNs.
Refer to caption: x1.png

In summary, our main contributions are as follows:

1. We observe an abnormal phenomenon during few-shot fine-tuning process on DMs: The image fidelity first enhances, then unexpectedly worsens with the appearance of noisy patterns, before improving again. We refer to the phase where noisy patterns appear as the corruption stage. We hope this observation could inform future research on DMs.

2. We provide a theoretical modeling for few-shot fine-tuning process on DMs, explaining the emergence and disappearance of the corruption stage. With this modeling, we pinpoint that the main issue stems from the constrained learned distribution of DMs inherent in few-shot fine-tuning process.

3. We innovatively incorporate BNNs to broaden the learned distribution, hence relieving such corruption. Experimental results confirm the effectiveness of this approach in improving on different metrics, including text prompt fidelity, image fidelity, generation diversity, and image quality.

## 2 Related Works

### 2.1 Diffusion Models and Few-shot Fine-tuning

Diffusion Models (DMs) [, , , ] are generative models that approximate a data distribution through the gradual denoising of a variable initially sampled from a Gaussian distribution. These models involve a forward diffusion process and a backward denoising process. In the forward process,
the extent of the addition of a noise $\varepsilon$ increases over time $t$, as described by the equation $x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\varepsilon$, where $x_{0}$ is a given original image and the range of time $t$ is $\left\{1,\dots,1000\right\}$ in general cases.
Conversely, in the backward process, the DMs aim to estimate the noise with a noise-prediction module $\epsilon_{\theta}$ and subsequently remove it from the noisy image $x_{t}$.
The discrepancy between the actual and predicted noise serves as the training loss, denoted as diffusion loss
$\mathcal{L}_{DM}:=\mathbb{E}_{\varepsilon\sim\mathcal{N}(0,1),t}\left||
\epsilon_{\theta}(x_{t},t)-\varepsilon|\right|_{2}^{2}.$

Few-shot fine-tuning in DMs [, , , ] aims at personalizing DMs with a limited set of images, facilitating the creation of tailored content. Gal *et. al.* [] introduced a technique that leverages new tokens within the embedding space of a frozen text-to-image model to capture the concepts presented in provided images, though this method has shown limitations in reproducing precise details of the input images [].
Ruiz *et. al.* proposed DreamBooth [], which fine-tunes most parameters in DMs leveraging a reconstruction loss that captures details, and a class-specific preservation loss that ensures the alignment with the textual prompts.
Additionally, Hu *et. al.* proposed LoRA, a lightweight fine-tuning approach that inserts low-rank layers to be learned while keeping other parameters frozen [].
Qiu *et. al.* presented OFT, a method that employs orthogonal transformations to enhance the quality of generation []. Although these methods generally succeed in capturing the details of training images, they suffer from the corruption stage observed in this paper.

### 2.2 Bayesian Neural Networks

Bayesian Neural Networks (BNNs) are a type of stochastic neural networks characterized by treating the parameters as random variables rather than fixed values [, , ].
The objective is to infer the posterior distribution $P(\theta|\mathcal{D})$ for the parameters $\theta$ given a dataset $\mathcal{D}$.
This approach endows BNNs with several distinct advantages, such as the capability to model the distributions for output, to mitigate overfitting, and to enhance model interpretability [, ].
One prevalent variant of BNNs is the mean-field variational BNN, also known as Bayes by Backprop, where the mean-field variational inference is applied to obtain the variational distribution $Q_{W}(\theta)$ to approximate the posterior distribution $P(\theta|\mathcal{D})$ [].
Recent studies have demonstrated that even a BNN module, which treats only a subset of parameters as random variables while maintaining the rest as fixed, can retain the benefits associated with full BNNs [, , ].
Our proposed method can be regarded as a natural progression of BNN principles applied to few-shot fine-tuning in DMs.

## 3 Corruption Stage in Few-shot Fine-tuning

### 3.1 Observation

Figure: (a) Fine-tuning on 1 image.
Refer to caption: x3.png

In this section, we explore the performance variation during the few-shot fine-tuning process of DMs.
Concretely, we fine-tune Stable Diffusion (SD) v1.5(^1^11https://huggingface.co/runwayml/stable-diffusion-v1-5) [] with DreamBooth [] on different number of training images, and record the average Dino similarity between the generated images and training images as a measure of the image fidelity [, ].

As shown in Fig. 2, the variation of the image fidelity is not monotonous during fine-tuning.
Concretely, the few-shot fine-tuning process can be approximately divided into the following phases:

- 1.
In the first a few iterations, the image fidelity improves quickly.
- 2.
Later, there is an abnormal decrease in the image fidelity.
We observe the generated images in this phase present with increasing noisy patterns, i.e., the gradual emergence of the corruption stage.
- 3.
Subsequently,
the generation fidelity recovers, and we observe the corruption patterns on generated images progressively diminish, i.e. the gradual disappearance of
the corruption stage.
Once corruption completely diminishes in this phase, the model enters a state close to overfitting, where it can only generate images identical to the training images.
As a result, it loses the ability to produce diverse images.

Fig. 2 also shows when the number of training images increases, the onset of corruption is delayed and its severity is lessened.

### 3.2 Theoretical Modeling on Few-shot Fine-tuning in DMs

In this section, we begin by modeling one-shot fine-tuning scenarios and then extend it to more general cases. This modeling is supported by an evidence on a specific case.

Modeling for One-shot Fine-tuning on DMs. We start with a representative condition where the dataset $\mathcal{D}$ contains only one training image $x^{\prime}$.
Under this condition, we suppose the fine-tuned DMs with parameter $\theta$ model the joint distribution of any original image $x_{0}$ and any noisy image at time $t$, i.e., $x_{t}$, as a multivariate Gaussian distribution $P_{\theta}(x_{0},x_{t})$.
Concretely, its marginal distribution of $x_{0}$ is assumed as $P_{\theta}(x_{0})=\mathcal{N}(x^{\prime},\sigma_{1}^{2})$ when the model is fine-tuned with only one image $x^{\prime}$.
Additionally, as the noisy image $x_{t}$ is obtained by a linear combination $x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon$ between $x_{0}$ and a unit Gaussian noise $\epsilon$, the marginal distribution of $x_{t}$ should be $P_{\theta}(x_{t})=\mathcal{N}(\sqrt{\alpha_{t}}x^{\prime},\alpha_{t}\sigma_{1}
^{2}+(1-\alpha_{t}))$.
Notably, the fine-tuning process in fact narrows the KL divergence between $P_{\theta}(x_{t}\mid x_{0}=x^{\prime})$ and $\mathcal{N}(\sqrt{\alpha_{t}}x^{\prime},(1-\alpha_{t}))$, thus these distributions should be increasingly close during fine-tuning [, ].

With this modeling, the main focus of the DMs, predicting the original image $x_{0}$ based on a noisy image $x_{t}$, is represented as $P_{\theta}(x_{0}|x_{t})$.
We find that it in fact approximates a Gaussian distribution related to both $x_{t}$ and the training image $x^{\prime}$:

$$ $\displaystyle\begin{aligned} P_{\theta}(x_{0}|x_{t})\approx\mathcal{N}(x^{ \prime}+\delta_{t}(x_{t},x^{\prime}),\frac{(1-\alpha_{t})}{\alpha_{t}\sigma_{1 }^{2}+(1-\alpha_{t})}),\\ \text{ where }\delta_{t}(x_{t},x^{\prime})=\frac{\sqrt{\alpha_{t}}\sigma_{1}^{ 2}}{\alpha_{t}\sigma_{1}^{2}+(1-\alpha_{t})}(x_{t}-\sqrt{\alpha_{t}}x^{\prime} ).\end{aligned}$ (1) $$

The most possible $x_{0}$, denoted as $\hat{x}_{0}$ in the view of the DMs is:

$$ $\displaystyle\hat{x}_{0}=\arg\max_{x_{0}}P_{\theta}(x_{0}|x_{t})\approx x^{ \prime}+\delta_{t}(x_{t},x^{\prime}).$ (2) $$

The derivation details are provided at the Appendix Sec. A.
Notably, the error term $\delta_{t}(x_{t},x^{\prime})$ represents the difference between the predicted original image $\hat{x}_{0}$ and the training image $x^{\prime}$.

To illustrate an extreme case in this modeling, we consider a scenario where $\sigma_{1}=0$, i.e., $\delta_{t}=0$.
Under this condition, for any input $x_{t}$, the DMs consistently reproduce the training image $x^{\prime}$ as described by Eq. (2).
This indicates that the DMs fully lose its intrinsic denoising ability in this extreme scenario, only regenerating the training image instead.

In the opposite extreme, where $\sigma_{1}=+\infty$, i.e., $\delta_{t}=\frac{1}{\sqrt{\alpha_{t}}}x_{t}-x^{\prime}$, the model’s prediction for $x_{0}$ is exactly $\frac{1}{\sqrt{\alpha_{t}}}x_{t}$. This indicates that the DMs entirely lose its ability to generate images. Instead, it only rescales $x_{t}$ based on the factor $\alpha_{t}$ at the time step $t$, leading to any noise in $x_{t}$ is also remaining in the generated image.

Extension to More General Cases.
We further extend our modeling to the case where $\mathcal{D}$ contains multiple training samples.
In specific, we assume the learned distribution of the original image $x_{0}$ of the DMs, i.e., $P_{\theta}(x_{0})$, is centered with an image set $\mathcal{I}_{\theta}$.
Under few-shot fine-tuning, as the training continues, $\mathcal{I}_{\theta}$ gradually approximates the training dataset $\mathcal{D}$.
On the other hand, for the pretrained DMs, we assume they learns a sufficiently good distribution, i.e. with a sufficiently large $\mathcal{I}_{\theta}$.
For all these DMs facing with noisy image $x_{t}$, we simplify their behavior as firstly finding a sample $x^{*}\in\mathcal{I}_{\theta}$ to minimize the error term $\delta_{t}(x_{t},x^{*})$, and then estimating the corresponding $\hat{x}_{0}$ according to Eq. (2).

Figure: Figure 3: Denoised images from pretrained and fine-tuned DMs using $x_{t}=0$ and $t=1000$. The pretrained DMs do not largely change $x_{t}$ as it is free of noise. In contrast, both DMs fine-tuned on 1 and 5 images transform $x_{t}$ to make it closely resembles one sample within the training dataset $\mathcal{D}$.
Refer to caption: x6.png

Notably, the pretrained DMs have an $\mathcal{I}_{\theta}$ large enough, hence
is capable of finding a $x^{*}$ s.t the error term $\delta_{t}(x_{t},x^{*})$ approximates 0, so the corruption generally does not happen in pretrained DMs.

Support for the Modeling.
To support the above modeling closely approximates the practical scenarios, we present a specific example where we set the “noisy” image $x_{t}=0$, and then make both the pretrained and fine-tuned DMs denoise this image $x_{t}$ which is entirely free of noise.
According to our modeling, both DMs should first find one sample $x^{*}$ within their own $\mathcal{I}_{\theta}$.
Naturally, $x_{t}=0$ is within $\mathcal{I}_{\theta}$ of the pretrained DMs (See Appendix Sec. D for more evidence.). Therefore, the pretrained DMs should leave this $x_{t}=0$ almost unchanged during denoising.

Figure: Figure 4: Estimated $\sigma_{1}$ under different training iterations for DreamBooth.
Refer to caption: x7.png

In comparison, the $\mathcal{I}_{\theta}$ of the fine-tuned DMs should gradually approximate the training dataset $\mathcal{D}$. Therefore, the fine-tuned DMs should first find one sample $x^{*}\in\mathcal{D}$, and then
predict the original image $x_{0}$ as proportional to $x^{*}$ according to Eq. (2).

Experimental results provided in Fig. 3 support our analysis, where we can observe the pretrained DMs do not largely change this $x_{t}=0$, but the fine-tuned DMs transform this $x_{t}$ to one of the samples in its training dataset $\mathcal{D}$, which indicates its $\mathcal{I}_{\theta}\approx\mathcal{D}$.

### 3.3 Explanation of the Corruption Stage.

In this section, we explain the corruption stage according to our modeling about few-shot fine-tuning process on DMs. We first present the reason for the emergence of the corruption stage, with an example showing the severity of such problem. We further demonstrate why the corruption stage gradually disappears, resulting in “overfitting” as the fine-tuning process continues.

Figure: Figure 5: Experimental results for pretrained and fine-tuned DMs with an additional noise $\delta^{\prime}$ introduced in a small region of the noisy image $x_{t}$ at $t=100$. The pretrained DMs effectively remove $\delta^{\prime}$, producing high-quality images. Conversely, the fine-tuned DMs fail to eliminate $\delta^{\prime}$, with output images showing corruption patterns.
Refer to caption: x8.png

Emergence of the Corruption Stage. As stated in Eq. (2), given any noisy image $x_{t}$, the fine-tuned DMs would predict the original image $\hat{x}_{0}=x^{*}+\delta_{t}$, where $x^{*}\in\mathcal{I}_{\theta}\approx\mathcal{D}$ after certain training iterations. The scale of the error term $\delta_{t}$ is related to $\left\|x_{t}-\sqrt{\alpha_{t}}x^{*}\right\|_{2}$ and $\sigma_{1}$. We estimate $\sigma_{1}$ based on $x_{t}$ sampled from $\mathcal{N}(0,1-\alpha_{t})$ and present the average $\sigma_{1,t}$ among different $t$s in Fig. 4. It shows that the $\sigma_{1}$ remains relatively high under moderate iterations, resulting in a significant $\delta_{t}$ once $x_{t}$ is not identical to $\sqrt{\alpha_{t}}x^{*}$.

Concretely, for the case with only one training image, i.e., $\mathcal{D}=\left\{x^{\prime}\right\}$, we set the noisy image $x_{t}=\sqrt{\alpha_{t}}x^{\prime}+\sqrt{1-\alpha_{t}}\varepsilon$ + $\delta^{\prime}$, where $\varepsilon\in\mathcal{N}(0,1)$ and $\delta^{\prime}$ is an additional noise introduced to a small region of the image. This $\delta^{\prime}$ simulates the case where the generating process of DMs is inaccurate in some $t$s. We further set the time variable $t=100$, and fine-tune DMs with 1000 iterations, with the estimated $\sigma_{1}\approx 4.8$ as shown in Fig. 4.
According to Eq. (2) and the analysis above, we can compute $\left\|\delta_{100}\right\|_{2}^{2}\approx 2.65\left\|\delta^{\prime}\right\|_
{2}^{2}$. It means the additional noise $\delta^{\prime}$ introduced is even expanded in this case, leading to a significant error term $\delta_{t}$. Fig. 5 shows the experimental results under this setting, where we observe an obvious error term $\delta_{t}$, resembling a corruption pattern.

Vanishing of the Corruption Stage. However, with the fine-tuning continues, $\sigma_{1}$ drops as shown in Fig. 4, leading to a decreasing prediction error $\delta_{t}$. This indicates that the corruption stage vanishes, and the fine-tuned DMs gradually move to the state where they only strictly regenerate the training image $x^{\prime}\in\mathcal{D}$.

In conclusion, the analysis in this Sec. 3.3 shows how the corruption stage happens when the learned distribution of DMs is highly limited with small $\mathcal{I}_{\theta}$ and a high standard deviation $\sigma_{1}$.

## 4 Applying BNNs to Few-shot Fine-tuning on DMs

### 4.1 Motivation

Based on our analysis, the corruption stage primarily arises from a limited learned distribution with a small $\mathcal{I}_{\theta}$.
Motivated by recent work on BNNs [, , , ], which model the parameters $\theta$ as random variables,
we propose to apply BNNs in the few-shot fine-tuning process on DMs as a simple yet effective method to expand $\mathcal{I}_{\theta}$.
Intuitively, the modeling of BNNs hinders the DMs to learn the exact distribution of the training dataset $\mathcal{D}$.
Therefore, the DMs are encouraged to learn a larger and more robust distribution to counter the randomness.
Moreover, the sampling randomness during the fine-tuning process can be regarded as an inherent data augmentation, which indicates the corresponding $\mathcal{I}_{\theta}$ is implicitly expanded.

### 4.2 Formulation

Modeling.
BNNs model the parameters $\theta$ as random variables.
Therefore, the learned distribution of DMs with BNN is $P(x|\mathcal{D})=\int P(x|\theta)P(\theta|\mathcal{D})d\theta$.
Concretely, $P(x|\theta)$ is the image distribution modeled by the DMs, and $P(\theta|\mathcal{D})$ is the posterior parameter distribution with a given dataset $\mathcal{D}$.

As the posterior distribution $P(\theta|\mathcal{D})$ is intractable, a variational distribution $Q_{W}(\theta)$ is applied to approximate it.
We model the variational distribution of each parameter $\theta$ as a Gaussian distribution: $\theta\sim\mathcal{N}(\mu_{\theta},\sigma_{\theta}^{2})$, where $W=\left\{\mu_{\theta},\sigma_{\theta}\right\}$ are trainable parameters.
Considering the fine-tuning process, we initialize the expectation term $\mu_{\theta}$ from the corresponding parameter of the pretrained DMs, denoted as $\theta_{0}$. Following previous work [], we apply the re-parameterization trick to obtain the gradients of the parameters, as detailed in Appendix Sec. B.

Training. During the fine-tuning process, the DMs are trained by minimizing the Kullback–Leibler (KL) divergence $\mathrm{KL}(Q_{W}(\theta)||P(\theta|\mathcal{D}))$,
which is equivalent to minimizing

$$ $\displaystyle\mathcal{L}=-\int Q_{W}(\theta)\log\frac{P(\theta,\mathcal{D})}{Q _{W}(\theta)}d\theta$ $\displaystyle=\mathbb{E}_{\theta\sim Q_{W}(\theta)}\underbrace{-\log P( \mathcal{D}|\theta)}_{\mathcal{L}_{DM}}+\underbrace{\mathrm{KL}(P(\theta)||Q_{ W}(\theta))}_{\mathcal{L}_{r}}.$ (3) $$

Following previous work [], the above loss $\mathcal{L}$ can be divided into two terms.
In DMs, the first term can be seen as the modeled probability for the training dataset $\mathcal{D}$,
and is equivalent to the expectation of the diffusion loss $\mathcal{L}_{DM}$ shown in Sec. 2.1 on the parameters $\theta$.
The second term can be seen as a regularization restricting the discrepancy between the variational distribution $Q_{W}(\theta)$ and the prior distribution $P(\theta)$.
We name it as the regularization loss $\mathcal{L}_{r}$.
In few-shot fine-tuning, we regard the pretrained DMs naturally represent the prior information, so we set the prior distribution $P(\theta)$ from the pretrained DMs, i.e., $P(\theta)=\mathcal{N}(\theta_{0},\sigma^{2})$,
where $\sigma$ is a hyperparameter which represents the parameter randomness.

In practice, we formulate our learning target as a linear combination of $\mathcal{L}_{DM}$ and $\mathcal{L}_{r}$ with a hyperparameter $\lambda$, i.e.,

$$ $\displaystyle W^{*}=\arg\min\limits_{W}\mathbb{E}_{\theta\sim Q_{W}(\theta)} \mathcal{L}_{DM}+\lambda\mathcal{L}_{r}.$ (4) $$

The training process is summarized in Appendix Alg. 1.
Empirically, we find that using only $\mathbb{E}_{\theta\sim Q_{W}(\theta)}\mathcal{L}_{DM}$, i.e., setting $\lambda$ as 0 0, is enough to improve few-shot fine-tuning.
Nevertheless, we can reach a further trade-off between the generation diversity and image fidelity by adjusting $\lambda$.

Inference.
During the inference, we explicitly replace each parameter $\theta$ with its mean value $\mu_{\theta}$ and perform inference just as DMs without BNNs.
It guarantees that we do not introduce any additional costs compared to fine-tuned DMs without BNNs when deployed in production.

Motivated by previous work on BNN modules [, , ], we only model a subset of parameters as random variables in practice, which reduces the computational costs.
Fine-tuning DMs with BNNs is compatible with existing few-shot fine-tuning methods, including DreamBooth [], LoRA [], and OFT [].
More details are presented in Appendix Sec. C.

## 5 Experiments

We apply BNNs to different few-shot fine-tuning methods across different tasks.
For object-driven generation, where the fine-tuned DMs synthesize images with the details of given objects, we use all the 30 classes from DreamBooth [] dataset, each containing 4-6 images.
For subject-driven generation, where the fine-tuned DMs synthesize images with the identities of given people, we follow previous research [], randomly selecting 30 classes of images from CelebA-HQ [], each containing 5 images.
Most training settings follow previous work [, , , ]. All experiments are conducted with 5 different seeds by default and we report the average performance here.
We use Stable Diffusion v1.5(^2^22https://huggingface.co/runwayml/stable-diffusion-v1-5) (SD v1.5) as
the default model for fine-tuning.
As for the BNNs, we set the default initialized standard derivation $\sigma_{\theta}$ and prior standard derivation $\sigma$ as 0.01.
The $\lambda$ is set as 0 by default.
We show more details in Appendix Sec.  F.1.

Following previous work [, , ], for each class, we fine-tune a DM and generate 100 images with various prompts.
These generated images are used to measure the performance of different few-shot fine-tuning methods.
In specific, we use Clip-T [] to measure the text prompt fidelity, Clip-I [] and Dino [] to assess image fidelity, and Lpips [] to evaluate generation diversity. Additionally, we apply Clip-IQA [] to measure no-reference image quality.
We show more details about these metrics in Appendix Sec. E.

### 5.1 Comparisons

**Table 1: Performance of fine-tuning with BNNs under object-driven and subject-driven generation.**
| Object-Driven Generation: DreamBooth Dataset | Subject-Driven Generation: CelebA Dataset |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Clip-T$\uparrow$ | Dino$\uparrow$ | Clip-I$\uparrow$ | Lpips$\uparrow$ | Clip-IQA$\uparrow$ | Method | Clip-T$\uparrow$ | Dino$\uparrow$ | Clip-I$\uparrow$ | Lpips$\uparrow$ | Clip-IQA$\uparrow$ |
| DreamBooth | 0.246 | 0.614 | 0.771 | 0.611 | 0.875 | DreamBooth | 0.186 | 0.642 | 0.723 | 0.511 | 0.789 |
| DreamBooth w/ BNNs | 0.256 | 0.633 | 0.785 | 0.640 | 0.893 | DreamBooth w/ BNNs | 0.205 | 0.696 | 0.757 | 0.515 | 0.811 |
| LoRA | 0.252 | 0.542 | 0.722 | 0.650 | 0.864 | LoRA | 0.216 | 0.602 | 0.656 | 0.644 | 0.804 |
| LoRA w/ BNNs | 0.261 | 0.618 | 0.769 | 0.678 | 0.890 | LoRA w/ BNNs | 0.227 | 0.604 | 0.663 | 0.688 | 0.824 |
| OFT | 0.233 | 0.649 | 0.786 | 0.629 | 0.861 | OFT | 0.164 | 0.675 | 0.728 | 0.549 | 0.784 |
| OFT w/ BNNs | 0.242 | 0.661 | 0.791 | 0.646 | 0.884 | OFT w/ BNNs | 0.185 | 0.696 | 0.743 | 0.570 | 0.798 |

Figure: Figure 6: Comparison of few-shot fine-tuning methods with and without BNNs across subject-driven and object-driven scenarios. We show both best-case and average-case generated images measured by Clip-I, Dino and Clip-IQA. See Appendix Sec. G and K for more details.
Refer to caption: x9.png

We apply BNNs on different few-shot fine-tuning methods under both object-driven and subject-driven generation tasks.
As shown in Tab. 1 and Fig. 6, BNNs bring considerable improvements on all few-shot fine-tuning methods across text prompt fidelity (Clip-T) and image fidelity (Dino and Clip-I).
These improvements arise from the expanded learned distribution contributed by BNNs, which makes the DMs more capable to generate reasonable images about the learned objects/subjects based on different prompts. BNNs also largely enhance the no-reference image quality (Clip-IQA).
It is mainly because BNNs largely reduce the corruption phenomenon, which also partially improve the image fidelity (Dino and Clip-I) as the corrupted images are semantically distant to training images.
Additionally, we observe BNNs boost the generation diversity (Lpips).
We believe it naturally comes from the randomness introduced by BNNs.

To further confirm the improvements brought by BNNs, we conduct a series of user studies, detailed in Appendix Sec. H.
The results suggest a notable preference of few-shot fine-tuning methods with BNNs over the ones without BNNs.

### 5.2 Generalization

In this section, we further demonstrate BNNs can be applied to broader scenarios with notable performance improvement, including different DMs, varying training steps and a different number of training images.
By default, we experiment on DreamBooth with BNNs for fine-tuning.

**Table 2: Performance under different DMs. All DMs are fine-tuned with DreamBooth on DreamBooth dataset.**
|  |  | Clip-T$\uparrow$ | Dino$\uparrow$ | Clip-I$\uparrow$ | Lpips$\uparrow$ | Clip-IQA$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
|  | w/o BNNs | 0.246 | 0.614 | 0.771 | 0.611 | 0.875 |
| SD v1.5 | w/ BNNs | 0.256 | 0.633 | 0.785 | 0.640 | 0.893 |
|  | w/o BNNs | 0.248 | 0.594 | 0.762 | 0.618 | 0.872 |
| SD v1.4 | w/ BNNs | 0.249 | 0.620 | 0.777 | 0.656 | 0.895 |
|  | w/o BNNs | 0.240 | 0.563 | 0.739 | 0.604 | 0.875 |
| SD v2.0 | w/ BNNs | 0.248 | 0.610 | 0.764 | 0.649 | 0.925 |

Different DMs. Following previous work [], we experiment on different DMs. Concretely, besides default SD v1.5, we also experiment on SD v1.4 and SD v2.0 []. We provide training details in Appendix Sec. F.2. Tab. 2 indicates that applying BNNs on different DMs of SD consistently improves few-shot fine-tuning across multiple metrics.

Figure: (a) Dino
Refer to caption: x10.png

Training Iterations.
Fig. 7 shows that our method consistently improves the image quality (Clip-IQA). It also improves the image fidelity (Dino) when the training steps are larger than $100\times\rm{Num}$, where $\rm{Num}$ represents the number of images utilized during fine-tuning. With fewer iterations, the model suffers from underfitting. In this case, BNNs may make the underfitting problem further severe as BNNs encourage the model to learn a larger distribution. This results in the slightly decreased image fidelity (Dino) observed in $100\times\rm{Num}$.

Numbers of Training Images.
We also experiment under different
numbers of training images. Concretely, we use the CelebA-HQ [] dataset as it contains enough images per class.
We randomly select five classes from the CelebA-HQ dataset and conduct experiments with different numbers of training images. We fix the training iterations to $250\times\rm{Num}$.

Fig. 8 illustrates that our method consistently improves image fidelity and quality under different numbers of training images. Such improvement is more obvious with more training images.

Figure: (a) Dino
Refer to caption: x12.png

This is primarily because the $250\times\rm{Num}$ generally results in more severe corruption problems when the training image number $\rm{Num}$ is larger, in which case BNNs bring in much improvement by expanding the learning distribution and relieving corruptions.

### 5.3 Ablation Study

Figure: (a) Different Initialized $\sigma_{\theta}$
Refer to caption: x14.png

Scale of Initialized $\sigma_{\theta}$. The initialized standard deviation $\sigma_{\theta}$ determines the extent of randomness during fine-tuning. We experiment with applying BNNs under varying initialized $\sigma_{\theta}$. Experimental results in Fig. 9(a) indicate that both the image fidelity (Dino) and quality (Clip-IQA) improves with a moderate initialized $\sigma_{\theta}$. However, when the initialized $\sigma_{\theta}$ is too large, the DMs collapse and the performance rapidly decreases. This indicates the DMs are almost randomly updated because of the too large randomness introduced in this scenario.

Trade-off Between Diversity and Fidelity with Adjusted $\lambda$. Eq. (4) indicates $\lambda$ controls the trade-off between the generation diversity and image fidelity.
As illustrated in Fig. 9(b), an increasing $\lambda$ leads to improving generation diversity (Lpips), albeit at the expense of image fidelity (Dino).

Where to Apply BNNs.
As mentioned in Sec. 4.2, we could model only a subset of parameters as random variables, i.e., applying BNNs on a part of layers in DMs. By default, we apply BNNs to all linear layers in the U-Net [] except the ones in the cross-attention modules and explore how different choices influence the performance and the training costs.

As shown in Tab. 2, the DM can achieve relatively good performance with only the upblock of the U-Net applied with BNNs, which reduces the ratio of the parameters modified to about $13.8\%$. We can further decrease the training costs by only applying BNNs on the normalization layer, i.e., Layer Normalization (LN) [] and Group Normalization (GN) [] layers. This decreases the ratio of the parameters modified to about $0.02\%$ with relatively strong performance.

Additionally, when BNNs are applied to cross-attention modules, there is a significant increase in text prompt fidelity (Clip-T) at the expenses of image fidelity (Dino and Clip-I). Intuitively, this is because the input image aligns with only a limited set of prompts, and the applying of BNNs in cross-attention modules produces a further broader distribution matching more prompts.

**Table 3: Comparison of performance when BNNs are applied to different layers in DMs. All experiments are conducted on DreamBooth dataset fine-tuning with DreamBooth. ‘N.A.’ refers to no BNN applied. ‘CrossAttn’ refers to BNN applied to cross-attention modules. We report the GPU memory costs and average time costs during fine-tuning for each class on one A100 GPU.**
| Layer Type | CrossAttn | Up Block Only | Clip-T$\uparrow$ | Dino$\uparrow$ | Clip-I$\uparrow$ | Lpips$\uparrow$ | Clip-IQA$\uparrow$ | Memory Costs(MB) | Time Costs(s) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| N.A. |  |  | 0.246 | 0.614 | 0.771 | 0.611 | 0.875 | 27236 | 933 |
| Linear |  |  | 0.256 | 0.633 | 0.785 | 0.640 | 0.893 | 31912 | 1107 |
| Linear |  | ✓ | 0.259 | 0.614 | 0.776 | 0.646 | 0.890 | 29674 | 1040 |
| Linear | ✓ |  | 0.272 | 0.611 | 0.760 | 0.672 | 0.897 | 33258 | 1157 |
| LN+GN |  |  | 0.254 | 0.650 | 0.792 | 0.643 | 0.889 | 27268 | 1088 |

## 6 Conclusion

In this paper, we focus on few-shot fine-tuning in DMs, and reveal an unusual “corruption stage” where image fidelity first improves, then deteriorates due to noisy patterns, before recovering. With theoretical modeling, we attribute this phenomenon to the constrained learned distribution inherent in few-shot fine-tuning. By applying BNNs to broaden the learned distribution, we mitigate the corruption. Experiment results across various fine-tuning methods and datasets underscore the versatility of our approach.

## Appendix A Drivation of Eq. ( 2 )

We first restate our assumptions formally:

- •
The joint distribution of $x_{0}$ and $x_{t}$ is modeled by the DM as a multivariate Gaussian distribution
$\displaystyle P_{\theta}([x_{0},x_{t}]^{T})=\mathcal{N}(\boldsymbol{\mu},
\boldsymbol{\Sigma})=\mathcal{N}([x^{\prime},\sqrt{\alpha_{t}}x^{\prime}]^{T},
\left[\begin{array}[]{cc}\sigma_{1}^{2}&c\\
c&\alpha_{t}\sigma_{1}^{2}+(1-\alpha_{t})\end{array}\right]),$
(7)
where $c$ represents the unknown covariance between $x_{0}$ and $x_{t}$.
- •
The conditional probability of $x_{t}$ given $x_{0}=x^{\prime}$ is
$\displaystyle P_{\theta}(x_{t}\mid x_{0}=x^{\prime})=\mathcal{N}(\sqrt{\alpha_
{t}}x^{\prime},(1-\alpha_{t})).$
(8)

Denote the inverse matrix of $\boldsymbol{\Sigma}$ as

$$ $\displaystyle\boldsymbol{\Sigma}^{-1}=\left[\begin{array}[]{cc}\lambda_{11}& \lambda_{12}\\ \lambda_{21}&\lambda_{22}\end{array}\right]=\frac{1}{|\boldsymbol{\Sigma}|} \left[\begin{array}[]{cc}\alpha_{t}\sigma_{1}^{2}+1-\alpha_{t}&-c\\ -c&\sigma_{1}^{2}\end{array}\right],$ (13) $$

where $|\boldsymbol{\Sigma}|=\sigma_{1}^{2}(\alpha_{t}\sigma_{1}^{2}+(1-\alpha_{t}))-
c^{2}$ is the determinant of $\boldsymbol{\Sigma}$.
According to the property of joint Gaussian distribution, the conditional distribution $P(x_{t}|x_{0})$ can be represented as

$$ $\displaystyle P(x_{t}|x_{0})=\mathcal{N}(\frac{\lambda_{22}\sqrt{\alpha_{t}x^{ \prime}}+\lambda_{12}x_{0}-\lambda_{12}x^{\prime}}{\lambda_{22}},\frac{1}{ \lambda_{22}}).$ (14) $$

According to Eq. (8),

$$ $\displaystyle\frac{1}{\lambda_{22}}=(1-\alpha_{t}),$ (15) $$

which means $c=\pm\sqrt{\alpha_{t}}\sigma_{1}^{2}$. Hence we know the joint distribution is

$$ $\displaystyle P_{\theta}([x_{0},x_{t}]^{T})=\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})=\mathcal{N}([x^{\prime},\sqrt{\alpha_{t}}x^{\prime}]^{T}, \left[\begin{array}[]{cc}\sigma_{1}^{2}&\pm\sqrt{\alpha_{t}}\sigma_{1}^{2}\\ \pm\sqrt{\alpha_{t}}\sigma_{1}^{2}&\alpha_{t}\sigma_{1}^{2}+(1-\alpha_{t})\end {array}\right]).$ (18) $$

Repeatedly, according to the property of joint Gaussian distribution, we have

$$ $\displaystyle P(x_{0}|x_{t})=\mathcal{N}(x^{\prime}\pm\frac{\sqrt{\alpha_{t}} \sigma_{1}^{2}(y-\sqrt{\alpha_{t}}x^{\prime})}{\alpha_{t}\sigma_{1}^{2}+(1- \alpha_{t})},\frac{1-\alpha_{t}}{\alpha_{t}\sigma_{1}^{2}+1-\alpha_{t}}).$ (19) $$

In practice, the positive sign is more reasonable as it indicates the deviations of the predicted image $x_{0}$ and the input noisy image $x_{t}$ are aligned.
Therefore,

$$ $\displaystyle P(x_{0}|x_{t})=\mathcal{N}(x^{\prime}+\frac{\sqrt{\alpha_{t}} \sigma_{1}^{2}(y-\sqrt{\alpha_{t}}x^{\prime})}{\alpha_{t}\sigma_{1}^{2}+(1- \alpha_{t})},\frac{1-\alpha_{t}}{\alpha_{t}\sigma_{1}^{2}+1-\alpha_{t}}).$ (20) $$

## Appendix B Details of Applying BNNs

The training process of applying BNNs in fine-tuning is summarized in Alg. 1.
To obtain the gradients of the variational parameters, i.e., gradients of $W=\{\mu_{\theta},\sigma_{\theta}\}$, we apply the commonly used reparameterization trick in BNNs.
Concretely, we first sample a unit Gaussian variable $\varepsilon_{\theta}$ for each $\theta$, and then perform $\theta=\mu_{\theta}+\sigma_{\theta}\times\varepsilon_{\theta}$ to obtain a posterior sample of $\theta$.
Hence, the gradients can be calculated by

$$ $\displaystyle\frac{\partial}{\partial W}\mathcal{L}$ $\displaystyle=\frac{\partial}{\partial W}\left[\mathbb{E}_{Q_{W}(\theta)} \mathcal{L}_{DM}+\mathcal{L}_{r}\right]$ (21) $\displaystyle=\mathbb{E}_{\varepsilon_{\theta}\sim\mathcal{N}(0,I)}\left[\frac {\partial\mathcal{L}_{DM}}{\partial\theta}\frac{\partial\theta}{\partial W}+ \frac{\partial\mathcal{L}_{r}}{\partial W}\right].$ (22) $$

We refer to the Proposition 1 in previous work [] for the detailed derivation.

Figure: Algorithm 1 Fine-tuning DMs with BNNs

## Appendix C Applying BNNs on Different Few-shot Fine-tuning Methods

#### Applying BNNs on DreamBooth.

DreamBooth is a full-parameter fine-tuning method, and is one of the mainstream fine-tuning methods [].
Therefore, all parameters in DreamBooth can be modeled as the BNNs parameters.

#### Applying BNNs on LoRA.

LoRA [] is a classic lightweight yet effective method for few-shot fine-tuning.
In LoRA layers, the weight matrix $\mathbf{W}\in\mathbb{R}^{d\times k}$ is modeled as a sum of fixed weight from the pretrained model and a trainable low-rank decomposition: $\mathbf{W}=\mathbf{W}_{0}+\mathbf{B}\mathbf{A}$, where $\mathbf{W}_{0}\in\mathbb{R}^{d\times k},\mathbf{B}\in\mathbb{R}^{d\times r},
\mathbf{A}\in\mathbb{R}^{r\times k}$ with rank $r$ [].
In our implementation, we only convert the up matrix $\mathbf{A}$ into random variables,
and the down matrix $\mathbf{B}$ is still kept as usual trainable parameters to make $P(\mathbf{W})$ a Gaussian distribution.
This design also reduces additional computational costs during training while keeps its effectiveness.

#### Applying BNNs on OFT.

OFT is a few-shot fine-tuning method where the weights are tuned only by orthogonal transformations [].
In OFT layers, the weight matrix $\mathbf{W}\in\mathbb{R}^{d\times k}$ is modeled as $\mathbf{W}=\mathbf{R}\mathbf{W}_{0}$,
where $\mathbf{R}$ is guaranteed to be an orthogonal matrix by $\mathbf{R}=(\boldsymbol{I}+0.5(\mathbf{Q}-\mathbf{Q}^{T}))(I-0.5(\mathbf{Q}-
\mathbf{Q}^{T}))^{-1}$,
and $\mathbf{Q}$ is the trainable parameter in the original OFT method.
To guarantee the orthogonality is not destroyed by random sampling in BNNs, we only convert the trainable parameter $\mathbf{Q}$ into random variables.
Therefore, after the above transformation, $\mathbf{R}$ is kept as an orthogonal matrix, and $\mathbf{W}$ is kept as an orthogonalization transformation of the original pretrained $\mathbf{W}_{0}$.

**Table 4: Prompts used for evaluation. [V] indicates the special token and [object] indicates the type of the object.**
| Prompts for object-driven generation | Prompts for subject-driven generation |
| --- | --- |
| a [V] [object] in the jungle | a photo of a [V] person wearing sunglasses |
| a [V] [object] in the snow | a photo of a [V] person with snowflakes in their hair |
| a [V] [object] on the beach | a photo of a [V] person with beachy hair waves |
| a [V] [object] on a cobblestone street | a photo of a [V] person wearing a beret |
| a [V] [object] on top of pink fabric | a photo of a [V] person with a neutral expression |
| a [V] [object] on top of a wooden floor | a photo of a [V] person with a contemplative look |
| a [V] [object] with a city in the background | a photo of a [V] person laughing heartily |
| a [V] [object] with a mountain in the background | a photo of a [V] person with an amused smile |
| a [V] [object] with a blue house in the background | a photo of a [V] person with forest green eyeshadow |
| a [V] [object] on top of a purple rug in a forest | a photo of a [V] person wearing a red hat |
| a [V] [object] wearing a red hat | a photo of a [V] person with a slight grin |
| a [V] [object] wearing a santa hat | a photo of a [V] person with a thoughtful gaze |
| a [V] [object] wearing a rainbow scarf | a photo of a [V] person wearing a black top hat |
| a [V] [object] wearing a black top hat and a monocle | a photo of a [V] person in a chef hat |
| a [V] [object] in a chef outfit | a photo of a [V] person in a firefighter helmet |
| a [V] [object] in a firefighter outfit | a photo of a [V] person in a police cap |
| a [V] [object] in a police outfit | a photo of a [V] person wearing pink glasses |
| a [V] [object] wearing pink glasses | a photo of a [V] person wearing a yellow headband |
| a [V] [object] wearing a yellow shirt | a photo of a [V] person in a purple wizard hat |
| a [V] [object] in a purple wizard outfit | a photo of a [V] person smiling |
| a red [V] [object] | a photo of a [V] person frowning |
| a purple [V] [object] | a photo of a [V] person looking surprised |
| a shiny [V] [object] | a photo of a [V] person winking |
| a wet [V] [object] | a photo of a [V] person yawning |
| a cube shaped [V] [object] | a photo of a [V] person laughing |

## Appendix D Details of the Validation Experiments

Proof for $x_{t}=0$ is within the $\mathcal{I}_{\theta}$ of the pretrained DMs. We provide two proofs that $x_{t}=0$ is within the $\mathcal{I}_{\theta}$ of the pretrained DMs. The SD v1.5 is used as the pretrained DM.

- •
As shown in Fig. 10(a), we use the prompt “a simple, solid gray image with no textures or variations” to generate images, and we observe the pretrained DM is capable of generating such images free of noise.
- •
We use the Img2Img Pipeline(^3^33https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_ diffusion_img2img.py) provided by diffusers with no prompt provided, i.e., unconditional generation. Then we set the input image as a blank one and the img2img strength as 0.1, which means input noisy image $x_{100}=\alpha_{100}\varepsilon$, where $\varepsilon\in\mathcal{N}(0,1)$. As shown in Fig. 10(b). we can also observe the denoised result is completely free of noise.

Both results support our argument that $x_{t}=0$ is naturally within $\mathcal{I}_{\theta}$ of the pretrained DMs.

Figure: (a) Generated Images based on given prompt
Refer to caption: x16.png

Settings for Experiments in Sec. 3.3.
We fine-tune a SD v1.5 with DreamBooth without PPL loss []. The learning rate is fixed to $5\times 10^{-6}$ and only the U-Net [] is fine-tuned. We use the backpack class in DreamBooth dataset as an example, and use prompt "a [V] backpack" to train where the “[V]" is the special token. We set $x_{100}=0$ and show the one-step denoised result using the Img2Img Pipeline provided by diffusers. During denoising, for both pretrained and fine-tuned DMs, the prompt is fixed to “a [V] backpack". More results are shown in Fig. 11.

Figure: Figure 11: More results for generated images when input $x_{t}=0$ under different DMs and different inference steps.
Refer to caption: x18.png

## Appendix E Metrics

We use following metrics to robustly measure different aspects of the fine-tuned DMs:

Text prompt fidelity: Following previous papers [, ] , we use the average similarity between the clip [] embeddings of the text prompt and generated images, denoted as Clip-T.

Image fidelity: Following previous papers [, ], we compute the average similarity between the clip [] and dino [] embeddings of the generated images and training images, denoted as Clip-I [] and Dino.

Generation Diversity: We compute the average Lpips [] distance between the generated images of the fine-tuned DMs, following previous papers [, ].

Image quality: We find that corruption largely decreases the visual quality of the images, making them unusable. However, full-reference images quality measurement cannot fully represent this kind of degradation in image quality. Therefore, we propose to add no-visual quality metric for measurements. We use Clip-IQA [] for measurements, which is one of the SOTA no-reference image quality measurements.

## Appendix F Settings of Fine-tuning

**Table 5: Standard deviation of the results shown in Tab. 1.**
| Object-Driven Generation: DreamBooth Dataset | Subject-Driven Generation: CelebA Dataset |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Clip-T | Dino | Clip-I | Lpips | Clip-IQA | Method | Clip-T | Dino | Clip-I | Lpips | Clip-IQA |
| DreamBooth | 0.0017 | 0.0057 | 0.0037 | 0.0051 | 0.0028 | DreamBooth | 0.0065 | 0.0179 | 0.0115 | 0.0087 | 0.0078 |
| DreamBooth w/ BNNs | 0.0016 | 0.0056 | 0.0035 | 0.0054 | 0.0023 | DreamBooth w/ BNNs | 0.0036 | 0.0102 | 0.0090 | 0.0191 | 0.0095 |
| LoRA | 0.0018 | 0.0087 | 0.0045 | 0.0048 | 0.0084 | LoRA | 0.0034 | 0.0085 | 0.0059 | 0.0060 | 0.0068 |
| LoRA w/ BNNs | 0.0032 | 0.0104 | 0.0064 | 0.0072 | 0.0021 | LoRA w/ BNNs | 0.0025 | 0.0049 | 0.0163 | 0.0078 | 0.0065 |
| OFT | 0.0030 | 0.0063 | 0.0041 | 0.0081 | 0.0045 | OFT | 0.0026 | 0.0194 | 0.0141 | 0.0130 | 0.0115 |
| OFT w/ BNNs | 0.0010 | 0.0028 | 0.0026 | 0.0062 | 0.0068 | OFT w/ BNNs | 0.0024 | 0.0078 | 0.0066 | 0.0061 | 0.0094 |

We experiment on applying BNNs on different fine-tuning methods with one A100 GPU. All experiments are conducted under all 30 classes with 5 different seeds by default and we report the average performance. The standard deviation of our main result in Tab. 1 is shown in Tab. 5.

### F.1 Few-shot Fine-tuning Hyper-parameters

The details of the parameters in the few-shot fine-tuning methods on our default model, i.e., SD v1.5, are presented below. We use $\rm{Num}$ to represent the number of images utilized for training.

Dreambooth: We use the training script provided by Diffusers(^4^44https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py). Only the U-Net is fine-tuned during the training process. By default, the number of training steps is set to $200\times\rm{Num}$ on DreamBooth dataset and $250\times\rm{Num}$ on CelebA, with a learning rate of $5\times 10^{-6}$. The batch size is set to 1, and the number of class images used for computing the prior loss is $200\times\rm{Num}$ by default. The prior loss weight remains fixed at 1.0. For the DreamBooth dataset, the training instance prompt is “a photo of a [V] {class prompt}”, where{class prompt} refers to the type of the image (such as dog, cat and so on). For the CelebA dataset, the training instance prompt is “a photo of a [V] person”.

LoRA: We use the training script provided by Diffusers(^5^55https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py). All default parameters remain consistent with the case in Dreambooth (No Prior), with the exception of the learning rate and training steps, which are adjusted to $1\times 10^{-4}$ and to $400\times\rm{Num}$, respectively.

OFT: We use the training script provided by the authors(^6^66https://github.com/Zeju1997/oft). All default parameters remain consistent with the case in Dreambooth (No Prior), with the exception of the learning rate, which is adjusted to $1\times 10^{-4}$.

For all experiments involving the applying of BNNs, we maintain the default settings for the number of learning steps, learning rate, training prompts, and other hyper-parameters.

For evaluation purposes, each training checkpoint generates four images per prompt, resulting in a total of 100 images from 25 different prompts. The prompts used for evaluation are displayed in Tab. 4, encompassing a broad set to thoroughly assess the variety and quality of the DMs.

### F.2 Training Setting for DMs with Different Architectures

In this section, we provide training settings for applying BNNs on different DMs under DreamBooth. The learning rate is fixed to $5\times 10^{-6}$ in all cases.
The numbers of training iterations are set as $200\times\mathrm{Num}$ and $400\times\mathrm{Num}$ for SD v1.4 and SD v2.0, respectively.

For BNNs applying on SD v1.4, we set the hyperparameter $\lambda=0.1$.
All other hyperparameters are set as default.

## Appendix G Best-case v.s. Average-case Generation

After few-shot fine-tuning, we observe that every DM exhibits failure in generation sometimes either due to mismatching the prompt or generating images largely different with the learned object. Therefore, it is not beneficial to make comparisons based on the worst-case generation. However, average-case generation can be representative, in addition to the best-case scenario. Specifically, we first filter the generated images by selecting the top 90% using CLIP-T and Dino to ensure alignment with the prompt and the learned concept. Subsequently, we employ CLIP-IQA to identify both the top-quality and average-quality images. This approach provides a more comprehensive evaluation of the models’ performance.

## Appendix H User Study

User Study Setting.

We conduct user study to comprehensively present the superiority of applying BNNs on few-shot fine-tuning DMs. Concretely, we follow previous work [, ] and conduct a structured human evaluation for generated images, involving 101 participants.
For this purpose, we utilize the DreamBooth dataset and the CelebA-HQ dataset.
Subsequently, we generate four images per subject or object using a random prompt selected from 25 prompts across five models trained with different seeds.

Each participant is requested to compare image pairs generated by models that have been fine-tuned with and without BNNs, using three baseline models: DreamBooth, LoRA, and OFT.
For each task, there are three binary-selection questions:

- •
Subject fidelity: Which of the given two images contains a subject or object that is most similar to the following reference image (one from the training dataset)?
- •
Text alignment: Which of the given two images best matches the text description provided below (the prompt used to generate the images)?
- •
Overall image quality: Which of the given two images exhibits the higher image quality?

To comprehensively compare the performance of fine-tuning with and without BNNs, we select both the best and average cases from the generated images as mentioned in Sec. G.

Results. The results are presented in Tab. 6, which shows the percentage of participants favoring each method (with and without BNNs) based on the criteria described above. It is evident that the methods with BNNs are preferred in most scenarios for both best-case and average-case generations. This preference is particularly significant in terms of text alignment and overall image quality.

**Table 6: User study results of fine-tuned DMs with and without BNNs across various measurements under both best-case and average-case scenarios. The table depicts the percentage of users favoring generated images from fine-tuned DMs with and without BNNs.**
| Best-case Generation | Average-case Generation |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Subject Fidelity | Text Alignment | Overall Image Quality | Method | Subject Fidelity | Text alignment | Overall Image Quality |
| DreamBooth | 34.4% | 32.3% | 30.2% | DreamBooth | 50.5% | 35.6% | 28.7% |
| DreamBooth w/ BNNs | 65.6% | 67.7% | 69.8% | DreamBooth w/ BNNs | 49.5% | 64.4% | 71.3% |
| LoRA | 48.0% | 34.7% | 31.6% | LoRA | 27.5% | 26.5% | 24.5% |
| LoRA w/ BNNs | 52.0% | 65.3% | 68.4% | LoRA w/ BNNs | 72.5% | 73.5% | 75.5% |
| OFT | 30.6% | 34.7% | 40.8% | OFT | 41.1% | 26.8% | 39.3% |
| OFT w/ BNNs | 69.4% | 65.3% | 59.2% | OFT w/ BNNs | 58.9% | 73.2% | 60.7% |

## Appendix I Limitations and Future Work

Even though few-shot fine-tuning DMs with BNNs applied has shown promising improvements, this paper also introduces a few interesting open problems.
Firstly, the extra randomness may make fine-tuning slower. This may lower the generation quality when the DMs are under-fitting.
In addition, the ability of learning extremely detailed patterns in the image may be reduced when the number of fine-tuning iterations is insufficient.
Future work could focus on these problems.

## Appendix J Broader Impact

This paper focuses on advancing few-shot fine-tuning techniques in DMs to provide more effective tools for creating personalized images in various contexts.
Previous few-shot fine-tuning methods have faced corruption phenomenon, as mentioned in this paper. Our approach, utilizing BNNs, addresses this phenomenon and provides generated images with higher quality.

However, there is potential for misuse, as malicious entities could exploit these technologies to deceive or misinform. Such challenges underscore the critical need for continuous exploration in this field. The development and ethical application of personalized generative models are not only paramount but also ripe for future research.

## Appendix K More Visualizations

We show more visualized results in Fig. 12 and Fig. 13.

Figure: Figure 12: More visualizations on subject-driven and object-driven scenarios.
Refer to caption: x19.png

Figure: Figure 13: More visualizations on subject-driven and object-driven scenarios.
Refer to caption: x21.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see weakness
soundness: 2
strengths: - This work tries to analyze the foundational diffusion model’s fine-tuning process. 
- The authors incorporate the user study to support the effectiveness.
summary: This paper aims to analyze the progress of few-shot fine-tuning of diffusion models and improve it. First, they present the empirical analysis on the few-shot fine-tuning, which shows the corruption stage with the abnormal decrease in the fidelity of generated images. Then, they propose heuristic modeling, that a marginal distribution of $x_0$ with a parameterized diffusion model $\theta$ is approximated by a multivariate Gaussian distribution, to explain the few-shot fine-tuning procedure. Based on this modeling, they tried to understand why the corruption stage occurs. Finally, they propose the Bayesian neural network (BNN) approach. Through their experiments, they show that their method improves various fine-tuning methods including Dreambooth, LoRA, and Orthogonal Finetuning.
weaknesses: **W1: Observed Issues with Incomplete Linkages**


1. **W1-1 Incomplete Rationale for “Finding a Sample to Minimize the Error Term”**
   - In line 229, the statement, *“pre-trained diffusion model finds a sample $( x^\star )$ to minimize the error term”*, appears potentially misleading. First, diffusion models do not involve an explicit search mechanism; the phrase *“finding a sample”* might imply a deliberate search process that does not actually occur within the diffusion framework.
   - Second, I question whether the experiments in Fig. 3 adequately support the claim that the diffusion model *“minimizes the error term.”* It seems the authors imply that an unchanged $ x_0 $ when $ x_t = 0 $ is an indicator of error minimization ($ \delta $). However, my understanding is that diffusion models generate $ x_0 $ as a memorized sample from the training dataset, rather than as a result of minimizing an error term. Specifically, two points should be considered:
      - **Corrupted State of $ x_t $:** If $ x_t $ is initialized in an out-of-distribution state, the pre-trained diffusion model may fail to denoise such samples effectively. This would challenge the claim that the model consistently generates a sample from the training data.
      - **Validation of Training Data:** It’s unclear whether the sample $ x_0 = 0 $ corresponds to a specific instance in the training data. The prompt used, *“a simple, solid gray image with no textures or variations,”* may merely cause the model to generate a gray image without textures, as prompted, rather than replicating a sample from the training dataset.


   **Suggestion:** The authors might consider initializing $ x_t $ with actual samples from the training dataset used to train the diffusion model, alongside corresponding text labels. If the model does not alter $ x_t $ in these cases and the pixel-wise differences between generated samples and training data remain small, this would better support the claim.


2. **W1-2 Lack of Theoretical Validation for Error Minimization in Fine-Tuning**
   - To establish that the fine-tuning process drives the error term $ \delta $ to zero, the authors should consider providing a theoretical analysis showing how diffusion losses are minimized. Without such a theoretical basis, it is challenging to understand why the heuristic modeling is effective or whether the error term $ \delta $ indeed approaches zero at the optimal point.


3. **W1-3 Incomplete Explanation for the Corruption Stage during Fine-Tuning**
   - Building on W1-2, without a theoretical analysis of how $ \delta $ changes throughout fine-tuning, it’s difficult to comprehend why the corruption stage occurs. According to the authors’ experiments, fine-tuning the diffusion model on a limited number of samples results in significant errors after several iterations. This raises questions about why naive fine-tuning leads to this corruption stage (i.e., why $ \delta $ is high and $ \sigma $ remains high), which remain unanswered by the current modeling and experimental outcomes. Consequently, I believe the cause of corruption is not thoroughly studied in this work, and the conclusion that *“after some fine-tuning stages, error exists due to limited training steps”* seems somewhat trivial.


4. **W1-4 Unclear Justification for Using Bayesian Neural Networks (BNNs)**
   - Extending from W1-3, without a clear understanding of the corruption stage, I am not fully convinced why a Bayesian Neural Network (BNN) is presented as the solution. The authors mention that *“the modeling of BNNs hinders the diffusion models from learning the exact distribution of the training dataset $ D $”* but do not elaborate sufficiently on why this is beneficial in mitigating the corruption stage. Additional clarification about the cause of the corruption stage and further explanation of how BNNs address this underlying issue are needed.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. While BNNs reduce corruption, has the paper considered additional regularization methods to prevent overfitting in later stages?

2. Impact of Hyperparameter Tuning.

3. Does the use of BNNs significantly increase training time or memory consumption?
soundness: 3
strengths: - This work identifies and thoroughly investigates the "corruption stage" in few-shot fine-tuning of DMs, and it presents an innovative solution using BNNs to expand the learned distribution.
- The rigorous experimental design compares different fine-tuning methods across object-driven and subject-driven tasks. Multiple quantitative metrics (e.g., Clip-T, Dino, Clip-I) validate the effectiveness of the proposed method, and user studies support these quantitative results.
- The paper follows a clear logical flow, beginning with an explanation of the corruption stage, followed by an introduction to the application and benefits of BNNs.
summary: This paper investigates the phenomenon of the "corruption stage" observed in few-shot fine-tuning of Diffusion Models (DMs), where image fidelity initially improves but then deteriorates due to noisy patterns, before recovering with severe overfitting. Through heuristic modeling, the authors attribute this corruption stage to the narrowed learned distribution inherent in few-shot fine-tuning. To address this, they propose using Bayesian Neural Networks (BNNs) to expand the learned distribution via variational inference, thereby mitigating the corruption. Experimental results demonstrate that this method significantly enhances image fidelity, quality, and diversity in both object-driven and subject-driven generation tasks.
weaknesses: 1. Most experiments focus on specific datasets like DreamBooth and the Stable Diffusion v1.5 model without testing other diffusion models or more advanced Stable Diffusion versions, such as SDXL, SD-v3/3.5. I understand that the authors may be trying to solve a fundamental problem, but whether this problem exists on more advanced models still needs to be proven.

2. While BNNs offer significant performance improvements, the paper lacks a detailed analysis of computational costs, such as memory usage and inference time, compared to traditional methods, which is essential for assessing applicability in real-world scenarios. In my experience, there is often a trade-off between computational cost and performance for BNN. Does this exist in this work? If it exists, could the author provide a quantitative comparison of memory usage and inference time between their BNN approach and traditional methods?

3. Although the paper claims that BNNs mitigate corruption and reduce overfitting, there is limited information on how BNNs dynamically balance image diversity and fidelity at different training stages. Could the authors provide plots or other metrics demonstrating this balance across various training stages?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See Weaknesses above.
soundness: 2
strengths: 1. The use of BNNs to address noise and degradation in diffusion models’ fine-tuning is innovative, providing new insights and effective mitigation strategies for this issue.

2. The paper provides a robust theoretical foundation and demonstrates the effectiveness of BNNs through various empirical tests, showcasing the improvement in image quality and diversity.

3. BNNs integrate smoothly with existing fine-tuning frameworks, and by not increasing inference costs, they retain practical relevance for computationally constrained applications.
summary: The paper investigates the “corruption stage” in diffusion models during few-shot fine-tuning, where initial image quality improvements degrade as training progresses, leading to noise and overfitting. The authors propose using Bayesian Neural Networks (BNNs) to mitigate this by broadening the learned distribution, effectively improving the fidelity and diversity of generated images. BNNs add no additional inference cost and are compatible with current fine-tuning methods.
weaknesses: 1. Although the study shows reasonable results within the specific task context, its testing is limited to the dataset used in DreamBooth, which is too small in scope. While the authors have experimented with fine-tuning using 1, 2, and 6 images, it remains unclear whether the observed phenomena ("corruption stage") would still occur when the dataset size is increased to 1000 images or when batch size is adjusted. Given approaches like ControlNet and IP-adapter, how can it be demonstrated that the issues observed by the authors persist in large-scale fine-tuning scenarios? Furthermore, while the authors express hope that their observations might inform future research on diffusion models, it is uncertain if the proposed use of BNNs remains effective in large-scale data fine-tuning settings.

2. The paper essentially identifies a new problem and applies BNNs to few-shot fine-tuning tasks such as LoRA, which makes the novelty of the contribution somewhat limited.

3. The experiments mainly focus on score-matching-based Stable Diffusion models. It is uncertain how the proposed approach would perform with the more recent flow-matching models or with models using DiT as the denoising backbone.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see the weaknesses for my detailed questions.
soundness: 3
strengths: - The observed phenomenon is compelling and would be of significant interest to the community, especially as many works focus on fine-tuning pre-trained diffusion models for downstream tasks. 
- The analytical experiments are well-designed and supported by reasonable explanations. 
- The proposed use of BNNs consistently improves performance across various fine-tuning methods.
summary: This paper presents an empirical observation of diffusion models during fine-tuning in a few-shot setting, where the generated image quality undergoes a “corruption stage.” The authors analyze this corruption phenomenon through a series of experiments, beginning with single-sample fine-tuning, and provide an explanation based on noise correction terms. Building on this, they propose using Bayesian Neural Networks (BNNs) to mitigate corruption across different fine-tuning mechanisms.
weaknesses: - This paper could benefit from a broader coverage and discussion of related works on the similar topic of generation dynamics, such as [a] and [b]. There are quite a few existing works that reveal relevant phenomena, despite that they tackle more broadly instead of focusing purely on fine-tuning scenarios. 
- Fig. 4 is somewhat confusing; the text indicates that it shows the average value of $\sigma_{1,t}$ across different values of t, yet later mentions t=100. It’s unclear how t-values are defined in Fig. 4 and in the experiments, and how t specifically impacts the BNN mitigation effects
- The authors include qualitative examples of the best and average-case generations; however, I am curious about the worst-case scenarios. A detailed discussion of failure cases would be equally important and valuable, in many generation related works.
- Something more minor, I think it would be beneficial to compare the actual fine-tuning cost versus the BNNs applied tuning cost in Tab. 3 (or separately in Appendices).
- There is one missing "**(**" in Eq. 3. Also isn't it $KL(Q_w(\theta)||P(\theta))$ rather than $KL(P(\theta)||Q_w(\theta))$, or did I miss anything?

---
[a] Spontaneous symmetry breaking in generative diffusion models, NeurIPS 2023

[b] Dynamical Regimes of Diffusion Models, 2024.

Meta-review: This paper addresses the corruption stage in few-shot fine-tuning of diffusion models, where image quality degrades due to overfitting. It introduces a Bayesian Neural Network approach to improve the fidelity and diversity of generated images. The method is compatible with existing fine-tuning techniques like Dreambooth and LoRA and adds no extra inference cost.

Reviewers concerned the choice of $x_t=0$. The authors state that this assumption was made to "closely approximate practical scenarios." However, they fail to provide a convincing explanation of whether this assumption is valid. Additionally, questions remain regarding the theoretical justification of the corruption stage and the matter of diversity.

Decision: Reject